README.Rmd

---
output: github_document
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

```{r, include = FALSE}
library(dplyr)
```

# Elegant Data Manipulation with Lenses

## Installation

```{r, eval = FALSE}
install.packages("lenses")
```

or

```{r, eval = FALSE}
devtools::install_github("cfhammill/lenses")
```

## Intro

When programming in R there are two fundamental operations we perform on our data.
We `view` some piece of the data, or we `set` some piece of the data to a particular
value. These two operations are so fundamental that R comes with many pairs 
of `view` and `set` functions. A classic example would be `names`. Names can be 
viewed `names(x)` and set `names(x) <- new_names`. Lenses are an extension
of the idea of `view`/`set` pairs, offering the following advantages:

- lenses are bidirectional (you can `set` anything you can `view`)
- they are composable (they can be put together to `set`/`view` nested data)
- they reduce code duplication
- they play well with piping - [see R4DS to learn more about piping](http://r4ds.had.co.nz/pipes.html)
- and they are non-destructive (lenses won't update your data without your permission)

In this document, we'll see a few common data manipulation operations and
how they can be improved with lenses.

## Simple manipulations

Let's take the iris data set for example, we want to perform
some manipulations on it.

```{r}
head(iris)
```

We're curious about the value of the 3rd element of the `Sepal.Length` column.
Using base R we can `view` it with: 

```{r}
iris$Sepal.Length[3]
```

we can update (set) the value by assigning into it:

```{r}
iris$Sepal.Length[3] <- 100
head(iris, 3)
```

and we can perform some operation to update it:

```{r}
iris$Sepal.Length[3] <- log(iris$Sepal.Length[3])
```

This works well, however, there are some problems. 

The first problem comes with having our  `view` and `set` functions separate. 
Composing our operations isn't easy, particularly when using pipes:

```{r}
iris %>%
  .$Sepal.Length %>%
  `[<-`(3, 20)
```

Whoops, that's not what we wanted. Here we see `Sepal.Length` with
the third element replaced, but where did the rest of `iris` go! So
we lose information when we pipe from a `view` to a `set`. 

R's `set`/`view` pairs also can't be composed with function 
compostion:

```{r}
`[<-`(`$`(iris, `Sepal.Length`), 3, 20)
```

still not what we want. It has the same problem above.

This is a failure of "bidirectionality", once you've chosen to use a 
`view` function, or a `set` function, you are locked into that direction.

Lack of composability and bidirectionality means that you frequently have
to duplicate your code. For example, if you want to apply an operation to the
third element of "Sepal.Length", you need to specify the chain of accessors
twice, once in `view` mode, and once in `set` mode, making your code messy 
and cumbersome:

```{r}
iris$Sepal.Length[3] <- iris$Sepal.Length[3] * 2
head(iris, 3)
```

We can fix both of these problems by using lenses.

## Using lenses

Lenses give you all the power of R's `view` and `set` functions plus
the advantages noted above. Especially important are the composition
and bidirectionality features. Each lens can be used with the
[`view`](https://cfhammill.github.io/lenses/reference/view.html),
and [`set`](https://cfhammill.github.io/lenses/reference/set.html)
functions.

Let's revisit the operations we performed above using lenses. 

The first thing we will do is construct a lens into the 
third element of the `Sepal.Length` component of a structure:

```{r}
library(lenses)

sepal_length3 <- index("Sepal.Length") %.% index(3)
```

In the above code we're creating two lenses, one
into `Sepal.Length` and another into element 3, using the 
[`index`](https://cfhammill.github.io/lenses/reference/set.html) 
function. We're then composing these two lenses with
[`%.%`](https://cfhammill.github.io/lenses/reference/lens-compose.html)
producing a new lens into our element of interest.

Note that this lens has no idea we're going to apply it
to `iris`. Lenses are constructed without knowing what data they 
will be applied to.

Now that we have a lens into the third element of `Sepal.Length`, 
we can examine the appropriate element of the `iris` dataset with the 
[`view`](https://cfhammill.github.io/lenses/reference/view.html) function:

```{r}
iris %>% view(sepal_length3)
```

We can update this element with the 
[`set`](https://cfhammill.github.io/lenses/reference/set.html) function:

```{r}
iris %>% set(sepal_length3, 50) %>% head(3)
```

And we can apply a function to change the data. To
do this we can apply a function 
[`over`](https://cfhammill.github.io/lenses/reference/over.html) 
the lens:

```{r}
iris %>% over(sepal_length3, log) %>% head(3)
```

Note that we never had to respecify what subpart
we wanted, the lens kept track for us. We saw that
the same lens can be used to both 
[`view`](https://cfhammill.github.io/lenses/reference/view.html) and 
[`set`](https://cfhammill.github.io/lenses/reference/set.html),
and that they can be composed easily with 
[`%.%`](https://cfhammill.github.io/lenses/reference/lens-compose.html).

## More interesting lenses

Now you have seen the main lens verbs and operations

1. [`view`](https://cfhammill.github.io/lenses/reference/view.html): 
  see the subpart of an object a lens is focussed on.
1. [`set`](https://cfhammill.github.io/lenses/reference/set.html): 
  set the subpart to a particular value, then return the
  whole object with the subpart updated.
1. [`over`](https://cfhammill.github.io/lenses/reference/over.html): 
  apply a function to the subpart, then return the 
  whole object with the subpart updated.
1. [`%.%`](https://cfhammill.github.io/lenses/reference/lens-compose.html): 
  compose two lenses to focus on a subpart of a subpart.

Now if all lenses had to offer was more composable indexing of vectors, 
you  might not be interested in integrating them into your workflows. 
But lenses can do a lot more than just pick and set elements in vectors. 

For example, this package provides lens-ified version of `dplyr::select`.
Unlike `select`, 
[`select_l`](https://cfhammill.github.io/lenses/reference/select_l.html) 
is bidirectional. This means you can 
[`set`](https://cfhammill.github.io/lenses/reference/set.html) the results 
of your selection.

let's select columns between `Sepal.Width` and `Petal.Width` and increment them by 10:

```{r}
iris %>%
  over(select_l(Sepal.Width:Petal.Width)
     , ~ . + 10
       ) %>%
  head(3)
```

Not only does [`select_l`](https://cfhammill.github.io/lenses/reference/select_l.html) 
create the appropriate lens for you with `dplyr::select` style column references, 
but [`over`](https://cfhammill.github.io/lenses/reference/over.html) allows us 
to declare anonymous functions like in `purrr`.

At this point I can imagine you saying, all this is very clear, but what good is it, 
I have `mutate`. Well that is a good point. It is hard to beat the convenience
of `mutate`. However, `select_l` has an advantage, it can be used on any named
object:

```{r}
iris %>%
  as.list %>%
  view(select_l(matches("Sepal")) %.%
       index(1) %.%
       index(1)
       ) 
```

You can use it with vectors, lists, data.frames, etc.

If `select_l` isn't enticing enough, have you ever wanted to `set` 
or modify the results of a `filter`?  This is not super easy to do in the `dplyr` 
universe. But our `lens`ified `filter`, 
[`filter_l`](https://cfhammill.github.io/lenses/reference/filter_l.html) does this 
with ease.

Let's set all "Sepal" columns where the row number is less than three to 
zero. And for fun let's also change the column names to all upper case:

```{r}
library(dplyr)

iris %>%
  mutate(row_num = seq_len(n())) %>%
  set(filter_l(row_num < 3) %.%
      select_l(matches("Sepal"))
    , 0) %>%
  over(names_l, toupper) %>%
  head(3)
```

You can even use mutate `over` your `filter_l`

```{r}
iris %>%
  mutate(row_num = seq_len(n())) %>%
  over(filter_l(row_num < 3)
     , ~ mutate(., Sepal.Length = 0)) %>%
  head(3)
```

As you can see, lenses can be smoothly integrated into your `tidyverse`
workflows, as well as your base R workflows. Giving you the powers
of compositionality and bidirectionality to improve your code.

## Mapping lenses

Frequently we end up in situations where we want to modify each element
of a nested object. This is especially cumbersome without lenses. Let's
imagine our data lives inside a larger structure. And additionally that
it isn't a nice data frame, but a list.

```{r}
packed_iris <- list(as.list(iris))

packed_iris %>% str(2)
```

say I want to add 10 to the first element of each column between `Sepal.Length` and
`Petal.Width`. Base R I might do something like:

```{r}
els_of_interest <-
  grep("Sepal|Petal", names(packed_iris[[1]]), value = TRUE)

packed_iris[[1]][1:4] <-
  lapply(packed_iris[[1]][1:4]
       , function(x){ x[1] <- x[1] + 10; x })

str(packed_iris, 2)
```

pretty ugly right?

To do this with lenses we can use the `map_l` function to promote a `lens` to
apply to each element of a list.

```{r}
els_l <-
  index(1) %.%
  select_l(Sepal.Length:Petal.Width) %.%
  map_l(index(1))

over_map(packed_iris, els_l, ~ . + 10) %>%
  str(2)
```

Here we use the `over_map` function to apply a function to each element,
you could equivalently use `over` with `lapply` as well. As you can see
setting and applying functions to multiple elements of nested data is
dramatically improved by using lenses.

## Polishing your own

You can make a lens from scratch (!) by passing `view` and `set` functions to 
the [`lens`](https://cfhammill.github.io/lenses/reference/lens.html) constructor:

```{r}
first_l <- lens(view = function(d) d[[1]],
                set  = function(d, x) { d[[1]] <- x; d })
```

As you can see, the `view` function must accept an element of data, while the `set`
function must accept such an element as well as the new value of the subpart,
and return the new data in its entirety - thus achieving composability -
 without modifying the original.

In order to avoid unpleasant surprises or inconsistencies for users,
an author of a `lens` (via [`lens`](https://cfhammill.github.io/lenses/reference/lens.html)) 
should ensure it obeys the following rules (the "Lenz laws", here paraphrased 
from [a Haskell lens tutorial](www.schoolofhaskell.com/school/to-infinity-and-beyond/pick-of-the-week/a-little-lens-starter-tutorial)):

1. View-Set: If you `view` some data with a lens, and then
`set` the data with that value, you get the input data back.
2. Set-View: If you `set` a value with a lens,
then `view` that value with the same lens, you get back what you put in.
3. Set-Set: If you `set` a value into some data with a lens, and
then `set` another value with the same lens, it's the same as only
doing the second `set`.

"Lenses" which do not satisfy these properties should be documented accordingly.
By convention, the few such specimens in this library are suffixed by "_il" ("illegal lens").
See the package [reference](https://cfhammill.github.io/lenses/reference/) for more.

## How do they work?

As you can see from the `lens` constructor, knowing how to implement `view` and `set`
for a lens turns out to be sufficient to implement the other verbs such as `over` and
- most importantly - lens composition (`%.%`). 

In our implementation, lenses are trivial. They simply store the provided
functions. A `lens` under the hood is a two element list with an 
element `view` and an element `set`.

## History of lens making

There is nothing particularly new about the lenses appearing here.
For a fairly comprehensive (and highly technical) history of lenses, see
[links here](https://github.com/ekmett/lens/wiki/History-of-Lenses) and
[this blog post](https://julesh.com/2018/08/16/lenses-for-philosophers/) .

---

Thanks to Leigh Spencer Noakes, Zsu Lindenmaier, and Lily Qiu for reading 
drafts of this document and providing very helpful feedback.