solution-day4-visualizing.qmd

---
title: "Solutions Day 4"
---

```{r setup, include=FALSE}
packages <- c("tidyverse", "lubridate", "rvest", "httr", "remotes", "knitr", "DT")
if (length(setdiff(packages, rownames(installed.packages()))) > 0) {
  install.packages(setdiff(packages, rownames(installed.packages())), repos = "https://cran.us.r-project.org")  
}

library(tidyverse)
library(lubridate)
library(readxl)
library(janitor)
library(knitr)
library(DT)

#myurl <- "https://www.cdc.gov/nchs/data/data_acces_files/NCHSURCodes2013.xlsx"
#download.file(myurl, (tf1 <- tempfile(fileext = ".xlsx")), mode = "wb")

#designations <- readxl::read_excel(tf1)
df <- read_csv("https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries.csv")

df_new <- df |> 
  mutate(GEOID=str_c(fipsStateCode, fipsCountyCode))


#county_pop <- read_csv("data/county_population.csv")
county_pop <- read_csv("https://www.andrewbatran.com/data/county_population.csv")

joined_new <- left_join(df_new, county_pop, by="GEOID") |> 
  mutate(year=year(incidentBeginDate))

annual_disasters <- joined_new |> 
  count(incidentType, year, name="total") 
  
annual_fires <- annual_disasters  |> 
  filter(incidentType=="Fire")


```

<span style="color:white">welcome to class!</span>


## Data Introduction

Before we begin, let's bring in the data we've been working with.

```{r imported, eval=T,  warning=F, message=F}
df <- read_csv("https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries.csv")

df_new <- df |> 
  mutate(GEOID=str_c(fipsStateCode, fipsCountyCode))

county_pop <- read_csv("data-raw/county_population.csv")

joined_new <- left_join(df_new, county_pop, by="GEOID") |> 
  mutate(year=year(incidentBeginDate))
```


Let's take a look at what were working with. Check out `joined_new` with the usual function we use:

```{r glimpse, eval=TRUE}
glimpse(joined_new)
```

Okay, let's transform the data like we did before. 

Can you count up how many **disasters** there have been **per year**? Not total disasters. Individual disasters.

Call the new column "total".

```{r annual_disasters, eval=TRUE}
annual_disasters <- joined_new |> 
  count(incidentType, year, name="total") 
  
    
annual_disasters
```

```{r annual_disasters-hint, eval=F}
You only need to add one new line. 
The function starts with a *c* and don't forget to name the column you're creating.
year should be the second argument in the function.
```

Alright, we've got a lot of data going back decades. 

The benefit of working quickly with data in R is how you can quickly visualize it to spot any trends.

Let's do that.

But before we do, let's create another data frame specifically for fires.

Filter incidentType for "Fire", please.

```{r fires, eval=TRUE}
annual_fires <- annual_disasters  |> 
  filter(incidentType=="Fire")

annual_fires
```


```{r fires-hint, eval=F}
The function starts with a *f* and don't forget ==
```


  

## Grammar of Graphics

The grammar of graphics lets you approach visualizations structurally, letting you combine and swap out graphical elements into figures that display data meaningfully.

It takes two lines of code.

This is what the code and chart looks like.

Run the code below.

```{r first_chart, eval=TRUE}
ggplot(data=annual_fires) +
  geom_col(aes(x=year, y=total)) 
```

Basically, every of a chart can created using these aesthetic components and mapping them:

### Mapping data to aesthetics

**Aesthetic**

* The visual property of a graph

* Position, shape, color, etc.

**Data**

* A column in a data set

-----

Here's are the core components of the chart above:

<table>
  <tr>
    <th class="cell-left">Data</th>
    <th class="cell-left">Aesthetic</th>
    <th class="cell-left">Graphic/Geometry</th>
  </tr>
  <tr>
    <td class="cell-left">Year</td>
    <td class="cell-left">Position (x-axis)&emsp;</td>
    <td class="cell-left">Column</td>
  </tr>
  <tr>
    <td class="cell-left">Total disasters</td>
    <td class="cell-left">Position (y-axis)</td>
    <td class="cell-left">Point</td>
  </tr>
</table>

-----

Here's how the data was mapped in ggplot2 code from the `annual_fires` data frame:

<table>
  <tr>
    <th class="cell-left">Data</th>
    <th class="cell-left">aes()</th>
    <th class="cell-left">geom</th>
  </tr>
  <tr>
    <td class="cell-left">year</td>
    <td class="cell-left">x</td>
    <td class="cell-left">geom_col()</td>
  </tr>
  <tr>
    <td class="cell-left">total</td>
    <td class="cell-left">y</td>
    <td class="cell-left">geom_col()</td>
  </tr>
</table>



### ggplot() template

Here's the dataframe called `annual_fires` as a reminder:

```{r preview}
annual_fires |> slice(1:5)
```

Okay, now that you see where all the pieces come from, here's how **ggplot()** works.

At its core you need to tell it what data you're using, what type of visual geometry you want to use, and what variables you want represented from the data.

**Important:** We have to use + signs between each line, not `|>`. This is because **ggplot()** was created before the **tidyverse** piping method was established.

```{r show-ggplot-template, eval=TRUE, echo=F}
include_graphics("images/flair.png")

```

---

```{r img0, echo = F, eval=FALSE, out.width="100%"}
library(knitr)
include_graphics("images/ggplot1.png")
``` 

### Grammatical layers

When constructing charts, so far we know about data, aesthetics, and geometries.

Think of these components as **layers**.

```{r img1a, echo = F, out.width="50%", fig.align="center"}
include_graphics("images/ggplot1a.png")
``` 

Add them to foundational `ggplot()` with +

These are all the arguments we can enhance the data viz with. 

Change the colors of the viz based on a column. Or the size of the shape.

Or the opacity or the gradient.

**Possible aesthetics**

```{r img2a, echo = F, out.width="100%"}
include_graphics("images/ggplot2a.png")
``` 

We can also swap out the different geometry types.

If you don't want a bar, you can use a line. Or a point.

You can even use shapefiles.

**Possible geoms**

```{r img3a, echo = F, out.width="60%", fig.align="center"}
include_graphics("images/ggplot3a.png")
``` 

THERE ARE [SO MANY GEOMS](https://twitter.com/search?q=geom_%20%23rstats&src=typed_query&f=image) for different visualizations. Here are the [official ones](https://ggplot2.tidyverse.org/reference/index.html#section-layer-geoms).

**Try the code from above again** but this time use **geom_point()** and then try it with **geom_line()**

```{r first_chart_geoms, eval=TRUE}
ggplot(data=annual_fires) +
  geom_point(aes(x=year, y=total)) 
```

You can really start to see the power of cycling quickly through different chart styles to see which one is most effective at telling the story you want to tell.

So after you have the very basic elements needed to create a chart, you can build and style it with more layers.

Because the defaults are rarely what you want and effective dataviz comes from small decisions you make along the way.

### Additional layers

There are many of other grammatical layers we can use to describe graphs.

We sequentially add layers onto the foundational `ggplot()` plot to create complex figures.


```{r img4a, echo = F, out.width="50%", fig.align="center"}
include_graphics("images/ggplot4a.png")
``` 

Scales change the properties of the variable mapping.

Here are a few examples:

<table>
  <tr>
    <th class="cell-left">Example layer</th>
    <th class="cell-left">What it does</th>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">scale_x_continuous()</code></td>
    <td class="cell-left">Make the x-axis continuous</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">scale_x_continuous(breaks = 1:5)&ensp;</code></td>
    <td class="cell-left">Manually specify axis ticks</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">scale_x_date()</code></td>
    <td class="cell-left">Considers x-axis dates</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">scale_color_gradient()</code></td>
    <td class="cell-left">Use a gradient</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">scale_fill_viridis_d()</code></td>
    <td class="cell-left">Fill with discrete viridis colors</td>
  </tr>
</table>

----

Check out the x-axis.

### Exercise 2

Now add `scale_x_continuous(limits=c(2010, 2022), breaks=2010:2022)` to the bottom of the code.


```{r scented2, eval=TRUE, warning=F, message=F}
ggplot(data=annual_fires) +
  geom_col(aes(x=year, y=total)) +
  scale_x_continuous(limits=c(2010, 2022), breaks=2010:2022)
  
```


Do you see the difference at the bottom of the chart compared to the one above it?

It limited the scope of the x-axis so it didn't go back to the '50s anymore.

And it specifically labeled the years 2010 through 2022.

## Facets

The next possible layer allows for small multiples. It's really neat.

Facets show subplots for different subsets of data.

<table>
  <tr>
    <th class="cell-left">Example layer</th>
    <th class="cell-left">What it does</th>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">facet_wrap(vars(incidentType))</code></td>
    <td class="cell-left">Plot for each disaster type</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">facet_wrap(vars(incidentType, year))</code>&emsp;</td>
    <td class="cell-left">Plot for each disaster type/year</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">facet_wrap(..., ncol = 1)</code></td>
    <td class="cell-left">Put all facets in one column</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">facet_wrap(..., nrow = 1)</code></td>
    <td class="cell-left">Put all facets in one row</td>
  </tr>
</table>

The table above shows all the different ways you can use facets-- you can break it out by one extra variable or even two.

We'll use the annual disasters this time so we have more than just the fires.

But we'll filter it to hurricanes and fires and floods.

And we can combine it with pipes before we use **ggplot()** it.

Add the `facet_wrap()` line on the variable **incidentType** (like the first example in the table above).

```{r facet_example1, eval=TRUE, warning=F, message=F}
annual_disasters |> 
  filter(incidentType %in% c("Hurricane", "Fire", "Flood")) |> 
ggplot() +
  geom_col(mapping=aes(x= year, y= total)) +
  scale_x_continuous(limits=c(2010, 2022), breaks=2010:2022) +
  facet_wrap(vars(incidentType))
```



Alright, looks like the x-axis labels are getting a little crowded.

We can't even read it!

### Try again!

Now, try it with `ncol=1` as an additional argument in `facet_wrap()`


```{r facet_example2, eval=TRUE, warning=F, message=F}
annual_disasters |> 
  filter(incidentType %in% c("Hurricane", "Fire", "Flood")) |> 
ggplot() +
  geom_col(mapping=aes(x= year, y= total)) +
  scale_x_continuous(limits=c(2010, 2022), breaks=2010:2022) +
  facet_wrap(vars(incidentType), ncol=1)
```


## Labels

Now we can add more customization to the chart.

To make it really shine!

<table>
  <tr>
    <th class="cell-left">Example layer</th>
    <th class="cell-left">What it does</th>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">labs(title = "Neat title")</code></td>
    <td class="cell-left">Title</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">labs(caption = "Something")</td>
    <td class="cell-left">Caption</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">labs(y = "Something")</td>
    <td class="cell-left">y-axis</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">labs(color = "Type")</code></td>
    <td class="cell-left">Title of size legend</td>
  </tr>
</table>

* Title should be "Disaster declarations since 2010"
* Label for the x-axis should be blank (aka "") because the years are obvious
* Label for the y-axis should be "Total"
* Caption should be "Data: FEMA"

Add those labels below:

```{r labels_example, eval=TRUE, warning=FALSE, message=FALSE}
annual_disasters |> 
  filter(incidentType %in% c("Hurricane", "Fire", "Flood")) |> 
ggplot() +
  geom_col(mapping=aes(x= year, y= total)) +
  scale_x_continuous(limits=c(2010, 2022), breaks=2010:2022) +
  facet_wrap(vars(incidentType), ncol=1) +
  labs(
    title = "Disaster declarations since 2010",
    x = "",
    y = "Total",
    caption= "Data: FEMA"
  )
```

```{r labels_example-hint, eval=F}
You only need to call labs() once.
Within parentheses, just separate the arguments with commas. You don't use the plus signs.
```


## Themes

Change the appearance of anything in the plot.

While you can customize every font, color, gradient, etc, you can set these styles up ahead of time or use the ones others have created.

There are many built-in themes.


<table>
  <tr>
    <th class="cell-left">Example layer</th>
    <th class="cell-left">What it does</th>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">theme_grey()</code></td>
    <td class="cell-left">Default grey background</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">theme_bw()</td>
    <td class="cell-left">Black and white</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">theme_dark()</td>
    <td class="cell-left">Dark</td>
  </tr>
  <tr>
    <td class="cell-left"><code class="remark-inline-code">theme_minimal()</code></td>
    <td class="cell-left">Minimal</td>
  </tr>
</table>

Try out the different themes listed above in the code below.

```{r themes-example, eval=TRUE, warning=FALSE, message=FALSE}
annual_disasters |> 
  filter(incidentType %in% c("Hurricane", "Fire", "Flood")) |> 
ggplot() +
  geom_col(mapping=aes(x= year, y= total)) +
  scale_x_continuous(limits=c(2010, 2022), breaks=2010:2022) +
  facet_wrap(vars(incidentType), ncol=1) +
  labs(
    title = "Disaster declarations since 2010",
    x = "",
    y = "Total",
    caption= "Data: FEMA"
  ) +
  theme_minimal()
```

### More themes

There are a collections of pre-built themes online, like the [ggthemes package](https://jrnold.github.io/ggthemes/).

Organizations often make their own custom themes, like [the BBC](https://bbc.github.io/rcookbook/).


```{r img10, echo = F, out.width="100%"}
include_graphics("images/bbc-cookbook.png")
``` 

### Theme adjustments

Make theme adjustments with `theme()`

There are a billion options here!

Add this chunk of code in the exercise below it:

```{r eval=F}
theme_bw() + 
theme(plot.title = element_text(face = "bold"),
      panel.grid = element_blank(),
      axis.title.y = element_text(face = "italic"))
```

### Exercise 5

```{r themes_example2, eval=TRUE}
annual_disasters |> 
  filter(incidentType %in% c("Hurricane", "Fire", "Flood")) |> 
ggplot() +
  geom_col(mapping=aes(x= year, y= total)) +
  scale_x_continuous(limits=c(2010, 2022), breaks=2010:2022) +
  facet_wrap(vars(incidentType), ncol=1) +
  labs(
    title = "Disaster declarations since 2010",
    x = "",
    y = "Total",
    caption= "Data: FEMA"
  ) +
theme_bw() + 
theme(plot.title = element_text(face = "bold"),
      panel.grid = element_blank(),
      axis.title.y = element_text(face = "italic"))

```

These were just a few examples of layers.

```{r img11, echo = F, fig.align="center", out.width="50%"}
include_graphics("images/ggplot5.png")
``` 

See the [ggplot2 documentation](https://ggplot2.tidyverse.org/reference/index.html) for complete examples of everything you can do

### Done!

Congrats on completing the walkthroughs for Class 4!

### But there's more


## Putting it all together

Usually we'd next go over all the different `geom_` visualizations you can create using a single data set.

* [A ggplot2 tutorial for beautiful plotting in R](https://cedricscherer.netlify.app/2019/08/05/a-ggplot2-tutorial-for-beautiful-plotting-in-r/)

But we're going to use more real-life data that I think will be relevant to your journalism.

The data set is raw deaths data from San Diego. It's a combination of 1997-2019 data from San Diego's [data portal](https://data.sandiegocounty.gov/Safety/Medical-Examiner-Cases/jkvb-n4p7/data) and 2020 data from a public information request on [MuckRock](https://www.muckrock.com/foi/santa-clara-county-732/santa-clara-county-medical-examiner-coroner-case-details-104835/).

Downloads this data [san_diego.csv](https://github.com/utdata/chj-r-introduction/raw/main/chj-day4/san_diego.csv) and place it in your project folder.

Make sure you've got the proper libraries loaded.


### Ready for the code?


```{r sd1, eval=T}
library(tidyverse)
library(janitor)
library(lubridate)

sd <- read_csv("data-raw/san_diego.csv")

sd <- clean_names(sd)

sd_adjusted <- sd |> 
  mutate(death_date=mdy(death_date)) |> 
  mutate(month=month(death_date, label=TRUE, abbr=TRUE)) 
```

```{r view_sd}
sd_adjusted
```

Alright, I've cleaned it up for you.

There's some really great data here. It's got gender, race, and several levels of manner of death stretching back to 1997 and through possibly October of 2020.

Let's start summarizing the data so we can start looking for trends.

### Exercise 7

Can you `count` up the number of `deaths` by manner_of_death by month and year, please?

```{r sd_month1, eval=TRUE}
sd_month <- sd_adjusted |> 
  count(year, month, manner_of_death, name="deaths") |> 
  mutate(date=mdy(paste0(month, " 1, ", year)))

sd_month
```


<div id="sd_month1-hint">
**Hint:** count / deaths
</div>

### Exercise 8

Now, fill in the blank below to create a line chart for each type of death

```{r sd_month_viz, eval=TRUE, fig.width=8}
sd_month |> 
  ggplot(aes(x=date, y=deaths, color=manner_of_death, group=manner_of_death)) +
  geom_line() +
  labs(title="Monthly deaths in San Diego",
       caption="Source; San Diego Medical Examiner")
```



Alright, a little messy. We can see some overall growing trend in one category, but that's it. 

Perhaps looking at the data by month is too granular. Let's step back and aggregate by year.

### By year

I went ahead and created a new `sd_year` dataframe counting up the deaths by year (while excluding October, November, and December) so we can compare prior years to this year.

```{r sd_year_data}
sd_year <- sd_adjusted |> 
  # if we're going to compare this year to previous years, we need to exclude data we don't have yet
  filter(!month %in% c("Oct", "Nov", "Dec")) |> 
  count(year, manner_of_death, name="deaths") 

datatable(sd_year)
```

### Exercise 9

Okay, your turn to make a chart.

Make me a faceted chart that breaks out all the individual **manner_of_death** into its own chart, small-multiple style.

```{r sd_year_chart, eval=TRUE, fig.width=8}
sd_year |> ggplot(aes(x=year, y=deaths)) +
  geom_col() +
  facet_wrap(vars(manner_of_death), ncol=4)
```


Alright, now we're getting somewhere. 

Looks like accidents have been trending up year over year.

If we focus on 2020, it looks like Natural causes have increased. But it also increased in 2019.

Suicides actually look down this year compared to prior years.

Hm...

What else can we do? 

We can try to measure **Excess Deaths**

Average each month by every year prior to 2020 and compare it to 2020's trend line.

I'll give you the code again.

We're going to use a function called `case_when` to create a new column called **year_type**. If the year is 2020, then it will be "2020" otherwise it will be "1997-2020". And then we find the average number of deaths for each month for those two groups.

```{r sd_group, warning=F, message=F}
sd_group <- sd_adjusted |> 
  filter(!month %in% c("Oct", "Nov", "Dec")) |> 
  count(year, month, manner_of_death, name="deaths") |> 
  mutate(year_type=case_when(
    year==2020 ~ "2020",
    TRUE ~ "1997-2019"
  )) |> 
  group_by(month, manner_of_death, year_type) |> 
  summarize(avg_deaths=mean(deaths, na.rm=T)) |> 
  filter(!is.na(month))

datatable(sd_group)
```

Looking very smooth.

Let's chart it.

### Exercise 10

Can you please create a faceted line chart of the data above? But with year_type as two different lines?

Fill in the three blanks to generate the chart.

```{r sd_group_viz, eval=TRUE, fig.width=8, fig.height=5}
ggplot(sd_group, aes(x=month, y=avg_deaths, color=year_type, group=year_type)) +
  geom_line() +
  facet_wrap(vars(manner_of_death), scales="free_y", ncol=2)
```



Great.

Now there's some interesting trend in July and August, right?

And it shows that maybe the last month of data is so low. It's possible that the data for the month is incomplete and should be excluded from analysis.

**How about we come up with a number for the nutgraf of a story?**

How many more accidental cause deaths are there in 2020 compared to the historical average?

*Exclude September since it seems so off*

Here's a look at the dataframe you can estimate this with.

```{r glimpse2}
datatable(sd_group)
```

What code do you put below to prove this sentence:

"There were X% more accidental deaths in in Jan 2020 than the historical average of January in San Diego"

*hint: you'll need filter(), pivot_wider(), group_by(), summarize(), mutate(), and math*

```{r excess_deaths, eval=TRUE}
sd_group  |> 
  filter(month=="Jan" & manner_of_death=="Accident") |>
  pivot_wider(names_from="year_type", values_from="avg_deaths") %>% 
  mutate(change=(`2020`-`1997-2019`)/`1997-2019`*100)

```

Good job!

### Further exploratory visual analysis

Alright, comparing 2020 to the average from previous years seems to be a good decision.

Some interesting trends that we could actually write about are surfacing.

Let's compare that overall instead of by month.

Here's the code. We're also excluding September based on what we discovered in the chart above.

```{r sd_group_compare}
sd_group_compare <- sd_adjusted |> 
  filter(!month %in% c("Sep", "Oct", "Nov", "Dec")) |> 
  count(year, manner_of_death, name="deaths") |> 
  mutate(year_type=case_when(
    year==2020 ~ "2020",
    TRUE ~ "1997-2019"
  )) |> 
  group_by(manner_of_death, year_type) |> 
  summarize(avg_deaths=round(mean(deaths, na.rm=T)))

datatable(sd_group_compare)
```


### Exercise 11

Run the code below with `manner_of_death` as x and `avg_deaths` as y.

Then swap them.

Which do you prefer and why?

```{r sd_group_compare_chart, eval=TRUE, fig.width=8}
ggplot(sd_group_compare, aes(x=manner_of_death, y=avg_deaths, fill=year_type)) +
  geom_bar(position="dodge", stat="identity") 
  
```



Alright, before we go, I want to clean things up some.

I want to get rid of the manners of death that have barely any and I want to reorder the labels so that it's in alphabetical order.

### Exercise 12

Take a look at the code below. Absorb it.

Then generate the code and see what pops up.

```{r sd_group_compare_chart_new, eval=TRUE}
sd_group_compare |> 
  filter(!manner_of_death %in% c("Other", "Family Paid Autopsy")) |> 
  filter(!is.na(manner_of_death)) |> 
  ggplot(aes(x=avg_deaths, y=forcats::fct_rev(manner_of_death),  fill=year_type)) +
  geom_bar(position="dodge", stat="identity") +
  labs(title="Manner of death in San Diego",
       subtitle="January and August deaths in 2020 compared to average deaths between 1997 and 2019",
       caption="Source: San Diego Medical Examiner",
       y="Manner of death",
       x="Average deaths",
       fill="Year") +
  theme_minimal()

```

### Story

So, what do you think the story is?

In San Diego, accidents are way up, suicides are slightly up, and meanwhile homicides are down.

What can we do next?

Well, dig into the accidents, perhaps, and see if there's any explanation for the huge increase.

Alright, congratulations on going on this exploratory data visualization journey.

Some of the answers won't appear right away unless you poke around and look at the data in as many ways as possible.