Tutorial6_Visualization.Rmd

% Tutorial 6: Visualizations
% DPI R Bootcamp
% Jared Knowles

```{r setuph, include=FALSE}
# set global chunk options
opts_chunk$set(fig.path='figure/slides6-', cache.path='cache/slides6-',fig.width=12,fig.height=9,message=FALSE,error=FALSE,warning=FALSE,echo=TRUE,size='tiny',dev='png',out.width='600px',out.height='350px')
library(eeptools)
```

# Overview
In this lesson we hope to learn:

- What is data visualization and why does it matter?
- How to draw diagnostic plots in base graphics
- Colors
- `ggplot2' 
- Basic geoms
- Layering and faceting plots
- Putting it together

<p align="center"><img src="img/dpilogo.png" height="81" width="138"></p>

```{r setup, message=FALSE,warning=FALSE,echo=FALSE,comment=FALSE}
load('data/smalldata.rda')
dropbox_source<-function(myurl){
  require(stringr)
  s<-str_extract(myurl,"[/][a-z][/]\\d+[/][a-z]*.*")
  new_url<-paste("http://dl.dropbox.com",s,sep="")
  source(new_url)
}
```

# Graphics Matter
- Graphics are a huge part of R and R encourages its users to use visualization to aid and present analysis
- Graphics are hard to get right, but tools can help make it easier
- Lots of great resources to help available for this online and linked to at the end of this document

# Visualization Philosophy
- Visualization is about thinking about what objects in our dataset map with what visual cues for the viewer
  - liberal use of `names(object)` helps
- What is the best way to show discrete variables, continuous variables, and connected variables
- We also need to be unafraid about trying transformations of variables, recoding variables, and reshaping variables to tell the story we are looking for
- What is a plot?

# Our data for this exercise
- As we now know, student data is quite fascinating
- We have many discrete attribute variables
  - Race, gender, economics, etc.
- We have some continuous variables
  - Test scores, attendance
- We have some grouping variables
  - Schools, districts, students, teachers
- Rich ground to make a number of visualizations that are useful


# Base graphics
- R has base graphics which are useful for quickly doing some plots

```{r basehist}
hist(df$readSS)
```

# Base graphics are simple
- You can use them to understand your data and do quick diagnostics
- The commands are pretty easy to remember
- You can do complex things with them if you like, but the syntax can be confusing

# Base graphics has some limitations
- What if we try to do a scatterplot of lots of data?

```{r basescatter,out.width='300px',out.height='200px'}
plot(df$readSS,df$mathSS)
```

- To add a line, we have to use another line of code 

```{r scatterbaseline,out.width='300px',out.height='200px'}
plot(df$readSS,df$mathSS)
lines(lowess(df$readSS~df$mathSS),col='red')
```


# Basic Plot
- Don't use R base graphics
- `ggplot2` is pretty much the new standard in R
- Create beautiful, clear plots with a nice consistent set of language conventions

```{r plot1,out.height='325px'}
library(ggplot2)
qplot(readSS,mathSS,data=df)
```


# Now, how?
- Apply best visualization principles, particularly the "Grammar of Graphics"
- `ggplot2` an R package does just this by breaking plots into a few basic components
  - aesthetics, geoms, layers, and scales
- Making high quality graphics is just a matter of applying the right data to the right aesthetics in an appropriate geom, adding layer for detail, and ensuring the scale is appropriate
- Let's talk about how to do that next

# What are the basic plot types?
```{r ggplot2plottypes,echo=FALSE,out.width='850px',out.height='520px'}
p1<-qplot(factor(grade),readSS,data=df,geom='boxplot')+theme_dpi(base_size=12)
p2<-qplot(readSS,data=df,geom='density')+theme_dpi(base_size=12)
p3<-qplot(readSS,data=df)+theme_dpi(base_size=12)
p4<-qplot(grade,readSS,data=df,geom='line',group=1,stat='quantile')+theme_dpi(base_size=12)
p5<-qplot(mathSS,readSS,data=df)+theme_dpi(base_size=12)
library(plyr)
df2<-ddply(df,.(schid),summarize,mean_math=mean(mathSS,na.rm=T))
p6<-qplot(factor(schid),mean_math,data=df2,geom='bar',stat='identity')+theme_dpi(base_size=12)
print(grid.arrange(p1,p2,p3,p4,p5,p6,nrow=2))
```


# What are some advanced plot types?
```{r ggplot2plottypesadv,echo=FALSE,out.width='850px',out.height='520px'}
p1<-qplot(factor(grade),readSS,data=df,geom='boxplot')+theme_dpi(base_size=12)+ stat_summary(fun.data = "mean_cl_boot", colour = "red",size=(1.2))
p2<-qplot(readSS,data=df,geom='density',fill=race)+theme_dpi(base_size=12)+opts(legend.position='bottom')
p3<-qplot(readSS,data=df,fill=factor(female),position='stack')+theme_dpi(base_size=12)+opts(legend.position='bottom')
p4<-qplot(grade,readSS,data=df,geom='line',group=1,stat='quantile')+theme_dpi(base_size=12)+geom_jitter(alpha=(.2))
p5<-qplot(mathSS,readSS,data=df,alpha=I(.25))+theme_dpi(base_size=12)+stat_density2d(aes(color=factor(df$disab)),size=I(1.03))+opts(legend.position='bottom')
df2<-ddply(df,.(schid),summarize,mean_math=mean(mathSS,na.rm=T),sd_math=sd(mathSS,na.rm=T))
p6<-ggplot(df2,aes(x=factor(schid),y=mean_math,ymin=mean_math-2*sd_math,ymax=mean_math+2*sd_math))+geom_pointrange()+geom_errorbar(size=I(1.03),width=.25,color='grey40')+theme_dpi(base_size=12)
print(grid.arrange(p1,p2,p3,p4,p5,p6,nrow=2))
```

# 
Your Turn: What are some examples of interesting visualizations we could use?
-------------------------------------------------------------------


# Understanding Grammar of Graphics through A Scatterplot
```{r smallscatter,out.width='300px',out.height='240px'}
qplot(readSS,mathSS,data=df,alpha=I(.3))+theme_dpi()
```

- What is a scatterplot?
- A 2d representation of data as coordinates, in this case the `readSS` is the x coordinate and `mathSS` is the y coordinate for each observation in our data
- Each observation is a point (*geom*)
- Both axes scale in a linear fashion (*scales*)

# Grammar of Graphics
- In one way of thinking about this, each data visualization has four components
- **Geometries** that represent data (*points, bars, lines*)
- **Statistics** that represent information about the data (*identity, mean, median, deviance*)
- **Scales** that map the geometries and statistics to space (*linear, quadratic, logrithmic*)
- A **coordinate system** and canvas to put all on


# Geoms
- **Geoms** are the way data is represented, you can think of it like a chart type in another programming language
- We have seen a number of examples, and **geoms** can be combined in unique ways to convey more data
- Quiz: Represent `df$mathSS` using **3 separate geoms**

# Geom Quiz
```{r geomquiz,out.width='350px',out.height='260px'}
qplot(mathSS,readSS,data=df)+theme_dpi()
qplot(mathSS,data=df)+theme_dpi()
qplot(factor(grade),mathSS,data=df,geom='line',group=stuid,alpha=I(.2))+theme_dpi()
```

# Aesthetics
- **geoms** allow us to only display a couple of data elements at once, to do more we need to map to other visual representations
- This is what *aesthetics* are
- aesthetics are colors, glyphs (shapes), and sizes of graph objects mapped to visual cues
- `ggplot2` has an extended syntax that makes this obvious

```{r extended,out.height='240px'}
ggplot(df,aes(x=readSS,y=mathSS))+geom_point()
# Identical to: qplot(readSS,mathSS,data=df)
```

- `aes` says we are specifying aesthetics, here we specified x and y to make a two dimensional graphic

# Examples of Aesthetics


```{r plot2,out.width='400px',out.height='300px'}
data(mpg)
qplot(displ,cty,data=mpg)+theme_dpi()
qplot(displ,cty,data=mpg,size=cyl)+theme_dpi()
qplot(displ,cty,data=mpg,shape=drv,size=I(3))+theme_dpi()
qplot(displ,cty,data=mpg,color=class)+theme_dpi()
```

# Experiment with Aesthetics
### Draw some plots with different aesthetics using our student level dataset

# Some Considerations with Aesthetics

- **aesthetics** are very sensitive to whether a variable is continuous or discrete or ordered
- **R** isn't always so worried about this!
- Why does it matter? Let's see a few examples

# Aesthetics Considerations (ordered)

```{r racesizemapping}
qplot(mathSS,readSS,data=df[1:100,],size=race,alpha=I(.8))+theme_dpi()
```
- Does this make sense?

# Another Aesthetics Concern (ordered)

```{r proflvlcolor}
df$proflvl2<-factor(df$proflvl,levels=c('advanced','basic','proficient','below basic'))
df$proflvl2<-ordered(df$proflvl2)
qplot(mathSS,readSS,data=df[1:100,],color=proflvl2,size=I(3))+scale_color_brewer(type='seq')+theme_dpi()
```

# Aesthetics Concern 2 (discrete and continuous)
- What aesthetics can we map continuous variables like `mathSS` to, and what can we map discrete characteristics like `race` to?

```{r badcontinuousmapping}
qplot(factor(grade),readSS,data=df[1:100,],color=mathSS,geom='jitter',size=I(3.2))+theme_dpi()
```
- What's wrong?

# Aesthetics Concern 2

```{r baddiscretemap}
qplot(factor(grade),readSS,data=df[1:100,],color=dist,geom='jitter',size=I(3.2))+theme_dpi()
```
- What's wrong?

# Thinking about Aesthetics

- One concern is discrete v. continuous variables

Aesthetic       Discrete                 Continuous
----------  ---------------------------  --------------------------------
Color       Disparate colors             Sequential or divergent colors
Size        Unique size for each value   mapping to radius of value
Shape       A shape for each value       **does not make sense**

# Another is ordered v. unordered


Aesthetic      Ordered                           Unordered
----------  --------------------------------     -------------------------
Color       Sequential or divergent colors      Rainbow
Size        Increasing or decreasing radius     **does not make sense**
Shape       **does not make sense**             A shape for each value


# Scales
- Scales are the way the numeric/categorical data is mapped to a visual representation
- They transform the geoms and aesthetics
- Scales preserve the mapping, but allow us to explore reshaping the data

```{r scaleexample,echo=FALSE}
p1<-qplot(factor(grade),mathSS,data=df,geom='jitter',alpha=I(.3))+theme_dpi(base_size=12)
p2<-qplot(factor(grade),mathSS,data=df,geom='jitter',alpha=I(.3))+scale_y_sqrt()+theme_dpi(base_size=12)
p3<-qplot(factor(grade),log2(mathSS),data=df,geom='jitter',alpha=I(.3))+theme_dpi(base_size=12)
p4<-qplot(grade,log2(mathSS),data=df,geom='jitter',alpha=I(.3))+
  scale_x_continuous(breaks=c(3,6,8),labels=c('young','middle','old'))+theme_dpi(base_size=12)
p5<-qplot(factor(grade),mathSS,data=df,geom='jitter',alpha=I(.3))+
  scale_y_reverse()+theme_dpi(base_size=12)
p6<-qplot(factor(grade),mathSS,data=df,geom='jitter',alpha=I(.3))+
  scale_y_continuous(breaks=c(250,400,600,700),
                     labels=c('minimal','basic','proficient','advanced'))+theme_dpi(base_size=12)
grid.arrange(p1,p2,p3,p4,p5,p6,nrow=2)
```

# Scales Caveats
- Despite not changing the  mapping of data to space, scales can dramatically influence the interpretation of our data
- Rescaling data should be done thoughtfully and with care
- Picking a good scale can be really difficult, and sometimes, a good scale just won't exist without reshaping or subsetting the data!

# Scales also apply to color and fill
- The colors we choose can carry important meaning, as we have discussed
- So can the shapes we use for points, or for lines
- It is important to keep this in mind

# Layers
- Exactly what they sound like, each plot is a simple series of layers
- One way to do layers is to break plots up into small multiples (see Tufte)

```{r smallfacets}
qplot(readSS,mathSS,data=df)+facet_wrap(~grade)+theme_dpi(base_size=12)+
  geom_smooth(method='lm',se=FALSE,size=I(1.2))
```

# We can also facet across more attributes

```{r smallfacets2}
qplot(readSS,mathSS,data=df)+facet_grid(ell~grade)+theme_dpi(base_size=12)+
  geom_smooth(method='lm',se=FALSE,size=I(1.2))
```

# Visualizing Categorical Data
- Visualization is not just limited to continuous data using scatterplots
- Sometimes we want to look at the density of data in different categories
- Sometimes we want to look at the size of groups compared to an expected group size
- What are some other examples?

# Structural Plots
```{r strucplot}
library(vcd)
df$proflvl<-factor(df$proflvl,levels=c("advanced","proficient","basic","below basic"))
a<-structable(proflvl~race,data=df)
mosaic(a,shade=TRUE)
```

# How do we interpret these plots?
- We use size and color to express proportionality and expected values
- This can be applied to other areas as well 

# Another example
```{r strucplot2}
library(vcd)
df$proflvl<-factor(df$proflvl,levels=c("advanced","proficient","basic","below basic"))
a<-structable(female~race,data=df)
mosaic(a,shade=TRUE)
```


# A few pro tips
- Don't be afraid to manipulate the data
  - Aggregate, combine, rescale variables to make the story easier to understand
- Don't use too much color
  - People print your work, color blindness is more prevalent than you think
  - Color is fickle. Monitors and projectors reproduce it differently.
- Don't overwhelm the user
  - Make your visual depictions as easy as possible to immediately understand that data
- Don't plot everything all at once

# Colors in R
- ggplot2 default colors are based on maximizing space between colors using HCL
- Read more here: [A tutorial on ggplot2 colors](http://is-r.tumblr.com/post/34693674524/ggtutorial-day-3-introduction-to-colors);  [HCL color](http://hclcolor.com)

```{r ggplot2colors,echo=FALSE}
library(gridExtra)
X <- c(1)
Y <- c(1)

DF <- as.data.frame(cbind(X,Y))
c.1 <-  ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))  + coord_flip()   +  theme(legend.position="none") + labs(x="", y="", title="One Colour") + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))

X <- c(1:2)
Y <- rep(1, length(X))
DF <- as.data.frame(cbind(X,Y))
c.2 <- ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))   + coord_flip()   +  theme(legend.position="none")   + labs(x="", y="", title="Two Colours")           + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))

X <- c(1:3)
Y <- rep(1, length(X))
DF <- as.data.frame(cbind(X,Y))
c.3 <- ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))  + coord_flip()   +  theme(legend.position="none")    + labs(x="", y="", title="Three Colours")     + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))

X <- c(1:4)
Y <- rep(1, length(X))
DF <- as.data.frame(cbind(X,Y))
c.4 <- ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))   + coord_flip()   +  theme(legend.position="none")+ labs(x="", y="", title="Four Colours")   + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))

X <- c(1:5)
Y <- rep(1, length(X))
DF <- as.data.frame(cbind(X,Y))
c.5 <- ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))   + coord_flip()   +  theme(legend.position="none")   + labs(x="", y="", title="Five Colours")    + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))

X <- c(1:6)
Y <- rep(1, length(X))
DF <- as.data.frame(cbind(X,Y))
c.6 <- ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))   + coord_flip()   +  theme(legend.position="none")  + labs(x="", y="", title="Six Colours")  + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))

X <- c(1:7)
Y <- rep(1, length(X))
DF <- as.data.frame(cbind(X,Y))
c.7 <- ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))   + coord_flip()   +  theme(legend.position="none")   + labs(x="", y="", title="Seven Colours")  + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))


X <- c(1:8)
Y <- rep(1, length(X))
DF <- as.data.frame(cbind(X,Y))
c.8 <- ggplot(data = DF) + geom_bar(aes(x = X, y = Y, fill = factor(X)))   + coord_flip()   +  theme(legend.position="none")  + labs(x="", y="", title="Eight Colours")   + scale_y_continuous(breaks=NULL, expand=c(0,0)) + scale_x_continuous(breaks=NULL, expand=c(0,0))

MfrowGG <- grid.arrange(c.1, c.2, c.3, c.4, c.5,
                         c.6, c.7, c.8, ncol=2)

```


# The R Colorspace is Huge
```{r colorwheel,out.width='300px',out.height='240px',echo=FALSE}
colwheel<-"https://dl.dropbox.com/u/1811289/colorwheel.R"
dropbox_source(colwheel)
col.wheel("magenta",nearby=2)
col.wheel("orange",nearby=2)
col.wheel("brown",nearby=2)
```

# Some Practical Advice
- Use [ColorBrewer palettes](http://colorbrewer2.org/): Available by default in ggplot2
- Designed for geographic mapping purposes to be easy to distinguish, easy to interpret across a diverse array of users, and sensitive to color-blindness
- As easy in ggplot2 as adding any color scale: `+scale_color_brewer(palette"X")`
- You can choose from sequential palettes, diverging palettes, qualitative palettes,and many more--depending on how many colors you need and what makes sense in your data
- Let's think about colors a little more together

# Above and Beyond
```{r premier,echo=FALSE,out.width='800px',out.height='520px'}
library(grid)
p1<-qplot(readSS,..density..,data=df,fill=race,
      position='fill',geom='density')+scale_fill_brewer(
        type='qual',palette=2)

p2<-qplot(readSS,..fill..,data=df,fill=race,
      position='fill',geom='density')+scale_fill_brewer(
        type='qual',palette=2)+ylim(c(0,1))+theme_bw()+
          opts(legend.position='none',
               axis.text.x=theme_blank(),
               axis.text.y=theme_blank(),
               axis.ticks=theme_blank(),
               panel.margin=unit(0,"lines"))+ylab('')+
                 xlab('')

vp<-viewport(x=unit(.65,"npc"),y=unit(.73,"npc"),
             width=unit(.2,"npc"),height=unit(.2,"npc"))
print(p1)
print(p2,vp=vp)

```


# Scary R Code
```{r premiernoeval,eval=FALSE,tidy=FALSE}
library(grid)
p1<-qplot(readSS,..density..,data=df,fill=race,
      position='fill',geom='density')+scale_fill_brewer(
        type='qual',palette=2)

p2<-qplot(readSS,..fill..,data=df,fill=race,
      position='fill',geom='density')+scale_fill_brewer(
        type='qual',palette=2)+ylim(c(0,1))+theme_bw()+
          opts(legend.position='none',
               axis.text.x=theme_blank(),
               axis.text.y=theme_blank(),
               axis.ticks=theme_blank(),
               panel.margin=unit(0,"lines"))+ylab('')+
                 xlab('')

vp<-viewport(x=unit(.65,"npc"),y=unit(.73,"npc"),
             width=unit(.2,"npc"),height=unit(.2,"npc"))
print(p1)
print(p2,vp=vp)

```


# Exercises 

1. Embed one plot in another plot in R using two different data elements from our data set. For example, plot a histogram of `readSS` inside a scatterplot of `readSS` and `mathSS`

2. Explore some examples on the ggplot2 website. What are some ways to overlay more than 3 dimensions of data in a single plot?

3. What types of data work best for what types of visualizations?

# References
1. [Hadley Wickham's JSM 2012 Presentation](http://www.stat.yale.edu/~jay/JSM2012/PDFs/ggplot2.pdf)
2. [Hadley Wickam's ggplot2 Intro Presentation](http://had.co.nz/ggplot2/resources/2007-vanderbilt.pdf)
3. [The ggplot2 Homepage](http://had.co.nz/ggplot2/)
4. [ggplot2 Documentation](http://had.co.nz/ggplot2/docs)
5. [Quick R: Basic Graphs](http://www.statmethods.net/graphs/index.html)
6. [Quick R: Advanced Graphs](http://www.statmethods.net/advgraphs/index.html)



# Session Info

It is good to include the session info, e.g. this document is produced with **knitr** version `r packageVersion('knitr')`. Here is my session info:

```{r session-info}
print(sessionInfo(), locale=FALSE)
```


# Attribution and License
<p xmlns:dct="http://purl.org/dc/terms/">
<a rel="license" href="http://creativecommons.org/publicdomain/mark/1.0/">
<img src="http://i.creativecommons.org/p/mark/1.0/88x31.png"
     style="border-style: none;" alt="Public Domain Mark" />
</a>
<br />
This work (<span property="dct:title">R Tutorial for Education</span>, by <a href="www.jaredknowles.com" rel="dct:creator"><span property="dct:title">Jared E. Knowles</span></a>), in service of the <a href="http://www.dpi.wi.gov" rel="dct:publisher"><span property="dct:title">Wisconsin Department of Public Instruction</span></a>, is free of known copyright restrictions.
</p>