Intro_to_data_science_tools-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting help\n",
    "\n",
    "Part of what makes tools like R more accessible than ever before is how easy it is to get help. For R and its primary data science library, tidyverse, the excellent (and free) book \"R for Data Science\" can be found at http://r4ds.had.co.nz/\n",
    "\n",
    "When doing basic analyses, simple web searches will often help you either find documentation or explanations for accomplishing most tasks. [Stack Overflow](https://stackoverflow.com/) is a great source and has specific areas for R and its libraries like dplyr which is the primary library for manipulating data: https://stackoverflow.com/questions/tagged/r+dplyr. The main pages for each library also offer great documentation like [dplyr](http://dplyr.tidyverse.org/). \n",
    "\n",
    "RStudio also provides several \"cheatsheets\" which are useful references here: https://www.rstudio.com/resources/cheatsheets/\n",
    "\n",
    "Finally, when you are using RStudio, you can get built in help by using the help function by adding a question mark before the command you want help with like ```?function()```. This function may not work with Jupyter notebooks depending on your operating system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding our data science libraries\n",
    "\n",
    "To get start our exploration of data science tools with R, we are going to use a dataset from the [Capital Bikeshare](https://www.capitalbikeshare.com/) in 2011 and 2012. A description of the dataset is [here](./data/bike_data_readme.txt).\n",
    "\n",
    "The first thing we need to do is load the [Tidyverse data science library](http://tidyverse.org).  R has thousands of additional libraries you can add to accomplish many different tasks without having to code them yourself. A partial listing is [here](https://cran.r-project.org/web/packages/available_packages_by_name.html). Given R's statistical and academic orgins, many of the scientific libraries were subjected to peer review as the code provided the method for scientific research papers- https://scholar.google.com/scholar?start=120&q=%22R+Core+Team%22. \n",
    "\n",
    "For our Jupyter Hub environment, we have pre-loaded many libraries, so you shouldn't need to install new ones for these exercises."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load tidyverse into your R session\n",
    "library(tidyverse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load data from a CSV file\n",
    "\n",
    "Loading data is easy. We will use the `read_csv` function to read the data from our file into a variable called `df` which is short for dataframe.  In R, you assign things to variables with the `<-` operator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df <- read_csv(\"data/bike_day.csv\") # run this code block...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you run this cell R will provide a printout of how it decided to encode the column headers and the datatype of each column. Datatypes are important given functions may require specific datatypes. For example, if you encode a column of your csv file with data like \"1-11-2018\" as a string or character type, R will not be able to perform any date functions on it.\n",
    "\n",
    "As with most things in R, you can override default functionality specify the datatypes as follows:\n",
    "\n",
    "```R\n",
    "df <- read_csv(\"data/bike_day.csv\", col_types = \n",
    "  cols(instant = col_integer(),\n",
    "  dteday = col_date(format = \"\"),\n",
    "  season = col_integer(),\n",
    "  yr = col_integer(),\n",
    "  mnth = col_integer(),\n",
    "  holiday = col_integer(),\n",
    "  weekday = col_integer(),\n",
    "  workingday = col_integer(),\n",
    "  weathersit = col_integer(),\n",
    "  temp = col_double(),\n",
    "  atemp = col_double(),\n",
    "  hum = col_double(),\n",
    "  windspeed = col_double(),\n",
    "  casual = col_integer(),\n",
    "  registered = col_integer(),\n",
    "  cnt = col_integer()\n",
    "  )\n",
    ")\n",
    "```\n",
    "\n",
    "As we will see later, RStudio makes this much easier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Look at the data you loaded\n",
    "\n",
    "To see your data, just enter your dataframe name into a code cell and execute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dataframe is a two dimensional matrix much like an Excel worksheet where each column has a name and data is presented in rows. \n",
    "\n",
    "In order to access the data from specific columns we can use the following syntax:\n",
    "\n",
    "```R\n",
    "dataframe$columnname\n",
    "```\n",
    "\n",
    "If you try this in a code cell, you will see all the potential columns show up to select if you hit `tab` after the `$` symbol.\n",
    "\n",
    "You can access ranges of columns or rows with this syntax:\n",
    "\n",
    "```R\n",
    "dataframe[rows, columns]\n",
    "dataframe[1:5,] # This would retrieve (what we call slice) the first 5 rows.\n",
    "dataframe[,1:5] # This would retrieve (what we call slice) the first 5 columns.\n",
    "```\n",
    "\n",
    "You can also slice columns based on their name.\n",
    "\n",
    "```R\n",
    "dataframe['columnname'] # one column named 'columnname'\n",
    "dataframe[c('columnname1','columnname2')] # two columns using `c()` combine function.\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# try slicing your dataframe here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize the data\n",
    "\n",
    "Creating a plot of your data is often one of the first approaches data scientists use to better understand the data and specifically look at the distribution of the data.\n",
    "\n",
    "R has two major plotting packages `plot()` and `ggplot()`. Ggplot2 is the most popular, so we will use it going forward. \n",
    "\n",
    "The general syntax is to set up an empty plot `ggplot()` and then add a geom or \"geometrical object\" to represent the data. Geoms inlcude points, bars, lines, and other graphical representations. Here is a simple example:\n",
    "\n",
    "```R\n",
    "ggplot(data = dataframe) # this creates an empty plot that you can add layers to using the \n",
    "\n",
    "+ # this adds the layer functions we want to the base plot\n",
    "\n",
    "geom_points(mapping = aes(x=x_column+name, y=y_column_name)) # geom_points says we want a scatterplot (points) and mapping = defines how we want to map variables into the x and y axes\n",
    "\n",
    "```\n",
    "\n",
    "The `aes()` stands for \"aesthetic\" and is where you define how you want to display the data on the plot. This term is derived from concepts behind the book [Grammar of Graphics](https://www.amazon.com/Grammar-Graphics-Statistics-Computing/dp/0387245448) which is also why ggplot has a \"gg\" in its name. \n",
    "\n",
    "It is easy to add custom formatting to a plot by adding on functions with a plus symbol `+`. Here are some of the basic ones:\n",
    "\n",
    "\n",
    "```R\n",
    "+ ggtitle(\"Main plot title\") # add a title\n",
    "+ ylab(\"This is the Y axis\") # add a axis label change y to x for other axis\n",
    "```\n",
    "\n",
    "Execute the code in the next code block. Try changing the geom from geom_point to other shapes. You can also add geoms that conduct statistical transforms to represent the data line `geom_smooth()` http://ggplot2.tidyverse.org/reference/geom_smooth.html.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(data=df) + \n",
    "  geom_point(aes(x=dteday, y=cnt)) # Note you can leave out 'mapping =' which is more conventional"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make your plot your own\n",
    "\n",
    "Try adding more formatting (titles, better axes labels, etc.) to the base plot using the `+` operator and optional formatting from this document- https://www.rstudio.com/wp-content/uploads/2015/03/ggplot2-cheatsheet.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your plot and ggplot options here and run...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using dplyr to continue the analysis\n",
    "\n",
    "After an intial visualization, most analyses will involve looking at different or more specific manipulations of the data in the dataframe. Dplyr offers several useful functions for this manipulation including:\n",
    "\n",
    "```R\n",
    "summarise() # create new variables which summarize the data, for example a mean of values\n",
    "filter() # filter out cases (usually rows based on their values)\n",
    "mutate() # add new variables to the dataframe\n",
    "select() # choose a subset of columns in a dataframe\n",
    "arrange() # sort the dataframe\n",
    "group_by() # analyze the data by grouping\n",
    "```\n",
    "\n",
    "These different functions can be combined to create a data pipeline where each step is a different operation on the dataframe. In dplyr, data can be pipelined between different steps using the pipeline operator `%>%` (more details are [here]( https://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html)).  Be sure to note that the pipeline operator `%>%` is not the same as the assignment operator `->`. Here is an example using both:\n",
    "\n",
    "```R\n",
    "df %>% \n",
    "select(col1, col2) %>% \n",
    "mutate(col_add = col1 + col2) -> df_new\n",
    "```\n",
    "\n",
    "The code block above does the following:\n",
    "\n",
    "* line1: send the dataframe to the next function/line\n",
    "* line 2: select the two columns called \"col1\" and col2\" from df and send to the next function/line (some suggest that the pipeline operator be pronounced \"then\" when reading through the code).\n",
    "* line 3: take the new dataframe (just two columns) and create a new column called \"col_add\" as a sum of the two columns and assign to a dataframe called \"df_new\".\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "The best way to guide an analysis is to start with a question to answer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do rentals vary by day of the week?\n",
    "\n",
    "To answer this question, we need to examine the count of rentals \"cnt\" in groups defined by the day of the week \"weekday\" where Sunday is 0. We can then look at the average count of rentals by weekday. Then we can plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df %>%\n",
    "  group_by(weekday) %>%\n",
    "  summarize(avg = mean(cnt))  %>%\n",
    "  ggplot(aes(x=weekday, y=avg)) +\n",
    "  geom_col()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do rentals vary by season?\n",
    "\n",
    "To answer this question, we need to examine the count of rentals \"cnt\" in groups defined by the season \"season\" (1:spring, 2:summer, 3:fall, 4:winter). We can then look at the average count of rentals by season. Then we can plot the results.\n",
    "\n",
    "Notice, we used the same code from above, but changed our group_by variable and subsequent x axis.\n",
    "\n",
    "Experiment with different versions of dplyr functions to better understand how they work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df %>% \n",
    "  group_by(season) %>%\n",
    "  summarize(average = mean(cnt)) %>%\n",
    "  ggplot(aes(x=season, y=average)) +\n",
    "  geom_col()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do rentals vary by temperature?\n",
    "\n",
    "For this analysis, try to make a visualization that can describe the relationship between rentals and temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# explore rentals versus temperature here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conduct an end-to-end analysis!\n",
    "\n",
    "Next, we will conduct a simplified end-to-end analysis of a weather dataset.\n",
    "\n",
    "## [Please continue in the next notebook page by clicking here](Weather_Data_Analysis.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}