05_15_inference_a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/COGS108/Lectures-Fa23/blob/master/Week_05/05_15_inference_a.ipynb)\n",
    "\n",
    "# Inference: Linear Regression\n",
    "\n",
    "- **simple linear regression**\n",
    "    - effect size\n",
    "    - p-value\n",
    "    - interpretation\n",
    "- **multiple linear regression**\n",
    "    - confounding\n",
    "- working with **transformed data**\n",
    "\n",
    "\n",
    "\n",
    "To answer our quiz questions today please open this link in seperate browser tab or window:\n",
    "\n",
    "https://forms.gle/4mdyN7coMuurGR3H9\n",
    "\n",
    "We will be going back and forth from the notebook to this quiz several times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Import seaborn and apply its plotting styles\n",
    "import seaborn as sns\n",
    "sns.set(style=\"white\", font_scale=2)\n",
    "\n",
    "# import matplotlib\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "# set plotting size parameter\n",
    "plt.rcParams['figure.figsize'] = (17, 7)\n",
    "\n",
    "# Statmodels & patsy\n",
    "import patsy\n",
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from scipy.stats import skewnorm\n",
    "\n",
    "#improve resolution\n",
    "#comment this line if erroring on your machine/screen\n",
    "%config InlineBackend.figure_format ='retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Question\n",
    "\n",
    "Does Poverty Percentage affect Teen Birth Rate?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$H_o$: There is no relationship between poverty percentage and teen birth rate ($\\beta = 0$)\n",
    "\n",
    "$H_a$: There is a relationship between poverty percentage and teen birth rate ($\\beta \\ne 0$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in file; specify that it is tab-separated file\n",
    "df = pd.read_csv('https://raw.githubusercontent.com/shanellis/datasets/master/index.txt', sep='\\t')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assumptions of Linear Regression\n",
    "\n",
    "1. Linear relationship\n",
    "2. No multicollinearity\n",
    "3. No auto-correlation\n",
    "4. Homoscedasticity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.scatterplot(x='PovPct', y='TeenBrth', data=df, s=100)\n",
    "plt.xlabel('Poverty Percentage')\n",
    "plt.ylabel('Teen Birth Rate');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see there is some sort of relationship here, but we want to *measure that realtionship*. Given that the increase appears linear over time (Assumption #1), linear regreasssion allows us to do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplot(1, 2, 1)\n",
    "sns.distplot(df['PovPct'])\n",
    "plt.xlabel('Poverty Percentage')\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "sns.distplot(df['TeenBrth'])\n",
    "plt.xlabel('Teen Birth Rate');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$outcome = \\beta_0 + \\beta_1*predictor$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's be sure we're on the same page about what our outcome is. Here, we're intererested in whether **Poverty Percentage (predictor)** impacts **Teen Birth Rate (outcome)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import statsmodels.api as sm\n",
    "# import statsmodels.formula.api as smf\n",
    "\n",
    "# OLD SKOOL way to do this\n",
    "# We can specify our model matrix using `pastsy`.\n",
    "outcome, predictors = patsy.dmatrices('TeenBrth ~ PovPct', df)\n",
    "model = sm.OLS(outcome, predictors)\n",
    "\n",
    "\n",
    "\n",
    "#modern version\n",
    "model = smf.ols(formula='TeenBrth ~ PovPct', data=df)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, then we just have to fit the model and look at the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## fit the model\n",
    "results = model.fit()\n",
    "\n",
    "## look at the results\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clicker Question #1\n",
    "\n",
    "What is the effect size of the relationship between Poverty Percentage and Teen Birth Rate?\n",
    "\n",
    "- A) 15.67\n",
    "- B) 2.03\n",
    "- C) 4.032\n",
    "- D) 0.495"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a lot of information in there. Let's focus on the three pieces we discussed last lecture:\n",
    "\n",
    "- `coef` : $\\beta_1$ estimate explaining the effect size\n",
    "- `std err` : standard error \n",
    "- `P>|t|` : the p-value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$outcome = \\beta_0 + \\beta_1*predictor$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Teen Birth = 15.67 + 2.03  * Poverty Percentage $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the Poverty Percentage were 0, the Teen Birth Rate would be **15.67** (The Intercept, $\\beta_0$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For every 1 unit increase in Poverty Percentage, you expect to see a **2.03** unit increase in Teen Birth Rate (The effect size, $\\beta_1$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## look at the results\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clicker Question #2\n",
    "\n",
    "Which value represents the expected Teen Birth Rate if the Poverty Percentage were 0?\n",
    "\n",
    "- A) 15.67\n",
    "- B) 2.03\n",
    "- C) 4.032\n",
    "- D) 0.495"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the orginal data (as before)\n",
    "sns.scatterplot(x='PovPct', y='TeenBrth', alpha=0.3, data=df, s=100)\n",
    "\n",
    "# Generate and plot the model fit line\n",
    "xs = np.arange(df['PovPct'].min(), df['PovPct'].max())\n",
    "ys = 2.0255 * xs + 15.67\n",
    "plt.plot(xs, ys, '--k', linewidth=4, label='Model')\n",
    "\n",
    "plt.xlabel('Poverty Percentage')\n",
    "plt.ylabel('Teen Birth Rate')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.lmplot(x='PovPct', y='TeenBrth', data=df);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model (the line) mathematically describes the relationship between the data points, but it doesn't explain the relationship *perfectly*. (All models are wrong!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this line is drawn in the following way:\n",
    "- if you were to draw a perpendicular line from each point to the line and calculate that distance\n",
    "- if you were to sum the distance across all points\n",
    "- this line is the one that minimizes that sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clicker Question #3\n",
    "\n",
    "If I told you the Poverty Percentage of a state was 15, what would you estimate would be its Teen Birth Rate? \n",
    "- A) ~ 0\n",
    "- B) ~ 30\n",
    "- C) ~ 40\n",
    "- D) ~ 50\n",
    "- E) ~ 60"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Teen Birth = 15.67 + 2.03  * Poverty Percentage $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clicker Question #4\n",
    "\n",
    "Which of the following is a reasonable estimate (guess) for a state with a Poverty Percentage of 20?\n",
    "\n",
    "- A) 55.99\n",
    "- B) 56.27\n",
    "- C) 56.5\n",
    "- D) A - C\n",
    "- E) None of the above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimates\n",
    "\n",
    "If I told you a locations' Poverty Percentage, what would you guess its Teen Birth Rate would be?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Teen Birth = 15.67 + 2.03  * Poverty Percentage $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## if Poverty Percentage were 18\n",
    "birth_rate = 15.67 + 2.03 * 18\n",
    "birth_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## if Poverty Percentage were 12\n",
    "birth_rate = 15.67 + 2.03 * 12\n",
    "birth_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## look at the results one more time for the clicker question\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results.pvalues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clicker Question #5\n",
    "\n",
    "What is our conclusion from this analysis? (Question: Does Poverty Percentage affect Teen Birth Rate?)\n",
    "\n",
    "- A) Reject the null; There is no relationship between Poverty Percentage and Teen Birth Rate\n",
    "- B) Reject the null; There is a relationship between Poverty Percentage and Teen Birth Rate\n",
    "- C) Fail to reject the null; There is no relationship between Poverty Percentage and Teen Birth Rate\n",
    "- D) Fail to reject the null; There is a relationship between Poverty Percentage and Teen Birth Rate\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiple Linear Regression\n",
    "\n",
    "So far, we've only been looking at the relationship of one measure (Poverty Percentage) on Teen Birth. \n",
    "\n",
    "*But*, Poverty Percentage is likely *not* the only thing that affects Teen Birth Rate.\n",
    "\n",
    "We could imagine that Violent Crime rates in a location may affect both Poverty Percentage and could possibly affect Teen Birth Rate. (A confounder!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is where **multiple linear regression** is incredibly helpful. Multiple linear regression allows you to measure the effect of multiple predictors on an outcome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "mod = smf.ols(formula='TeenBrth ~ ViolCrime', data=df)\n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "mod = smf.ols(formula='TeenBrth ~ PovPct + ViolCrime', data=df)\n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clicker Question #6\n",
    "\n",
    "Which has a larger effect on Teen Birth Rate?\n",
    "\n",
    "- A) Poverty Percentage\n",
    "- B) Violent Crime\n",
    "- C) Effect is equal across all predictors\n",
    "- D) No predictors have an effect on outcome"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Teen Birth = 17.68 + (1.63  * Poverty Percentage) + (0.4 * Violent Crime) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the Poverty Percntage *and* Violent Crime were both 0, the Teen Birth Rate would be **17.68** (The Intercept)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holding Violent crime constant, for every 1 unit increase in Poverty Percentage, you expect to see a **1.63** unit increase in Teen Birth Rate (The effect size, $\\beta_1$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holding Poverty Percentage constant, for every 1 unit increase in Violent crime, you'd expect to see a **0.4** unit increase in Teen Birth Rate (The effect size, $\\beta_2$)\n",
    "\n",
    "\n",
    "### What about an interaction between our 2 variables?\n",
    "\n",
    "Sometimes relationships exist between two or more independet variables such that the slope of regression of independent variable #1 changes with the value of independet variable #2 and vice versa. That is, there is a multiplicative relationship between the variables.  This is an interaction term, and its specified by a * in the model equations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "mod = smf.ols(formula='TeenBrth ~ PovPct * ViolCrime', data=df)\n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Teen Birth = 0.82 + (2.4  * PP) + (3.4 * VC) + -0.14 * PP * VC $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## No Multiple Colinearity\n",
    "\n",
    "If we have two predictors, we'll want to consider how similar the values are between predictors (typically, before your generate the model, but we'll do it after for sake of argument today)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# relationship between predictors\n",
    "sns.scatterplot(x='PovPct', y='ViolCrime', data=df, s=100)\n",
    "plt.xlabel('Poverty Percentage')\n",
    "plt.ylabel('Violent Crime');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['ViolCrime'] > 50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outlier handling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing DC\n",
    "df = df[df['ViolCrime'] < 50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# relationship between predictors\n",
    "# after removing outlier\n",
    "sns.scatterplot(x='PovPct', y='ViolCrime', data=df, s=100)\n",
    "plt.xlabel('Poverty Percentage')\n",
    "plt.ylabel('Violent Crime');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run model\n",
    "\n",
    "mod = smf.ols(formula='TeenBrth ~ PovPct + ViolCrime', data=df)  \n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clicker Question #7\n",
    "\n",
    "After outlier removal, which has a larger effect on Teen Birth Rate?\n",
    "\n",
    "- A) Poverty Percentage\n",
    "- B) Violent Crime\n",
    "- C) Effect is equal across all predictors\n",
    "- D) No predictors have an effect on outcome"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holding Violent crime constant, for every 1 unit increase in Poverty Percentage, you expect to see a **1.19** unit increase in Teen Birth Rate (The effect size, $\\beta_1$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holding Poverty Percentage constant, for every 1 unit increase in Violent crime, you'd expect to see a **1.63** unit increase in Teen Birth Rate (The effect size, $\\beta_2$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Removing the DC outlier led to a different estimate for the effect of violent crime in these data. The inclusion or removal of data in your analyses can affect the results, so it's always important to carefully consider what question you're asking and what data you have. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multicolinearity returns\n",
    "Lets force a colinearity problem by making a synthetic variable to show you what to look out for...\n",
    "\n",
    "specifically note that \n",
    "1. condition number is large... that's diagnosing the issue\n",
    "2. coefficient on violent crime changes dramatically from before\n",
    "3. coefficient on psuedoVC is similar in magnitude to violent crime, robbing it of the explanatory power it deserves\n",
    "4. the sum of coefficients doesn't arrive at the total it was at in previous model... again robbing it of explanatroy power"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['psuedoVC'] = df['ViolCrime'] * 3.6 + 112 + 5*pd.np.random.rand(len(df))\n",
    "mod = smf.ols(formula='TeenBrth ~ PovPct + ViolCrime + psuedoVC', data=df)  \n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression with transformed data\n",
    "\n",
    "So far, we've been working with data that were approximately Normal and didn't require transformation. But that won't always be the case..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sleep = pd.read_csv('https://raw.githubusercontent.com/shanellis/datasets/master/msleep.csv')\n",
    "sleep.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we were interested in quantifying the **effect that REM sleep has on total sleep**?\n",
    "\n",
    "We may hypothesize that the more REM sleep an animal gets the less total sleep it needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(sleep['sleep_rem'][sleep['sleep_rem'].notnull()],hist=True)\n",
    "plt.xlabel('REM Sleep');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw previously that these data are skewed right, and discussed that transforming these data could help us use them in analysis..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.scatterplot(x='sleep_rem', y='sleep_total', data=sleep, s=100)\n",
    "plt.xlabel('REM Sleep')\n",
    "plt.ylabel('Total Sleep');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This relationship is not linear...so linear regression would not be appropriate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Log Transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add log-transofmed column to sleep_rem & brainwt\n",
    "sleep['sleep_rem10'] = np.log10(sleep['sleep_rem'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(sleep['sleep_rem10'][sleep['sleep_rem'].notnull()], hist=True)\n",
    "plt.xlabel('log10(REM Sleep)');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.scatterplot(x='sleep_rem10', y='sleep_total', data=sleep, s=100)\n",
    "plt.xlabel('log10(REM Sleep)')\n",
    "plt.ylabel('Total Sleep');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear regression with transformed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# carry out regression using log-transformed predictor\n",
    "\n",
    "mod = smf.ols(formula='sleep_total ~ sleep_rem10', data=sleep)\n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.lmplot(x='sleep_rem10',y='sleep_total',data=sleep)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.lmplot(x='sleep_rem',y='sleep_total',data=sleep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we see that the $\\beta_1$ estimate for `sleep_rem10` is 10.89....but remember that this value is on the log scale."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Total Sleep = 8.57 + 10.89  * log_{10}(REM Sleep) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To interpret this coefficient, we would say that\n",
    "\n",
    "a 1 unit increase in REM sleep, Total Sleep increases by $\\approx$ $10.89/100$ units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# interpreting a log transformed coefficient\n",
    "10.89 / 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, for each 1 hour increase in REM Sleep, Total Sleep increases by 0.11 hours. \n",
    "\n",
    "That's not a large effect, but it is different than what we hypothesized at the beginning!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More on interpretation of log transformed variables can be read [here](https://www.cscu.cornell.edu/news/statnews/stnews83.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why is log-transformation not math trickery?\n",
    "\n",
    "First, some reading on data transformation can be found [here](http://fmwww.bc.edu/repec/bocode/t/transint.html). It's not beautifully formatted, but I think it is written in a more-straightforward manner than some [other](https://stats.stackexchange.com/questions/4831/regression-transforming-variables/4833#4833) [places](https://stats.stackexchange.com/questions/298/in-linear-regression-when-is-it-appropriate-to-use-the-log-of-an-independent-va) [online](https://www.researchgate.net/post/Why_do_we_do_transformation_before_data_analysis).\n",
    "\n",
    "But a basic understanding can be had in just a few bullet points\n",
    "- measurement scales may be arbitrary. there's nothing to say log transforms aren't a better measurement scale than the original\n",
    "- you are already familiar with many logrithmic measurement scales: decibels and Richter scale for measuring earthqueakes are two examples\n",
    "- lots of things are power-law distributed or close to power-law distributed... the frequency of words in a language, human vision and auditory abilities in perception, the number of social media followers people have. \n",
    "- log transforms make a process that had been multiplicative or power law like ($a*b^n$) into an additive process ($a+n*b$) which really helps when you're trying to apply linear regression for obvious reasons\n",
    "- log transforms can reduce heteroscadacity and skewness, which may make the data amenable to linear regression when it hadn't been before"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to remember is there is (in most cases) nothing special about how the data are originally expressed.\n",
    "\n",
    "In our example above, there is nothing about \"hours\" as the unit that was chosen that makes these data \"correct\".\n",
    "\n",
    "So, while it _feels_ like data transformation is trickery, our initial unit of hours is...in some ways arbitrary and something that we chose.\n",
    "\n",
    "This is where we'll start with our argument that it's ok to transform (or think of it as *re-express*) our data\n",
    "so that it can be (still-accurately) used with well-studied models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the point a little more concretely, pH (measurement of acidity) is measured on the log scale. It _could_ be measured (transformed) off of the log scale. Those measurements would still explain a compound's acidity...it would just be on a different scale."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words:\n",
    "    \n",
    "> \"Transformations are needed because there is no guarantee that the world works on the scales it happens to be measured on.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What *does* differ however, is the interprtation. Linear scales tell us absolute change, while logarithmic scales tell us relative change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dat = skewnorm.rvs(5, 1, size=1000)\n",
    "sns.distplot(dat, kde=False, bins=20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dat_log = np.log10(dat)\n",
    "sns.distplot(dat_log, kde=False, bins=20);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#original value\n",
    "dat[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log 10 transformed value\n",
    "dat_log[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# math that is actually happening\n",
    "10.0*dat_log[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# linear scale tells you asolute \n",
    "# difference between two points\n",
    "dat[1] - dat[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log scale tells you relative \n",
    "# difference between two points\n",
    "dat_log[1] - dat_log[0]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}