data_cleaning_exercises.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "[YCMI_CBDS Summer Course] data-cleaning-exercises",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yxvkM7TcSr-i",
        "colab_type": "text"
      },
      "source": [
        "Let's go ahead and setup our usual libraries:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zkscuS3gSsow",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import plotnine as p9"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O3X9ZSINnnPI",
        "colab_type": "text"
      },
      "source": [
        "And the connection to Google Drive to get the course data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iW2VHPq0nqZF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WKsMHKQdQeoa",
        "colab_type": "text"
      },
      "source": [
        "### Tidy data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "85Ss4Q6Xn2H3",
        "colab_type": "text"
      },
      "source": [
        "Read some of the early Connecticut daily coronavirus case report data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yoixxQV5QjvK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "coronavirus = pd.read_csv('/content/gdrive/My Drive/[YCMI_CBDS Summer Course] Data/ct-coronavirus-counts.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zWuLngbmoO45",
        "colab_type": "text"
      },
      "source": [
        "Preview the data by checking the `.head()`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q-PtQVr2oRdM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6JI4tvcxqEZW",
        "colab_type": "text"
      },
      "source": [
        "**IS THIS TIDY DATA? WHY OR WHY NOT?**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fv9eapzuovQ3",
        "colab_type": "text"
      },
      "source": [
        "The data here is based on the total counts as of the state's daily data releases. Changes in the number of reported cases do not necessarily correspond to the date people got sick or the date they were tested."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5oHfVo2vofqr",
        "colab_type": "text"
      },
      "source": [
        "Use `pd.to_datetime` to set the `Date` column to date objects."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TjvvMxlEopVq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tFQaCzyuqLkk",
        "colab_type": "text"
      },
      "source": [
        "Plot the number of cases in New Haven over time. Hint: you can rotate the x-axis labels by adding e.g. `p9.theme(axis_text_x=p9.element_text(angle=20))`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9cDA8-KvqQd_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "12BPHwFMtG1T",
        "colab_type": "text"
      },
      "source": [
        "Switch to a semi-log-y graph by adding a scale transformation of `log10` to the y-axis; i.e. add `p9.scale_y_continuous(trans='log10')`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NYXh4pWYtSy5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zh2GPSSntzc6",
        "colab_type": "text"
      },
      "source": [
        "Notice that you got a runtime warning about log10 and the number 0. This is because at the beginning of our dataset there were 0 cases in Hamden, and you can't display 0 on a log graph (why?).\n",
        "\n",
        "The curve appears to be close to a straight line between April 5, 2020 and \n",
        "April 17, 2020. **What would a true straight line mean?**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gUc7yfrpvNLI",
        "colab_type": "text"
      },
      "source": [
        "**How hard would it be to add curves for all the other cities in our dataset? What makes it hard?**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bgbfMDZxuLSF",
        "colab_type": "text"
      },
      "source": [
        "Use `pd.melt` to turn the dataset into tidy data with three columns: `Date`, `City`, `Cases`. Verify that you did what you think you did.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ymFYMpwxsfqx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JRXFnmZqvlsl",
        "colab_type": "text"
      },
      "source": [
        "Now using this tidy version of the dataset, plot all the cities on the same graph (with a semi-log-y axis). You can do this in a single line not much longer than the line for plotting just New Haven."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "EmkhkmRdv6Zm",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ihNs7Audwm1X",
        "colab_type": "text"
      },
      "source": [
        "Finally, add markers in addition to the lines to highlight the actual data points."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y3M2gHPOwybE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOLcfV9nQkmF",
        "colab_type": "text"
      },
      "source": [
        "### Missing data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0vcI29JsTKkO",
        "colab_type": "text"
      },
      "source": [
        "Let's load the \"Pima Indians Diabetes Database\" dataset from the web; this data comes from the National Institute of Diabetes and Digestive and Kidney Diseases and consists solely of women over age 21.\n",
        "\n",
        "This data was used in an early work with neural networks to analyze diabetes data: Smith, J. W., Everhart, J. E., Dickson, W. C., Knowler, W. C., & Johannes, R. S. (1988, November). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Annual Symposium on Computer Application in Medical Care (p. 261). American Medical Informatics Association."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cq5pYJe-Qvc5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "diabetes = pd.read_csv('https://raw.githubusercontent.com/susanli2016/Machine-Learning-with-Python/master/diabetes.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "awlwN8XnTRrC",
        "colab_type": "text"
      },
      "source": [
        "Examine the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TfcWtE10TOOu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GcqprginTVPr",
        "colab_type": "text"
      },
      "source": [
        "Verify that there are no columns that contain null values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q71A03jUTa2b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FkHvJ6bWTb58",
        "colab_type": "text"
      },
      "source": [
        "In some but not all columns in this particular dataset, values of 0 indicate missing data. (Note: insulin measurements here are serum insulin levels and should never be 0.) Replace the zero values with `pd.NA` to indicate that the data is missing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aItEoBwLd7zL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6u0r2wydestJ",
        "colab_type": "text"
      },
      "source": [
        "How many missing values were found in each column?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2zcY2tVveu4r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rdcKkyxFg-nx",
        "colab_type": "text"
      },
      "source": [
        "What is the mean `Age`? How does the mean `Age` for participants with a `SkinThickness` (Triceps skin fold thickness in mm) value compare to those without a `SkinThickness` value?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PCIScZbEh1QA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5b_44ixRfmGm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VOAayBwdh8-T",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VNtY-SlwizkA",
        "colab_type": "text"
      },
      "source": [
        "Make a violin plot comparing the distribution of ages for people who do and do not have `SkinThickness` measurements. To make this easier, it may help to add a `hasThickness` column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vVFq4cdnf-5s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8EYRaOpXjWGn",
        "colab_type": "text"
      },
      "source": [
        "What fraction of people under age 25 have `SkinThickness` measurements? What about people over age 50?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OOHuXkCNkEpa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SbpmTCH_kX3u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KOlvOaZHj__x",
        "colab_type": "text"
      },
      "source": [
        "Without attempting any further statistical tests, do you think the the missing values of `Skinthickness` are MAR, MCAR, or MNAR?"
      ]
    }
  ]
}