Weather Data Clustering using k-Means.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:2.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Clustering with scikit-learn\n",
    "\n",
    "<br><br></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will learn how to perform k-means lustering using scikit-learn in Python. \n",
    "\n",
    "We will use cluster analysis to generate a big picture model of the weather at a local station using a minute-graunlarity data. In this dataset, we have in the order of millions records. How do we create 12 clusters our of them?\n",
    "\n",
    "**NOTE:** The dataset we will use is in a large CSV file called *minute_weather.csv*. Please download it into the *weather* directory in your *Week-7-MachineLearning* folder. The download link is: https://drive.google.com/open?id=0B8iiZ7pSaSFZb3ItQ1l4LWRMTjg "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Importing the Necessary Libraries<br></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.cluster import KMeans\n",
    "import utils\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from itertools import cycle, islice\n",
    "import matplotlib.pyplot as plt\n",
    "from pandas.tools.plotting import parallel_coordinates\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Creating a Pandas DataFrame from a CSV file<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('./weather/minute_weather.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\">Minute Weather Data Description</p>\n",
    "<br>\n",
    "The **minute weather dataset** comes from the same source as the daily weather dataset that we used in the decision tree based classifier notebook. The main difference between these two datasets is that the minute weather dataset contains raw sensor measurements captured at one-minute intervals. Daily weather dataset instead contained processed and well curated data. The data is in the file **minute_weather.csv**, which is a comma-separated file.\n",
    "\n",
    "As with the daily weather data, this data comes from a weather station located in San Diego, California. The weather station is equipped with sensors that capture weather-related measurements such as air temperature, air pressure, and relative humidity. Data was collected for a period of three years, from September 2011 to September 2014, to ensure that sufficient data for different seasons and weather conditions is captured.\n",
    "\n",
    "Each row in **minute_weather.csv** contains weather data captured for a one-minute interval. Each row, or sample, consists of the following variables:\n",
    "\n",
    "* **rowID:** \tunique number for each row\t(*Unit: NA*)\n",
    "* **hpwren_timestamp:**\ttimestamp of measure\t(*Unit: year-month-day hour:minute:second*)\n",
    "* **air_pressure:** air pressure measured at the timestamp\t(*Unit: hectopascals*)\n",
    "* **air_temp:**\tair temperature measure at the timestamp\t(*Unit: degrees Fahrenheit*)\n",
    "* **avg_wind_direction:**\twind direction averaged over the minute before the timestamp\t(*Unit: degrees, with 0 means coming from the North, and increasing clockwise*)\n",
    "* **avg_wind_speed:**\twind speed averaged over the minute before the timestamp\t(*Unit: meters per second*)\n",
    "* **max_wind_direction:**\thighest wind direction in the minute before the timestamp\t(*Unit: degrees, with 0 being North and increasing clockwise*)\n",
    "* **max_wind_speed:**\thighest wind speed in the minute before the timestamp\t(*Unit: meters per second*)\n",
    "* **min_wind_direction:**\tsmallest wind direction in the minute before the timestamp\t(*Unit: degrees, with 0 being North and inceasing clockwise*)\n",
    "* **min_wind_speed:**\tsmallest wind speed in the minute before the timestamp\t(*Unit: meters per second*)\n",
    "* **rain_accumulation:**\tamount of accumulated rain measured at the timestamp\t(*Unit: millimeters*)\n",
    "* **rain_duration:**\tlength of time rain has fallen as measured at the timestamp\t(*Unit: seconds*)\n",
    "* **relative_humidity:**\trelative humidity measured at the timestamp\t(*Unit: percent*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Data Sampling<br></p>\n",
    "\n",
    "Lots of rows, so let us sample down by taking every 10th row. <br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sampled_df = data[(data['rowID'] % 10) == 0]\n",
    "sampled_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Statistics\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sampled_df.describe().transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sampled_df[sampled_df['rain_accumulation'] == 0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sampled_df[sampled_df['rain_duration'] == 0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Drop all the Rows with Empty rain_duration and rain_accumulation\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "del sampled_df['rain_accumulation']\n",
    "del sampled_df['rain_duration']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rows_before = sampled_df.shape[0]\n",
    "sampled_df = sampled_df.dropna()\n",
    "rows_after = sampled_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "How many rows did we drop ?\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rows_before - rows_after"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sampled_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Select Features of Interest for Clustering\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features = ['air_pressure', 'air_temp', 'avg_wind_direction', 'avg_wind_speed', 'max_wind_direction', \n",
    "        'max_wind_speed','relative_humidity']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "select_df = sampled_df[features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "select_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "select_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Scale the Features using StandardScaler\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = StandardScaler().fit_transform(select_df)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Use k-Means Clustering\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=12)\n",
    "model = kmeans.fit(X)\n",
    "print(\"model\\n\", model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "What are the centers of 12 clusters we formed ?\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "centers = model.cluster_centers_\n",
    "centers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:2.75em;color:purple; font-style:bold\"><br>\n",
    "\n",
    "Plots\n",
    "<br><br></p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first create some utility functions which will help us in plotting graphs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Function that creates a DataFrame with a column for Cluster Number\n",
    "\n",
    "def pd_centers(featuresUsed, centers):\n",
    "\tcolNames = list(featuresUsed)\n",
    "\tcolNames.append('prediction')\n",
    "\n",
    "\t# Zip with a column called 'prediction' (index)\n",
    "\tZ = [np.append(A, index) for index, A in enumerate(centers)]\n",
    "\n",
    "\t# Convert to pandas data frame for plotting\n",
    "\tP = pd.DataFrame(Z, columns=colNames)\n",
    "\tP['prediction'] = P['prediction'].astype(int)\n",
    "\treturn P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Function that creates Parallel Plots\n",
    "\n",
    "def parallel_plot(data):\n",
    "\tmy_colors = list(islice(cycle(['b', 'r', 'g', 'y', 'k']), None, len(data)))\n",
    "\tplt.figure(figsize=(15,8)).gca().axes.set_ylim([-3,+3])\n",
    "\tparallel_coordinates(data, 'prediction', color = my_colors, marker='o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "P = pd_centers(features, centers)\n",
    "P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dry Days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "parallel_plot(P[P['relative_humidity'] < -0.5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warm Days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "parallel_plot(P[P['air_temp'] > 0.5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cool Days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "parallel_plot(P[(P['relative_humidity'] > 0.5) & (P['air_temp'] < 0.5)])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}