ds-for-telco.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "   # Load the data\n",
    "\n",
    "We need to load data from a file in to a Spark DataFrame.\n",
    "Each row is an observed customer, and each column contains\n",
    "attributes of that customer.\n",
    "\n",
    "[Data from UCI data set repo, hosted by SGI](https://www.sgi.com/tech/mlc/db/churn.all)\n",
    "\n",
    "    Fields:\n",
    "\n",
    "    state: discrete.\n",
    "    account length: numeric.\n",
    "    area code: numeric.\n",
    "    phone number: discrete.\n",
    "    international plan: discrete.\n",
    "    voice mail plan: discrete.\n",
    "    number vmail messages: numeric.\n",
    "    total day minutes: numeric.\n",
    "    total day calls: numeric.\n",
    "    total day charge: numeric.\n",
    "    total eve minutes: numeric.\n",
    "    total eve calls: numeric.\n",
    "    total eve charge: numeric.\n",
    "    total night minutes: numeric.\n",
    "    total night calls: numeric.\n",
    "    total night charge: numeric.\n",
    "    total intl minutes: numeric.\n",
    "    total intl calls: numeric.\n",
    "    total intl charge: numeric.\n",
    "    number customer service calls: numeric.\n",
    "    churned: discrete.\n",
    "\n",
    "'Numeric' and 'discrete' do not adequately describe the fundamental differencecs in the attributes.\n",
    "\n",
    "Area codes are considered numeric, but they a better thought of as a categorical variable. This is because attributes that are really numeric features have a reasonable concept of distance between points. Area codes do not fall into this cateogory. They can have a small distance |area_code_1 - area_code_2| but that distance doesn't correspond to a similarity in area codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.sql import SQLContext\n",
    "from pyspark.sql.types import *\n",
    "\n",
    "sqlContext = SQLContext(sc)\n",
    "schema = StructType([ \\\n",
    "    StructField(\"state\", StringType(), True), \\\n",
    "    StructField(\"account_length\", DoubleType(), True), \\\n",
    "    StructField(\"area_code\", StringType(), True), \\\n",
    "    StructField(\"phone_number\", StringType(), True), \\\n",
    "    StructField(\"intl_plan\", StringType(), True), \\\n",
    "    StructField(\"voice_mail_plan\", StringType(), True), \\\n",
    "    StructField(\"number_vmail_messages\", DoubleType(), True), \\\n",
    "    StructField(\"total_day_minutes\", DoubleType(), True), \\\n",
    "    StructField(\"total_day_calls\", DoubleType(), True), \\\n",
    "    StructField(\"total_day_charge\", DoubleType(), True), \\\n",
    "    StructField(\"total_eve_minutes\", DoubleType(), True), \\\n",
    "    StructField(\"total_eve_calls\", DoubleType(), True), \\\n",
    "    StructField(\"total_eve_charge\", DoubleType(), True), \\\n",
    "    StructField(\"total_night_minutes\", DoubleType(), True), \\\n",
    "    StructField(\"total_night_calls\", DoubleType(), True), \\\n",
    "    StructField(\"total_night_charge\", DoubleType(), True), \\\n",
    "    StructField(\"total_intl_minutes\", DoubleType(), True), \\\n",
    "    StructField(\"total_intl_calls\", DoubleType(), True), \\\n",
    "    StructField(\"total_intl_charge\", DoubleType(), True), \\\n",
    "    StructField(\"number_customer_service_calls\", DoubleType(), True), \\\n",
    "    StructField(\"churned\", StringType(), True)])\n",
    "\n",
    "churn_data = sqlContext.read \\\n",
    "    .format('com.databricks.spark.csv') \\\n",
    "    .load('churn.all', schema = schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Basic DataFrame operations\n",
    "\n",
    "Dataframes essentially allow you to express sql-like statements. We can filter, count, and so on. [DataFrame Operations documentation.](http://spark.apache.org/docs/latest/sql-programming-guide.html#dataframe-operations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "count = churn_data.count()\n",
    "voice_mail_plans = churn_data.filter(churn_data.voice_mail_plan == \" yes\").count()\n",
    "\n",
    "\"%d, %d\" % (count, voice_mail_plans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Exercise: Basic DataFrame operations\n",
    "\n",
    "How many customers have one or more customer service calls? How many have more than one? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Feature Visualization\n",
    "\n",
    "The data vizualization workflow for large data sets is usually:\n",
    "\n",
    "* Sample data so it fits in memory on a single machine.\n",
    "* Examine single variable distributions.\n",
    "* Examine joint distributions and correlations.\n",
    "* Look for other types of relationships.\n",
    "\n",
    "[DataFrame#sample() documentation](http://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc1-docs/api/python/pyspark.sql.html#pyspark.sql.DataFrame.sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sample_data = churn_data.sample(False, 0.5, 83).toPandas()\n",
    "sample_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## DataTypes\n",
    "\n",
    "![datatypes where continuous and integer are subtypes of numeric data and distinct from the categorical type](slides/datatypes.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The type of visualization we do depends on the data type, so lets define what columns have different properties first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "numeric_cols = [\"account_length\", \"number_vmail_messages\", \"total_day_minutes\",\n",
    "                \"total_day_calls\", \"total_day_charge\", \"total_eve_minutes\",\n",
    "                \"total_eve_calls\", \"total_eve_charge\", \"total_night_minutes\",\n",
    "                \"total_night_calls\", \"total_intl_minutes\", \"total_intl_calls\",\n",
    "                \"total_intl_charge\"]\n",
    "\n",
    "categorical_cols = [\"state\", \"international_plan\", \"voice_mail_plan\", \"area_code\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Seaborn\n",
    "\n",
    "Seaborn is a library for statistical visualization that is built on matplotlib.\n",
    "\n",
    "Great support for:\n",
    "* plotting distributions\n",
    "* regression analyses\n",
    "* plotting with categorical splitting\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "![Screen capture of Seaborn home page.](slides/seaborn_home_page.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Feature Distributions\n",
    "\n",
    "We want to examine the distribution of our features, so start with them one at a time.\n",
    "\n",
    "Seaborn has a standard function called [dist()](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.distplot.html#seaborn.distplot) that allows us to easily examine the distribution of a column of a pandas dataframe or a numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sb\n",
    "\n",
    "sb.distplot(sample_data['number_customer_service_calls'], kde=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can examine feature differences in the distribution of our features when we condition (split) our data in whether they churned or not.\n",
    "\n",
    "[BoxPlot docs](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.boxplot.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sb.boxplot(x=\"churned\", y=\"number_customer_service_calls\", data=sample_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Joint Distributions\n",
    "\n",
    "Looking at joint distributions of data can also tell us a lot, particularly about redundant features. [Seaborn's PairPlot](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.pairplot.html#seaborn.pairplot) let's us look at joint distributions for many variables at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "example_numeric_data = sample_data[[\"total_intl_minutes\", \"total_intl_calls\",\n",
    "                                       \"total_intl_charge\", \"churned\"]]\n",
    "sb.pairplot(example_numeric_data, hue=\"churned\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Clearly, there are some strong linear relationships between some variables, let's get a general impression of the correlations between variables by using [Seaborn's heatmap functionality.](http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.heatmap.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "corr = sample_data[[\"account_length\", \"number_vmail_messages\", \"total_day_minutes\",\n",
    "                    \"total_day_calls\", \"total_day_charge\", \"total_eve_minutes\",\n",
    "                    \"total_eve_calls\", \"total_eve_charge\", \"total_night_minutes\",\n",
    "                    \"total_night_calls\", \"total_intl_minutes\", \"total_intl_calls\",\n",
    "                    \"total_intl_charge\"]].corr()\n",
    "\n",
    "sb.heatmap(corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's generate a pair plot for all numerical variables that we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "reduced_numeric_cols = [\"account_length\", \"number_vmail_messages\", \"total_day_calls\",\n",
    "                        \"total_day_charge\", \"total_eve_calls\", \"total_eve_charge\",\n",
    "                        \"total_night_calls\", \"total_intl_calls\", \"total_intl_charge\"]\n",
    "\n",
    "sb.pairplot(sample_data[reduced_numeric_cols + ['churned']], hue=\"churned\", palette='Paired')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Build a classification model using MLLib\n",
    "\n",
    "We want to build a predictive model.\n",
    "\n",
    "The overall process:\n",
    "\n",
    "![Model building diagraming showing input data split into train and test data. The training data is aligned with a feature extraction and model training set, which outputs a fitter model. The testing data alignes with a feature extraction, scoring/model application, and model evaluation step.](slides/model flow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Feature Extraction and Model Training\n",
    "\n",
    "We need to:\n",
    "* Code features that are not already numeric\n",
    "* Gather all features we need into a single column in the DataFrame.\n",
    "* Split labeled data into training and testing set\n",
    "* Fit the model to the training data.\n",
    "\n",
    "## Feature Extraction\n",
    "We need to define our input features.\n",
    "\n",
    "[PySpark Pipeline Docs](https://spark.apache.org/docs/1.5.0/api/python/pyspark.ml.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import StringIndexer\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "\n",
    "label_indexer = StringIndexer(inputCol = 'churned', outputCol = 'label')\n",
    "plan_indexer = StringIndexer(inputCol = 'intl_plan', outputCol = 'intl_plan_indexed')\n",
    "\n",
    "assembler = VectorAssembler(\n",
    "    inputCols = ['intl_plan_indexed'] + reduced_numeric_cols,\n",
    "    outputCol = 'features')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Model Training\n",
    "\n",
    "We can now define our classifier and pipeline. With this done, we can split our labeled data in train and test sets and fit a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.classification import DecisionTreeClassifier\n",
    "\n",
    "classifier = DecisionTreeClassifier(labelCol = 'label', featuresCol = 'features')\n",
    "\n",
    "pipeline = Pipeline(stages=[plan_indexer, label_indexer, assembler, classifier])\n",
    "\n",
    "(train, test) = churn_data.randomSplit([0.7, 0.3])\n",
    "model = pipeline.fit(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Model Evaluation\n",
    "\n",
    "The most important question to ask:\n",
    "    \n",
    "    Is my predictor better than random guessing?\n",
    "\n",
    "How do we quantify that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Measure the area under the ROC curve, abreviated to AUROC.\n",
    "\n",
    "![example of an roc curve](slides/roc_curve.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "\n",
    "predictions = model.transform(test)\n",
    "evaluator = BinaryClassificationEvaluator()\n",
    "auroc = evaluator.evaluate(predictions, {evaluator.metricName: \"areaUnderROC\"})\n",
    "aupr = evaluator.evaluate(predictions, {evaluator.metricName: \"areaUnderPR\"})\n",
    "\"The AUROC is %s and the AUPR is %s.\" % (auroc, aupr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Exercise: Fit a RandomForestClassifier\n",
    "\n",
    "Fit a random forest classifier to the data. Try experimenting with different values of the `maxDepth`, `numTrees`, and `entropy` parameters to see which gives the best classification performance. Do the settings that give the best classification performance on the training set also give the best classification performance on the test set?\n",
    "\n",
    "Have a look at the [documentation](http://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.classification.RandomForestClassifier]documentation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}