notebooks/17.In_Depth-Linear_Models.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Linear models\n",
    "Linear models are useful when little data is available or for very large feature spaces as in text classification. In addition, they form a good case study for regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Linear models for regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "All linear models for regression learn a coefficient parameter ``coef_`` and an offset ``intercept_`` to make predictions using a linear combination of features:\n",
    "\n",
    "```\n",
    "y_pred = x_test[0] * coef_[0] + ... + x_test[n_features-1] * coef_[n_features-1] + intercept_\n",
    "```\n",
    "\n",
    "The difference between the linear models for regression is what kind of restrictions or penalties are put on ``coef_`` as regularization , in addition to fitting the training data well.\n",
    "The most standard linear model is the 'ordinary least squares regression', often simply called 'linear regression'. It doesn't put any additional restrictions on ``coef_``, so when the number of features is large, it becomes ill-posed and the model overfits.\n",
    "\n",
    "Let us generate a simple simulation, to see the behavior of these models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_regression\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X, y, true_coefficient = make_regression(n_samples=200, n_features=30, n_informative=10, noise=100, coef=True, random_state=5)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=5, train_size=60, test_size=140)\n",
    "print(X_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Linear Regression\n",
    "\n",
    "$$ \\text{min}_{w, b} \\sum_i || w^\\mathsf{T}x_i + b  - y_i||^2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "linear_regression = LinearRegression().fit(X_train, y_train)\n",
    "print(\"R^2 on training set: %f\" % linear_regression.score(X_train, y_train))\n",
    "print(\"R^2 on test set: %f\" % linear_regression.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "print(r2_score(np.dot(X, true_coefficient), y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 5))\n",
    "coefficient_sorting = np.argsort(true_coefficient)[::-1]\n",
    "plt.plot(true_coefficient[coefficient_sorting], \"o\", label=\"true\")\n",
    "plt.plot(linear_regression.coef_[coefficient_sorting], \"o\", label=\"linear regression\")\n",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "def plot_learning_curve(est, X, y):\n",
    "    training_set_size, train_scores, test_scores = learning_curve(est, X, y, train_sizes=np.linspace(.1, 1, 20))\n",
    "    estimator_name = est.__class__.__name__\n",
    "    line = plt.plot(training_set_size, train_scores.mean(axis=1), '--', label=\"training scores \" + estimator_name)\n",
    "    plt.plot(training_set_size, test_scores.mean(axis=1), '-', label=\"test scores \" + estimator_name, c=line[0].get_color())\n",
    "    plt.xlabel('Training set size')\n",
    "    plt.legend(loc='best')\n",
    "    plt.ylim(-0.1, 1.1)\n",
    "\n",
    "plt.figure()    \n",
    "plot_learning_curve(LinearRegression(), X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Ridge Regression (L2 penalty)\n",
    "\n",
    "**The Ridge estimator** is a simple regularization (called l2 penalty) of the ordinary LinearRegression. In particular, it has the benefit of being not computationally more expensive than the ordinary least square estimate.\n",
    "\n",
    "$$ \\text{min}_{w,b}  \\sum_i || w^\\mathsf{T}x_i + b  - y_i||^2  + \\alpha ||w||_2^2$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The amount of regularization is set via the `alpha` parameter of the Ridge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "ridge_models = {}\n",
    "training_scores = []\n",
    "test_scores = []\n",
    "\n",
    "for alpha in [100, 10, 1, .01]:\n",
    "    ridge = Ridge(alpha=alpha).fit(X_train, y_train)\n",
    "    training_scores.append(ridge.score(X_train, y_train))\n",
    "    test_scores.append(ridge.score(X_test, y_test))\n",
    "    ridge_models[alpha] = ridge\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(training_scores, label=\"training scores\")\n",
    "plt.plot(test_scores, label=\"test scores\")\n",
    "plt.xticks(range(4), [100, 10, 1, .01])\n",
    "plt.xlabel('alpha')\n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 5))\n",
    "plt.plot(true_coefficient[coefficient_sorting], \"o\", label=\"true\", c='b')\n",
    "\n",
    "for i, alpha in enumerate([100, 10, 1, .01]):\n",
    "    plt.plot(ridge_models[alpha].coef_[coefficient_sorting], \"o\", label=\"alpha = %.2f\" % alpha, c=plt.cm.viridis(i / 3.))\n",
    "    \n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Tuning alpha is critical for performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plot_learning_curve(LinearRegression(), X, y)\n",
    "plot_learning_curve(Ridge(alpha=10), X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Lasso (L1 penalty)\n",
    "**The Lasso estimator** is useful to impose sparsity on the coefficient. In other words, it is to be prefered if we believe that many of the features are not relevant. This is done via the so-called l1 penalty.\n",
    "\n",
    "$$ \\text{min}_{w, b} \\sum_i \\frac{1}{2} || w^\\mathsf{T}x_i + b  - y_i||^2  + \\alpha ||w||_1$$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "lasso_models = {}\n",
    "training_scores = []\n",
    "test_scores = []\n",
    "\n",
    "for alpha in [30, 10, 1, .01]:\n",
    "    lasso = Lasso(alpha=alpha).fit(X_train, y_train)\n",
    "    training_scores.append(lasso.score(X_train, y_train))\n",
    "    test_scores.append(lasso.score(X_test, y_test))\n",
    "    lasso_models[alpha] = lasso\n",
    "plt.figure()\n",
    "plt.plot(training_scores, label=\"training scores\")\n",
    "plt.plot(test_scores, label=\"test scores\")\n",
    "plt.xticks(range(4), [30, 10, 1, .01])\n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 5))\n",
    "plt.plot(true_coefficient[coefficient_sorting], \"o\", label=\"true\", c='b')\n",
    "\n",
    "for i, alpha in enumerate([30, 10, 1, .01]):\n",
    "    plt.plot(lasso_models[alpha].coef_[coefficient_sorting], \"o\", label=\"alpha = %.2f\" % alpha, c=plt.cm.viridis(i / 3.))\n",
    "    \n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 5))\n",
    "plot_learning_curve(LinearRegression(), X, y)\n",
    "plot_learning_curve(Ridge(alpha=10), X, y)\n",
    "plot_learning_curve(Lasso(alpha=10), X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Instead of picking Ridge *or* Lasso, you can also use [ElasticNet](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html), which uses both forms of regularization and provides a parameter to assign a weighting between them. ElasticNet typically performs the best amongst these models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Linear models for classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "All linear models for classification learn a coefficient parameter ``coef_`` and an offset ``intercept_`` to make predictions using a linear combination of features:\n",
    "```\n",
    "y_pred = x_test[0] * coef_[0] + ... + x_test[n_features-1] * coef_[n_features-1] + intercept_ > 0\n",
    "```\n",
    "\n",
    "As you can see, this is very similar to regression, only that a threshold at zero is applied.\n",
    "\n",
    "Again, the difference between the linear models for classification what kind of regularization is put on ``coef_`` and ``intercept_``, but there are also minor differences in how the fit to the training set is measured (the so-called loss function).\n",
    "\n",
    "The two most common models for linear classification are the linear SVM as implemented in LinearSVC and LogisticRegression.\n",
    "\n",
    "A good intuition for regularization of linear classifiers is that with high regularization, it is enough if most of the points are classified correctly. But with less regularization, more importance is given to each individual data point.\n",
    "This is illustrated using an linear SVM with different values of ``C`` below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### The influence of C in LinearSVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In LinearSVC, the `C` parameter controls the regularization within the model.\n",
    "\n",
    "Lower `C` entails more regularization and simpler models, whereas higher `C` entails less regularization and more influence from individual data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from figures import plot_linear_svc_regularization\n",
    "plot_linear_svc_regularization()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Similar to the Ridge/Lasso separation, you can set the `penalty` parameter to 'l1' to enforce sparsity of the coefficients (similar to Lasso) or 'l2' to encourage smaller coefficients (similar to Ridge)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Multi-class linear classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_blobs\n",
    "plt.figure()\n",
    "X, y = make_blobs(random_state=42)\n",
    "plt.figure(figsize=(8, 8))\n",
    "plt.scatter(X[:, 0], X[:, 1], c=plt.cm.tab10(y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "linear_svm = LinearSVC().fit(X, y)\n",
    "print(linear_svm.coef_.shape)\n",
    "print(linear_svm.intercept_.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 8))\n",
    "plt.scatter(X[:, 0], X[:, 1], c=plt.cm.tab10(y))\n",
    "line = np.linspace(-15, 15)\n",
    "for coef, intercept in zip(linear_svm.coef_, linear_svm.intercept_):\n",
    "    plt.plot(line, -(line * coef[0] + intercept) / coef[1])\n",
    "plt.ylim(-10, 15)\n",
    "plt.xlim(-10, 8);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Points are classified in a one-vs-rest fashion (aka one-vs-all), where we assign a test point to the class whose model has the highest confidence (in the SVM case, highest distance to the separating hyperplane) for the test point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <b>EXERCISE</b>:\n",
    "     <ul>\n",
    "      <li>\n",
    "      Use LogisticRegression to classify the digits data set, and grid-search the C parameter.\n",
    "      </li>\n",
    "      <li>\n",
    "      How do you think the learning curves above change when you increase or decrease alpha?\n",
    "Try changing the alpha parameter in ridge and lasso, and see if your intuition was correct.\n",
    "      </li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "digits = load_digits()\n",
    "X_digits, y_digits = digits.data, digits.target\n",
    "\n",
    "# split the dataset, apply grid-search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/17A_logreg_grid.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/17B_learning_curve_alpha.py"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}