02- Regression models and linear prediction/Lab/Linear_models_lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning for Neuroscience, <br>Department of Brain Sciences, Faculty of Medicine, <br> Imperial College London\n",
    "### Contributors: Francesca Palermo, Nan Fletcher-Lloyd, Alex Capstick, Yu Chen\n",
    "**Winter 2022**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "from sklearn import datasets, linear_model\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.inspection import DecisionBoundaryDisplay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1: Linear Regression\n",
    "\n",
    "This task aims to train one or more linear regression models using [Diabetes dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#diabetes-dataset). Please click to see descriptive information about this dataset. We will provide instructions and example code step by step for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Load the dataset, split to training and test sets, train one or more linear models on the training data, and evaluate the performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the diabetes dataset\n",
    "diabetes = datasets.load_diabetes()\n",
    "diabetes_X, diabetes_y = diabetes.data, diabetes.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Split training and test set first ##\n",
    "## In this example, test set is 10% samples of the whole dataset. ##\n",
    "## You could change the training and test size and see the difference in model performance ##\n",
    "diabetes_X_train, diabetes_X_test, diabetes_y_train, diabetes_y_test = train_test_split(\n",
    "    diabetes_X, diabetes_y, test_size=0.1, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create a linear regression model ##\n",
    "## This example code only trains a vanialla linear regression model with least square error. ##\n",
    "regr = linear_model.LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Please try other linear models as well, such as Ridge, Lasso, useful information can be found [here](https://scikit-learn.org/stable/modules/linear_model.html#linear-model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create some other linear model here to replace the above one ##\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Train the model using all features ##\n",
    "regr.fit(diabetes_X_train, diabetes_y_train)\n",
    "\n",
    "## Make predictions using the testing set ##\n",
    "diabetes_y_pred = regr.predict(diabetes_X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Evaluate the predictions of above model ##\n",
    "## You could add more metrics here, such as mean absolute error ##\n",
    "\n",
    "# The coefficients\n",
    "print(\"Coefficients: \\n\", regr.coef_)\n",
    "# The mean squared error\n",
    "print(\"Mean squared error: %.2f\" % mean_squared_error(diabetes_y_test, diabetes_y_pred))\n",
    "# The coefficient of determination: 1 is perfect prediction\n",
    "print(\"Coefficient of determination: %.2f\" % r2_score(diabetes_y_test, diabetes_y_pred))\n",
    "# Add other metrics you are interesed below:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Try to visualize and get some insights of the features "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## This example code display a 2D scatter figure for each feature and the target value ##\n",
    "\n",
    "for i in range(diabetes_X.shape[1]):\n",
    "    plt.scatter(diabetes_X[:,i],diabetes_y)\n",
    "    plt.xlabel(diabetes.feature_names[i])\n",
    "    plt.ylabel('Target')\n",
    "    plt.show()\n",
    "    plt.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Please enter your own visualization code here. ##\n",
    "## For example, you could try to plot the histgram of each feature ## \n",
    "## Find how to plot histgram here: https://matplotlib.org/stable/gallery/statistics/hist.html ## \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: According to above results, select one or more features to train the linear model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Select one or more features according to your visualization to train the model ##\n",
    "\n",
    "selected_features = [3] ## You could add or change selected features in the square brackets, such as [2,3]\n",
    "N_s = len(selected_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Train the model using the selected features ##\n",
    "regr.fit(diabetes_X_train[:,selected_features], diabetes_y_train)\n",
    "\n",
    "## Make predictions using the testing set ##\n",
    "diabetes_y_pred = regr.predict(diabetes_X_test[:,selected_features])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Evaluate the predictions of above model ##\n",
    "## You could add more metrics here, such as mean absolute error ##\n",
    "\n",
    "# The coefficients\n",
    "print(\"Coefficients: \\n\", regr.coef_)\n",
    "# The mean squared error\n",
    "print(\"Mean squared error: %.2f\" % mean_squared_error(diabetes_y_test, diabetes_y_pred))\n",
    "# The coefficient of determination: 1 is perfect prediction\n",
    "print(\"Coefficient of determination: %.2f\" % r2_score(diabetes_y_test, diabetes_y_pred))\n",
    "# Add other metrics you are interesed below:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## If only one feature selected to train the model, visualize the fitted line after training ##\n",
    "\n",
    "if N_s == 1:\n",
    "    # Plot outputs\n",
    "    plt.scatter(diabetes_X_test[:,selected_features], diabetes_y_test, color=\"black\")\n",
    "    plt.plot(diabetes_X_test[:,selected_features], diabetes_y_pred, color=\"blue\", linewidth=3)\n",
    "\n",
    "    plt.xticks(())\n",
    "    plt.yticks(())\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now try to find a subset of features to train the model so that it can outperform the model trained by all features. Through above steps, you should have some ideas about  how to choose features for a linear model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## You can reuse above code and just record your results here ##\n",
    "\n",
    "## best subset of features: []\n",
    "## Mean squared error:\n",
    "## Mean absolute error:\n",
    "## Coefficient of determination: \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2: Linear Classification\n",
    "\n",
    "This task aims to train one or more linear classification models using [Breast cancer dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#breast-cancer-dataset). Please click to see descriptive information about this dataset. We will provide instructions and example code step by step for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the breast cancer dataset\n",
    "bcancer = datasets.load_breast_cancer()\n",
    "bcancer_X, bcancer_y = bcancer.data, bcancer.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Split training and test set first ##\n",
    "## The same function used in task 1 to split the training and test set ##\n",
    "bcancer_X_train, bcancer_X_test, bcancer_y_train, bcancer_y_test = train_test_split(\n",
    "    bcancer_X, bcancer_y, test_size=0.1, random_state=42\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Now try to visualize the features first before training a classifier\n",
    "\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Input your own visualiztion code here ##\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## You can also uncomment the example code to see an visualization example ##\n",
    "\n",
    "## This example code display estimated distributions of each feature in each class ##\n",
    "\n",
    "for i in range(bcancer_X.shape[1]):\n",
    "    sns.kdeplot(bcancer_X[bcancer_y==0,i],fill=True)\n",
    "    sns.kdeplot(bcancer_X[bcancer_y==1,i],fill=True)\n",
    "\n",
    "    plt.legend(['Malignant','Benigh'])\n",
    "    plt.xlabel(bcancer.feature_names[i])\n",
    "    plt.ylabel('Probability density')\n",
    "    plt.show()\n",
    "    plt.close()\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Train a linear classifier using TWO selected features and plot decision boundaries learned by the model. More information of linear models implemented in scikit-learn can be found [here](https://scikit-learn.org/stable/modules/classes.html?highlight=linear+model#module-sklearn.linear_model)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Select two features to train, change the feature index here to see different effectiveness between features ##\n",
    "selected_features = [0,1] \n",
    "\n",
    "## Create a classifier and fit it by the training data ##\n",
    "classifier = linear_model.LogisticRegression(penalty='none',solver='lbfgs').fit(bcancer_X_train[:,selected_features], bcancer_y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create some other linear model here to replace the above one, such as SGDClassifier, RidgeClassifier ##\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Plot decision boundaries in the 2D-dimension feature map ##\n",
    "disp = DecisionBoundaryDisplay.from_estimator(\n",
    "     classifier, bcancer_X[:,selected_features], response_method=\"predict\",\n",
    "     xlabel=bcancer.feature_names[selected_features[0]], ylabel=bcancer.feature_names[selected_features[1]],\n",
    "     alpha=0.5,\n",
    " )\n",
    "disp.ax_.scatter(bcancer_X[:,selected_features[0]], bcancer_X[:,selected_features[1]], c=bcancer_y, edgecolor=\"k\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Evaluate classifier on test set ##\n",
    "print('coeficients: {}'.format(np.round(classifier.coef_,3)))\n",
    "accuracy = classifier.score(bcancer_X_test[:,selected_features], bcancer_y_test)\n",
    "print('accuracy %.3f' % accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Now add regularization to the classifier (see the example code below), and repeat step 2 to see how regularization works on the decision boundaries,  coeficients, and what's the effect on the testing performance. The options of regularization criteria of each type of classifier can be found in their scikit learn webpage, e.g.  details of [logistic regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Example code of specifying regularization for the classifier ##\n",
    "#classifier = linear_model.LogisticRegression(penalty='l2',solver='lbfgs').fit(bcancer_X_train[:,selected_features], bcancer_y_train)\n",
    "#classifier = linear_model.LogisticRegression(penalty='l1',solver='liblinear').fit(bcancer_X_train[:,selected_features], bcancer_y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Change the test size to a larger portion (in the second cell under task 2) and repeat step 3. The regularization may not do good to the model performance when the training set is small, it may cause underfitting of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Record your results of different settings here ##\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Summarize  the key ideas that you have learned about linear models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## What are important factors for training a linear model on regression or classification tasks? ##\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py38",
   "language": "python",
   "name": "py38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}