PCA1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Andrew Carr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.preprocessing import scale\n",
    "from sklearn.model_selection import GridSearchCV, train_test_split, cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.decomposition import PCA\n",
    "import time\n",
    "from math import floor\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 1\n",
    "Apply PCA to the cancer dataset to reduce the dimension of the feature space to each of 15, 10, and 5.    Are there any features or combinations of features for which PCA is not a suitable method to use?  Explain.  WARNING: remember to center your data (subtract the mean) and also normalize it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before mean 61.890712339519624 var 228.29740508276657\n",
      "after mean -6.118909323768877e-16 var 1.0\n",
      "original data shape (569, 30)\n",
      "reduced data shape (569, 15)\n",
      "original data shape (569, 30)\n",
      "reduced data shape (569, 10)\n",
      "original data shape (569, 30)\n",
      "reduced data shape (569, 5)\n"
     ]
    }
   ],
   "source": [
    "#load data set\n",
    "data = load_breast_cancer()\n",
    "data, targets, names = data.data, data.target, data.feature_names\n",
    "print(\"before mean {} var {}\".format(np.mean(data), np.sqrt(np.var(data))))\n",
    "data = scale(data)\n",
    "print(\"after mean {} var {}\".format(np.mean(data), np.sqrt(np.var(data))))\n",
    "\n",
    "\n",
    "#for each feature space (5, 10, 15) reduce feature space\n",
    "for n in [15,10,5]:\n",
    "    reducer = PCA(n_components=n)\n",
    "    reducer.fit(data)\n",
    "    new_data = reducer.transform(data)\n",
    "    print(\"original data shape {}\".format(data.shape))\n",
    "    print(\"reduced data shape {}\".format(new_data.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See data below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean radius, mean texture, mean perimeter, mean area, mean smoothness, mean compactness, mean concavity, mean concave points, mean symmetry, mean fractal dimension, radius error, texture error, perimeter error, area error, smoothness error, compactness error, concavity error, concave points error, symmetry error, fractal dimension error, worst radius, worst texture, worst perimeter, worst area, worst smoothness, worst compactness, worst concavity, worst concave points, worst symmetry, worst fractal dimension\n"
     ]
    }
   ],
   "source": [
    "print(\", \".join(names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Yes the data looks like it is split into 3 different groups: Means, Errors, and Worsts. There are 10 of the same feature in each of these 3 groups. While these 3 groups have similar units, they are measuring fundamentally different things. This means that PCA will not do well in combining them into single features. \n",
    "\n",
    "However, as we see (with Mitch's homework) it doesn't actually have much of an effect on the end accuracy results or time to completion. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 2\n",
    "Apply three of your favorite classification methods to the full cancer data set and also to the PCA-reduced data.  Analyze and evaluate the performance (time and accuracy) for each combination.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n components 5\n",
      "\tnaive bayes full data accuracy 0.9590643274853801\t\ttime 0.0012028217315673828\n",
      "\tnaive bayes pca accuracy 0.9298245614035088\t\t\ttime 0.0010149478912353516\n",
      "\n",
      "\tlogistic regression full data accuracy 0.9883040935672515\ttime 0.0024499893188476562\n",
      "\tlogistic regression pca accuracy 0.9649122807017544\t\ttime 0.0011322498321533203\n",
      "\n",
      "\tsvm full data accuracy 0.9883040935672515\t\t\ttime 0.004132986068725586\n",
      "\tsvm pca accuracy 0.9473684210526315\t\t\t\ttime 0.0038750171661376953\n",
      "\n",
      "n components 10\n",
      "\tnaive bayes full data accuracy 0.9590643274853801\t\ttime 0.0009217262268066406\n",
      "\tnaive bayes pca accuracy 0.9181286549707602\t\t\ttime 0.0010197162628173828\n",
      "\n",
      "\tlogistic regression full data accuracy 0.9883040935672515\ttime 0.0022199153900146484\n",
      "\tlogistic regression pca accuracy 0.9824561403508771\t\ttime 0.0014829635620117188\n",
      "\n",
      "\tsvm full data accuracy 0.9883040935672515\t\t\ttime 0.0035812854766845703\n",
      "\tsvm pca accuracy 0.9590643274853801\t\t\t\ttime 0.0033979415893554688\n",
      "\n",
      "n components 15\n",
      "\tnaive bayes full data accuracy 0.9590643274853801\t\ttime 0.00092315673828125\n",
      "\tnaive bayes pca accuracy 0.9005847953216374\t\t\ttime 0.001065969467163086\n",
      "\n",
      "\tlogistic regression full data accuracy 0.9883040935672515\ttime 0.002489805221557617\n",
      "\tlogistic regression pca accuracy 0.9824561403508771\t\ttime 0.0016598701477050781\n",
      "\n",
      "\tsvm full data accuracy 0.9883040935672515\t\t\ttime 0.0036978721618652344\n",
      "\tsvm pca accuracy 0.9766081871345029\t\t\t\ttime 0.003532886505126953\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for n in [5,10,15]:\n",
    "    print(\"n components {}\".format(n))\n",
    "    reducer = PCA(n_components=n)\n",
    "    reducer.fit(data)\n",
    "    new_data = reducer.transform(data)\n",
    "    seed = 3264\n",
    "    train_x, test_x, train_y, test_y = train_test_split(data, targets, train_size=0.7,random_state=seed)\n",
    "    r_train_x, r_test_x, r_train_y, r_test_y = train_test_split(new_data, targets, train_size=0.7, random_state=seed)\n",
    "    #favorite classification 1\n",
    "    nb_clf = GaussianNB()\n",
    "    start = time.time()\n",
    "    nb_clf.fit(train_x,train_y)\n",
    "    print(\"\\tnaive bayes full data accuracy {}\\t\\ttime {}\".format(nb_clf.score(test_x, test_y), time.time()-start))\n",
    "    \n",
    "    nb_clf = GaussianNB()\n",
    "    start = time.time()\n",
    "    nb_clf.fit(r_train_x, r_train_y)\n",
    "    print(\"\\tnaive bayes pca accuracy {}\\t\\t\\ttime {}\\n\".format(nb_clf.score(r_test_x, r_test_y), time.time()-start))\n",
    "\n",
    "\n",
    "    \n",
    "    #favorite classification 2\n",
    "    log_clf = LogisticRegression()\n",
    "    start = time.time()\n",
    "    log_clf.fit(train_x,train_y)\n",
    "    print(\"\\tlogistic regression full data accuracy {}\\ttime {}\".format(log_clf.score(test_x, test_y), time.time()-start))\n",
    "    \n",
    "    log_clf = LogisticRegression()\n",
    "    start = time.time()\n",
    "    log_clf.fit(r_train_x, r_train_y)\n",
    "    print(\"\\tlogistic regression pca accuracy {}\\t\\ttime {}\\n\".format(log_clf.score(r_test_x, r_test_y), time.time()-start))\n",
    "\n",
    "\n",
    "    #favorite classification 3\n",
    "    svm_clf = SVC()\n",
    "    start = time.time()\n",
    "    svm_clf.fit(train_x,train_y)\n",
    "    print(\"\\tsvm full data accuracy {}\\t\\t\\ttime {}\".format(svm_clf.score(test_x, test_y), time.time()-start))\n",
    "    \n",
    "    svm_clf = SVC()\n",
    "    start = time.time()\n",
    "    svm_clf.fit(r_train_x, r_train_y)\n",
    "    print(\"\\tsvm pca accuracy {}\\t\\t\\t\\ttime {}\\n\".format(svm_clf.score(r_test_x, r_test_y), time.time()-start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "It is interesting to notice that Naive Bayes is faster on the full data than it is on the PCA data. However, with LogisticRegression PCA beats out the full dataset handily. The train test split has more effect on the testing accuracy than splitting up the features as we see with Mitchell Probst's homework. He split up his data by Mean, Error, and Worst and did PCA on each of those features, and with the same seed (trying different seeds) we found that we went back and forth on accuracy. With me winning sometimes and his winning others. This seems to show, that while some of the components may not be super well suited for PCA. They still work fine (as we see here) without pulling out features like Mitch did. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 3\n",
    "Find some aspect of your final project for which PCA is an appropriate dimension-reduction method.  Apply PCA and analyze the results and performance.  Compare to your results without PCA.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#apply PCA\n",
    "df = pd.read_csv(\"recipe_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = df.dropna()\n",
    "\n",
    "def my_floor(li):\n",
    "    return floor(li['rating'])\n",
    "\n",
    "df['rating_floor']  = df.apply(my_floor, axis=1)\n",
    "df['rating_floor'] = df['rating_floor'].astype(int)\n",
    "\n",
    "X = df[['calories', 'fat','protein', 'sodium']]\n",
    "Y = df['rating_floor']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time of pca decomp 0.008898019790649414\n",
      "full data score 0.4640276468740182\t\ttime 1.5698440074920654\n",
      "PCA score 0.4436066603832862\t\t\ttime 1.7538659572601318\n"
     ]
    }
   ],
   "source": [
    "#compare results\n",
    "start = time.time()\n",
    "reducer = PCA(n_components=2)\n",
    "reducer.fit(X)\n",
    "new_X = reducer.transform(X)\n",
    "print(\"time of pca decomp {}\".format(time.time() - start))\n",
    "\n",
    "\n",
    "\n",
    "train_x, test_x, train_y, test_y = train_test_split(X, Y, train_size=0.7, random_state=seed)\n",
    "r_train_x, r_test_x, r_train_y, r_test_y = train_test_split(new_X, Y, train_size=0.7, random_state=seed)\n",
    "\n",
    "rf_clf = RandomForestClassifier(n_estimators=100)\n",
    "start = time.time()\n",
    "rf_clf.fit(train_x, train_y)\n",
    "print(\"full data score {}\\t\\ttime {}\".format(rf_clf.score(test_x,test_y), time.time()-start))\n",
    "\n",
    "rf_clf = RandomForestClassifier(n_estimators=100)\n",
    "start = time.time()\n",
    "rf_clf.fit(r_train_x, r_train_y)\n",
    "print(\"PCA score {}\\t\\t\\ttime {}\".format(rf_clf.score(r_test_x,r_test_y), time.time()-start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 4\n",
    "Repeat what you did in the previous problem, but replacing PCA by a random projection. Try 5 different random projections and compare the results and performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time of random projection 0.010297775268554688\n",
      "Projection score 0.43669494187873076\t\t\ttime 1.6062350273132324\n",
      "time of random projection 0.009592771530151367\n",
      "Projection score 0.43512409676405905\t\t\ttime 1.4497499465942383\n",
      "time of random projection 0.009931087493896484\n",
      "Projection score 0.4344957587181904\t\t\ttime 1.420179843902588\n",
      "time of random projection 0.009760856628417969\n",
      "Projection score 0.4448633364750236\t\t\ttime 1.422384262084961\n",
      "time of random projection 0.00988006591796875\n",
      "Projection score 0.4410933081998115\t\t\ttime 1.4695649147033691\n",
      "\n",
      "Projection average score 0.43845428840716305\n",
      "Projection average time 0.009892511367797851\n"
     ]
    }
   ],
   "source": [
    "#compare results\n",
    "accuracy = []\n",
    "times = []\n",
    "for i in range(5):\n",
    "    start = time.time()\n",
    "    reducer = PCA(n_components=2, svd_solver='randomized')\n",
    "    reducer.fit(X)\n",
    "    new_X = reducer.transform(X)\n",
    "    proj_time = time.time() - start\n",
    "    times.append(proj_time)\n",
    "    print(\"time of random projection {}\".format(proj_time))\n",
    "\n",
    "    r_train_x, r_test_x, r_train_y, r_test_y = train_test_split(new_X, Y, train_size=0.7, random_state=seed)\n",
    "\n",
    "    rf_clf = RandomForestClassifier(n_estimators=100)\n",
    "    start = time.time()\n",
    "    rf_clf.fit(r_train_x, r_train_y)\n",
    "    acc = rf_clf.score(r_test_x,r_test_y)\n",
    "    accuracy.append(acc)\n",
    "    print(\"Projection score {}\\t\\t\\ttime {}\".format(acc, time.time()-start))\n",
    "\n",
    "print(\"\\nProjection average score {}\".format(np.mean(accuracy)))\n",
    "print(\"Projection average time {}\".format(np.mean(times)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice here that random projections are not quite as accurate, and not quite as fast. However, that is because we are working with something that is already in a very low dimension AND the PCA solver is highly optimized. Therefore, we won't see much difference. "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}