Lab0 - Classify Irises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classify Irises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this laboration we will try to build a neural net to identify what type of Iris a flower is. We will also evaluate the Artificial Neural Net by comparing it to other methods.\n",
    "\n",
    "The dataset is a classical dataset in machinelearning collected in 1936 by Edgar Anderson and first published by Ronald Fisher. By measuring the sepal width, sepal length, petal width and petal length Fisher tried to specify differences between the spicies.\n",
    "\n",
    "This is a simple example and often used when teaching ML algorithms. Since it contains of a reasonable amount of parameters it is easy to understand the data. It is also very good in the sense that there are some overlaps that might be very hard to detect accuratly. It therefor illustrates the complexity of building an AI that that has an accuracy of 100%.\n",
    "\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/2560/1*7bnLKsChXq94QjtAiRn40w.png \"Three types of Irises the we want to classify.\")\n",
    "\n",
    "As you see those flowers are quite similar. Compared to a botanist the NN might not be perfect. But compared to someone who tries to identify the flower based on a flower book it might be quite accurate. Let's see how to do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we load all libraries needed for this notebook\n",
    " \n",
    "import pandas as pd # Load the Pandas libraries with alias 'pd'\n",
    "import tensorflow as tf # A library developed by google making GPU calculations simple\n",
    "import seaborn as sns # A nice library for plotting, alternative to matplotlib\n",
    "from matplotlib import pyplot as plt # Another plotting library\n",
    "\n",
    "from sklearn.preprocessing import normalize #\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import preprocessing\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read data from file iris.csv' from the data folder\n",
    "# Open the csv to see the data\n",
    "try:\n",
    "    # I you run local or on binder:\n",
    "    data = pd.read_csv(\"./data/iris.csv\") \n",
    "except:\n",
    "    # If you run on colab\n",
    "    data = pd.read_csv(\"https://raw.githubusercontent.com/simjoh/AI-Lab-Neural-Net-and-Python/master/data/iris.csv\") \n",
    "# Preview the first 10 lines of the loaded data \n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the data\n",
    "Before starting to do any modifications of the data or attempts to build some kind of AI/ML model. We must understand the data. This explorative phase is very important in order to succesful build a model. By understanding the data we might be able to improve the dataset.\n",
    "\n",
    "One method in the pandas.DataFrame class is the `describe` method which shows some statistical information about each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols=data.columns\n",
    "sns.set(font_scale=2)\n",
    "sns.pairplot(vars=cols[:-1], data=data, hue='Species', height=5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete data\n",
    "By reading the data we can se that the parameter \"Id\" has a very strong correlation with the type of Iris. This is a pure artificial correlation existing only because of the structure of the data.\n",
    "\n",
    "Id should therefor be deleted. One way to do this is with the `pop` method included in the dataframe class.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    data.pop('Id')\n",
    "except:\n",
    "    print('Id already deleted')\n",
    "data.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(vars=cols[1:-1], data=data, hue='Species', height=5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Correlation of data\n",
    "One method to analyze how well one variable describes another variable is by correlations.\n",
    "\n",
    "$\\rho(x,y)=\\frac{\\sum\\limits_{i=1}^n(x_i-\\bar{x})(y_i-\\bar{y})}{\\sum\\limits_{i=1}^n(x_i-\\bar{x})^2\\sum\\limits_{i=1}^n(y_i-\\bar{y})^2}$\n",
    "\n",
    "The correlation is a method included in the dataframe `corr`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the syntax here:\n",
    "corr = data.corr()\n",
    "corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# Set up the matplotlib figure\n",
    "sns.set(font_scale=1)\n",
    "f, ax = plt.subplots(figsize=(7, 5))\n",
    "\n",
    "# Generate a custom diverging colormap\n",
    "cmap = sns.diverging_palette(220, 10, as_cmap=True)\n",
    "\n",
    "# Draw the heatmap with the mask and correct aspect ratio\n",
    "sns.heatmap(corr, cmap=cmap, vmax=1,vmin=-1, center=0,# mask=mask,\n",
    "            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the the figure and table aboove we can see that two of the variables are well correlated. That means it is quite likely that one of the well correlated variables could be deleted without impacting the accuracy of the model.\n",
    "\n",
    "### Splitting of data into training and test \n",
    "Note that in this notebook we are normalizing the entire dataset in one operation. In a real scenario when new flowers are to be classified these has to be scaled in the same way. One could take a set of the flowers as third (validation) set before the normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shuffle the data each time \n",
    "data = data.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "# Split the data into training and test data\n",
    "X = data.iloc[:,:-1].values\n",
    "y = data.iloc[:,-1].values\n",
    "X_normalized = normalize(X,axis=0)\n",
    "\n",
    "total_length = len(data)\n",
    "train_length = int(0.8*total_length)\n",
    "test_length = int(0.2*total_length)\n",
    "\n",
    "# Split into train and test\n",
    "X_train = X_normalized[:train_length]\n",
    "X_test = X_normalized[train_length:]\n",
    "y_train = y[:train_length]\n",
    "y_test = y[train_length:]\n",
    "\n",
    "le = preprocessing.LabelEncoder()\n",
    "le.fit(y_train)\n",
    "train_label = le.transform(y_train)\n",
    "y_train = tf.keras.utils.to_categorical(train_label,num_classes=3)\n",
    "\n",
    "le2 = preprocessing.LabelEncoder()\n",
    "le2.fit(y_test)\n",
    "test_label = le2.transform(y_test)\n",
    "y_test=tf.keras.utils.to_categorical(test_label,num_classes=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the model\n",
    "We use Keras to build up the neural network. In the first layer we set how many input neurons to use which should match number of input variables. In the cell below there is 2 hidden layers of 30 neurons in each layer and the rectified linear unit is used as activation function.\n",
    "\n",
    "By setting batch_size to 1 we use stochastic gradient descent and by setting it to the sample size (120) gradient decent is used. By setting it to a fraction of the sample size we used the mini batch gradidient descent. \n",
    "\n",
    "The first line is used for early stopping of the training you can tune the parameters when to stop the training process, it might be used to reduce overfitting but it would be better to avoid over training by adjusting the net."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "esCallback = (tf.keras.callbacks.EarlyStopping(monitor='loss',# or val_loss to reduce overfitting\n",
    "                           min_delta=0,\n",
    "                           patience=10,\n",
    "                           verbose=1,\n",
    "                           mode='min'))\n",
    "model=tf.keras.Sequential()\n",
    "model.add(tf.keras.layers.Dense(30, input_dim=4, activation='relu'))\n",
    "model.add(tf.keras.layers.Dense(30, activation='relu'))\n",
    "model.add(tf.keras.layers.Dense(3, activation='softmax'))\n",
    "model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
    "history = model.fit(X_train,y_train,validation_data=(X_test,y_test),batch_size=20,epochs=500,verbose=1, callbacks=[esCallback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Validate results\n",
    "By looking output from the model fit. We can se that the accuracy of training data keeps raising throuout the process of finding constants for each neuron in the net. However looking att the accuracy of the validation set there is some kind of maximum after quite few iteration. \n",
    "\n",
    "We can for a reasonable high amount epochs by plotting the history of the accuracy and loss. What is a reasonable amount of epochs?\n",
    "\n",
    "Note that loss function and cost function is the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# summarize history for accuracy\n",
    "sns.set(font_scale=1)\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()\n",
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss / cost')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Better net?\n",
    "In the example above there are quite a lot of neurons in each layer of the net. What happens if we follow the rule of thumb and use somewhere between the amount of outputs classes and number of input variables in each layer? \n",
    "\n",
    "Do we see any diffrence in the loss and what happens to the accuracy? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model2=tf.keras.Sequential()\n",
    "model2.add(tf.keras.layers.Dense(4,input_dim=4,activation='relu'))\n",
    "model2.add(tf.keras.layers.Dropout(0.1))\n",
    "model2.add(tf.keras.layers.Dense(4))\n",
    "model2.add(tf.keras.layers.Dense(3,activation='softmax'))\n",
    "model2.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
    "history2 = model2.fit(X_train,y_train,validation_data=(X_test,y_test),batch_size=20,epochs=500,verbose=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# summarize history for accuracy\n",
    "plt.plot(history2.history['acc'])\n",
    "plt.plot(history2.history['val_acc'])\n",
    "plt.title('model2 accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()\n",
    "# summarize history for loss\n",
    "plt.plot(history2.history['loss'])\n",
    "plt.plot(history2.history['val_loss'])\n",
    "plt.title('model2 loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Another way of splitting the data\n",
    "In this celll the numpy split is used to store the data into tree sets train, validation and test with a fraction of 0.6, 0.2 and 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Normalize the data before splitting it train test and validation set.\n",
    "data = data.sample(frac=1).reset_index(drop=True)\n",
    "data.iloc[:,:-1] = normalize(data.iloc[:,:-1].values,axis=0)\n",
    "\n",
    "# Split the data into train test and validation sets.\n",
    "train, validate, test = np.split(data.sample(frac=1), [int(.6*len(data)), int(.8*len(data))])\n",
    "\n",
    "# Split data into \n",
    "train_x = train.iloc[:,:-1].values\n",
    "train_y = train.iloc[:,-1].values\n",
    "\n",
    "test_x = test.iloc[:,:-1].values\n",
    "test_y = test.iloc[:,-1].values\n",
    "\n",
    "validate_x = validate.iloc[:,:-1].values\n",
    "validate_y = validate.iloc[:,-1].values\n",
    "\n",
    "# Convert the label to a understandable format for tensorflow\n",
    "le = preprocessing.LabelEncoder()\n",
    "le.fit(train_y)\n",
    "\n",
    "train_label = le.transform(train_y)\n",
    "train_y = tf.keras.utils.to_categorical(train_label,num_classes=3)\n",
    "\n",
    "test_label = le.transform(test_y)\n",
    "test_y = tf.keras.utils.to_categorical(test_label,num_classes=3)\n",
    "\n",
    "validate_label = le.transform(validate_y)\n",
    "validate_y = tf.keras.utils.to_categorical(validate_label, num_classes=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What about another activation function\n",
    "Usually Rectified Linear Unit (ReLU) is a good choice as activation function. In the example below we use the sigmoid function instead. Look at at the curves of the history and the accuracy and see how they compare with the curves above whe ReLU is used. The slow learning rate in the beginning and the end is due to the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_sigmoid =tf.keras.Sequential()\n",
    "model_sigmoid.add(tf.keras.layers.Dense(200,input_dim=4,activation='sigmoid'))\n",
    "model_sigmoid.add(tf.keras.layers.Dense(100,activation='sigmoid'))\n",
    "model_sigmoid.add(tf.keras.layers.Dense(3,activation='softmax'))\n",
    "model_sigmoid.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
    "history_sigmaid = model_sigmoid.fit(train_x,train_y,validation_data=(X_test,y_test),batch_size=20,epochs=300,verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# summarize history for accuracy\n",
    "plt.plot(history_sigmaid.history['acc'])\n",
    "plt.plot(history_sigmaid.history['val_acc'])\n",
    "plt.title('model2 accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()\n",
    "# summarize history for loss\n",
    "plt.plot(history_sigmaid.history['loss'])\n",
    "plt.plot(history_sigmaid.history['val_loss'])\n",
    "plt.title('model2 loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "predicted_test = np.argmax(model_sigmoid.predict(test_x),axis=1)\n",
    "\n",
    "predicted_validation = np.argmax(model_sigmoid.predict(validate_x),axis=1)\n",
    "predicted_train = np.argmax(model_sigmoid.predict(train_x),axis=1)\n",
    "\n",
    "print('True test : ', np.argmax(test_y,axis=1))\n",
    "print('Pred test : ', predicted_test)\n",
    "print()\n",
    "print('True val  : ', np.argmax(validate_y,axis=1))\n",
    "print('Pred val  : ', predicted_validation)\n",
    "print()\n",
    "print('True train: ', np.argmax(train_y,axis=1)[:30])\n",
    "print('Pred train: ', predicted_train[:30])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['Prediction']=predicted_test==np.argmax(test_y,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set(font_scale=1)\n",
    "f, ax = plt.subplots(figsize=(7, 7))\n",
    "plt.title('Note that variables are normalized')\n",
    "sns.scatterplot(test.SepalLengthCm, test.SepalWidthCm, hue=test.Species, style=test.Prediction, legend='full' )\n",
    "plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Own work\n",
    "Investigate how well the model predicts the training data and validation data\n",
    "\n",
    "1. Make a new column in dataframe `train` and `validate` with the predicted value.\n",
    "2. Plot each spicie with a different color and mark them with different markers if prediction is correct or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other algorithms\n",
    "Random forest is another method in ML. The procedure for doing RF or NN is pretty similar and the results in this case is rather comparable. But notice that the time needed to train RF is significant shorter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = RandomForestClassifier(n_estimators = 100)\n",
    "clf.fit(train_x, train_y)\n",
    "print('RF pred test: ', np.argmax(clf.predict(test_x), axis=1))\n",
    "print('NN pred test: ', predicted_test)\n",
    "print('True    test: ', np.argmax(test_y, axis=1))\n",
    "print()\n",
    "print('RF pred valid: ', np.argmax(clf.predict(validate_x), axis=1))\n",
    "print('NN pred valid: ',  predicted_validation)\n",
    "print('True    valid: ', np.argmax(validate_y, axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}