Sem-Python-4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Seminar Python 4: Using the scikit-learn package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For installing the package (to enable the import), the pip command is used - (Windows) Command prompt / (UNIX) terminal/shell:\n",
    "\n",
    "`pip install scikit-learn`\n",
    "\n",
    "The same is true for other Python packages: scipy, six, cycler, pyparsing, kiwisolver, python-dateutil, matplotlib, pytz, pandas, seaborn, numpy, sklearn, statsmodels etc.  \n",
    "It may be necessary to upgrade the PIP (Python package installer) - see details here: https://datatofish.com/upgrade-pip/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Check if package/module is avilable (handle import exceptions using try...except)\n",
    "try:                 \n",
    "    import sklearn\n",
    "    print('Import OK.')\n",
    "except ImportError as err: \n",
    "    print(err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#running pip in Jupyter Lab: # IPython \"magic command\"\n",
    "%pip install scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-Means clustering with Python and scikit-learn\n",
    "The K-Means method groups the observations from a dataset into K (=2, 3...) clusters - groups of related / similar observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 1. Grouping a dataset in 3 clusters\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "X = np.array([[5,3],\n",
    "     [10,15],\n",
    "     [15,12],\n",
    "     [24,10],\n",
    "     [30,45],\n",
    "     [85,70],\n",
    "     [71,80],\n",
    "     [60,78],\n",
    "     [55,52],\n",
    "     [80,91]])\n",
    "kmeans = KMeans(n_clusters=3)\n",
    "kmeans.fit(X)\n",
    "print(kmeans.cluster_centers_)\n",
    "print(kmeans.labels_)\n",
    "f1 = plt.figure()\n",
    "plt.scatter(X[:,0],X[:,1], label='True Position')\n",
    "f2 = plt.figure()\n",
    "plt.scatter(X[:,0], X[:,1], c=kmeans.labels_, cmap='rainbow')\n",
    "f3 = plt.figure()\n",
    "plt.scatter(kmeans.cluster_centers_[:,0] ,kmeans.cluster_centers_[:,1], color='black')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is used as a machine learning technique, in order to find classes (clusters) of items based on a training set; based on these clusters, we can predict if other items (in the test set) are likely to belong to one of the classes (classification). See more details [here](https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-6a6e67336aa1).\n",
    "\n",
    "**Application example**  \n",
    "The sinking of the Titanic in 1912 has produced 1505 casualties out of the total 2224 people on board (passengers and crew members).  \n",
    "We are using the datasets `train.csv` and `test.csv`, which contain passenger data. The training dataset includes the `Survived` column.  \n",
    "We may consider the hypothesis that survival was influenced by attributes like age, sex, ticket (passenger) class etc. We use K-Means to group passengers (training set) into two clusters - survivors and victims. Then we predict if other passengers were likely part of one or the other class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 1. Import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Exemplu 2. Step 2. Read data from files, print first 5 records\n",
    "pd.options.display.max_columns = 12\n",
    "test = pd.read_csv('test.csv')\n",
    "train = pd.read_csv('train.csv')\n",
    "\n",
    "print('*****test*****')\n",
    "print(test.head())\n",
    "print('*****train*****')\n",
    "print(train.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 3. Compute statistics\n",
    "print('*****test_stats*****')\n",
    "print(test.describe())\n",
    "print('*****train_stats*****')\n",
    "print(train.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Certain algorithms do not allow missing values. Therefore, these should be handled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 4. View columns in the train set, check for missing values\n",
    "print(train.columns.values)\n",
    "\n",
    "print('*****train missing values *****')\n",
    "print(train.isna())\n",
    "print('*****test missing values*****')\n",
    "print(test.isna())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 5. Calculate no. of missing values\n",
    "print(\"*****In the train set*****\")\n",
    "print(train.isna().sum())\n",
    "print(\"\\n\")\n",
    "print('*****In the test set*****')\n",
    "print(test.isna().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 6. Replace missing values with column average, using  fillna()\n",
    "train.fillna(train.mean(), inplace=True)\n",
    "test.fillna(test.mean(), inplace=True)\n",
    "print(train.isna().sum())\n",
    "print(test.isna().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 7. Evaluate survival depending on Pclass, Sex, SibSp\n",
    "print(train[['Pclass', 'Survived']].groupby(['Pclass'], as_index=False).mean().sort_values(by='Survived', ascending=False))\n",
    "print(train[[\"Sex\", \"Survived\"]].groupby(['Sex'], as_index=False).mean().sort_values(by='Survived', ascending=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 8. Chart for analyzing Age-Survived and Pclass-Survived\n",
    "g = sns.FacetGrid(train, col='Survived')\n",
    "g.map(plt.hist, 'Age')\n",
    "grid = sns.FacetGrid(train, col='Survived', row='Pclass')\n",
    "grid.map(plt.hist, 'Age')\n",
    "grid.add_legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 9. Show info about the training set\n",
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Pas 10. Remove non-numeric columns with no relevance for survival\n",
    "train = train.drop(['Name','Ticket', 'Cabin','Embarked'], axis=1)\n",
    "test = test.drop(['Name','Ticket', 'Cabin','Embarked'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Pas 11. Transform data type for column Sex\n",
    "labelEncoder = LabelEncoder()\n",
    "labelEncoder.fit(train['Sex'])\n",
    "labelEncoder.fit(test['Sex'])\n",
    "train['Sex'] = labelEncoder.transform(train['Sex'])\n",
    "test['Sex'] = labelEncoder.transform(test['Sex'])\n",
    "train.info()\n",
    "test.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 12. We use X as an array (numpy-array) based on the train set\n",
    "#without the Survived column; y is a vector based on the Survived column\n",
    "X = np.array(train.drop(['Survived'], 1).astype(float))\n",
    "y = np.array(train['Survived'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 13. Create a KMeans model with 2 clusters\n",
    "#(survivors / casualties). \n",
    "\n",
    "kmeans = KMeans(n_clusters=2) \n",
    "#kmeans = KMeans(n_clusters=2, max_iter=600)\n",
    "kmeans.fit(X)\n",
    "\n",
    "#Other argumnents for Kmeans:\n",
    "'''KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,\n",
    "    n_clusters=2, n_init=10, n_jobs=1, precompute_distances='auto',\n",
    "    random_state=None, tol=0.0001, verbose=0)'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 2. Step 14. Evaluate results\n",
    "correct = 0\n",
    "for i in range(len(X)):\n",
    "    predict_me = np.array(X[i].astype(float))\n",
    "    predict_me = predict_me.reshape(-1, len(predict_me))\n",
    "    prediction = kmeans.predict(predict_me)\n",
    "    if prediction[0] == y[i]:\n",
    "        correct += 1\n",
    "\n",
    "print(correct/len(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model has an accuracy of about 50%.\n",
    "It may be imporved by scaling the input data.  \n",
    "Example 2, step 15: open and run `Ex_2_15.ipynb`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- DO NOT RUN THIS CELL --- #\n",
    "#Example 2. Complete code for KMmeans example - see also ex_2.py,\n",
    "#-can be run as stand-alone program (in a terminal / IDLE / PyCharm)\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "pd.options.display.max_columns = 12\n",
    "test = pd.read_csv('test.csv')\n",
    "train = pd.read_csv('train.csv')\n",
    "\n",
    "print('*****test*****')\n",
    "print(test.head())\n",
    "print('*****train*****')\n",
    "print(train.head())\n",
    "\n",
    "print('*****test_stats*****')\n",
    "print(test.describe())\n",
    "print('*****train_stats*****')\n",
    "print(train.describe())\n",
    "\n",
    "print(train.columns.values)\n",
    "\n",
    "print(train.isna())\n",
    "print(test.isna())\n",
    "\n",
    "print('*****In the train set*****')\n",
    "print(train.isna().sum())\n",
    "print(\"\\n\")\n",
    "print('*****In the test set*****')\n",
    "print(test.isna().sum())\n",
    "\n",
    "train.fillna(train.mean(), inplace=True)\n",
    "test.fillna(test.mean(), inplace=True)\n",
    "print(train.isna().sum())\n",
    "print(test.isna().sum())\n",
    "\n",
    "\n",
    "print(train[['Pclass', 'Survived']].groupby(['Pclass'], as_index=False).mean().sort_values(by='Survived', ascending=False))\n",
    "print(train[[\"Sex\", \"Survived\"]].groupby(['Sex'], as_index=False).mean().sort_values(by='Survived', ascending=False))\n",
    "print(train[[\"SibSp\", \"Survived\"]].groupby(['SibSp'], as_index=False).mean().sort_values(by='Survived', ascending=False))\n",
    "\n",
    "g = sns.FacetGrid(train, col='Survived')\n",
    "g.map(plt.hist, 'Age')\n",
    "grid = sns.FacetGrid(train, col='Survived', row='Pclass')\n",
    "grid.map(plt.hist, 'Age')\n",
    "grid.add_legend()\n",
    "plt.show()\n",
    "\n",
    "train.info()\n",
    "\n",
    "train = train.drop(['Name','Ticket', 'Cabin','Embarked'], axis=1)\n",
    "test = test.drop(['Name','Ticket', 'Cabin','Embarked'], axis=1)\n",
    "\n",
    "labelEncoder = LabelEncoder()\n",
    "labelEncoder.fit(train['Sex'])\n",
    "labelEncoder.fit(test['Sex'])\n",
    "train['Sex'] = labelEncoder.transform(train['Sex'])\n",
    "test['Sex'] = labelEncoder.transform(test['Sex'])\n",
    "\n",
    "train.info()\n",
    "\n",
    "test.info()\n",
    "\n",
    "X = np.array(train.drop(['Survived'], 1).astype(float))\n",
    "\n",
    "y = np.array(train['Survived'])\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "kmeans = KMeans(n_clusters=2)\n",
    "kmeans.fit(X_scaled)\n",
    "KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=600,\n",
    "    n_clusters=2, n_init=10, n_jobs=1, precompute_distances='auto',\n",
    "    random_state=None, tol=0.0001, verbose=0)\n",
    "correct = 0\n",
    "for i in range(len(X)):\n",
    "    predict_me = np.array(X[i].astype(float))\n",
    "    predict_me = predict_me.reshape(-1, len(predict_me))\n",
    "    prediction = kmeans.predict(predict_me)\n",
    "    if prediction[0] == y[i]:\n",
    "        correct += 1\n",
    "\n",
    "print(correct/len(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic regression\n",
    "This machine learning technique is used for classification problems - predicting if an item is likely to belong or not to a class (binary logistic regression) - e.g. passenger is survivor or not.  \n",
    "It works by fitting a regression model based on the sigmoid (logistic) function - instead of a line, like linear regression. For details, see [this page](https://towardsdatascience.com/introduction-to-logistic-regression-66248243c148).  \n",
    "The regression model is determined by using the training dataset. Then we can attempt predictions for the items in the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 3. logistic regression\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "pd.options.display.max_columns = 12\n",
    "test = pd.read_csv('test1.csv')\n",
    "train = pd.read_csv('train.csv')\n",
    "\n",
    "print('*****test*****')\n",
    "print(test[:4])\n",
    "print('*****train*****')\n",
    "print(train[:4])\n",
    "\n",
    "train.fillna(train.mean(), inplace=True)\n",
    "test.fillna(test.mean(), inplace=True)\n",
    "print(train.isna().sum())\n",
    "print(test.isna().sum())\n",
    "\n",
    "train['IsFemale'] = (train['Sex'] == 'female').astype(int)\n",
    "test['IsFemale'] = (test['Sex'] == 'female').astype(int)\n",
    "\n",
    "predictors = ['Pclass', 'IsFemale', 'Age']\n",
    "X_train = train[predictors].values\n",
    "X_test = test[predictors].values\n",
    "y_train = train['Survived'].values\n",
    "y_test= test['Survived'].values\n",
    "print(X_train[:5])\n",
    "print(y_train[:5])\n",
    "\n",
    "model = LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
    "intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
    "penalty='l2', random_state=None, solver='lbfgs', tol=0.0001,\n",
    "verbose=0, warm_start=False)\n",
    "\n",
    "model.fit(X_train, y_train)\n",
    "y_predict = model.predict(X_test)\n",
    "print(y_predict)\n",
    "\n",
    "print((y_test == y_predict).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 4. Simple linear regression (OLS - Ordinary Least Squares)\n",
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "pd.options.display.max_columns = 12\n",
    "test = pd.read_csv('test.csv')\n",
    "train = pd.read_csv('train.csv')\n",
    "\n",
    "train.fillna(train.mean(), inplace=True)\n",
    "test.fillna(test.mean(), inplace=True)\n",
    "\n",
    "train['IsFemale'] = (train['Sex'] == 'female').astype(int)\n",
    "test['IsFemale'] = (test['Sex'] == 'female').astype(int)\n",
    "\n",
    "X_train = train['IsFemale'].values\n",
    "X_train = sm.add_constant(X_train)\n",
    "\n",
    "y_train = train['Survived'].values\n",
    "\n",
    "model = sm.OLS(y_train, X_train)\n",
    "\n",
    "results = model.fit()\n",
    "print(results.params)\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example 5. Multiple linear regression\n",
    "import pandas as pd\n",
    "import statsmodels.formula.api as smf\n",
    "\n",
    "pd.options.display.max_columns = 12\n",
    "test = pd.read_csv('test.csv')\n",
    "train = pd.read_csv('train.csv')\n",
    "\n",
    "\n",
    "train.fillna(train.mean(), inplace=True)\n",
    "test.fillna(test.mean(), inplace=True)\n",
    "\n",
    "train['IsFemale'] = (train['Sex'] == 'female').astype(int)\n",
    "test['IsFemale'] = (test['Sex'] == 'female').astype(int)\n",
    "\n",
    "\n",
    "X = pd.DataFrame(train, columns=['Pclass', 'IsFemale', 'Age'])\n",
    "y = train['Survived']\n",
    "\n",
    "results = smf.ols('y ~ Pclass + IsFemale + Age', data=train).fit()\n",
    "print(results.params)\n",
    "\n",
    "print(round(results.predict(train[:5])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Referenes\n",
    "1. J. VanderPlas, Python Data Science Handbook: https://jakevdp.github.io/PythonDataScienceHandbook/index.html, Cap. 5\n",
    "1. https://stackabuse.com/k-means-clustering-with-scikit-learn/ \n",
    "2. https://www.datacamp.com/community/tutorials/k-means-clustering-python\n",
    "3. Wes McKinney, 2nd Edition of Python for Data Analysis DATA WRANGLING WITH PANDAS, NUMPY, AND IPYTHON, O’Reilley\n",
    "4. https://towardsdatascience.com/logistic-regression-detailed-overview-46c4da4303bc \n",
    "5. https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.RegressionResults.html "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}