04 - Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering\n",
    "\n",
    "In contrast to *supervised* machine learning, *unsupervised* learning is used when there is no \"ground truth\" from which to train and validate label predictions. The most common form of unsupervised learning is *clustering*, which is simllar conceptually to *classification*, except that the the training data does not include known values for the class label to be predicted. Clustering works by separating the training cases based on similarities that can be determined from their feature values. Think of it this way; the numeric features of a given entity can be thought of as vector coordinates that define the entity's position in n-dimensional space. What a clustering model seeks to do is to identify groups, or *clusters*, of entities that are close to one another while being separated from other clusters.\n",
    "\n",
    "For example, let's take a look at a dataset that contains measurements of different species of wheat seed.\n",
    "\n",
    "> **Citation**: The seeds dataset used in the this exercise was originally published by the Institute of Agrophysics of the Polish Academy of Sciences in Lublin, and can be downloaded from the UCI dataset repository (Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# load the training dataset\n",
    "data = pd.read_csv('data/seeds.csv')\n",
    "\n",
    "# Display a random sample of 10 observations (just the features)\n",
    "features = data[data.columns[0:6]]\n",
    "features.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the dataset contains six data points (or *features*) for each instance (*observation*) of a seed. So you could interpret these as coordinates that describe each instance's location in six-dimensional space.\n",
    "\n",
    "Now, of course six-dimensional space is difficult to visualise in a three-dimensional world, or on a two-dimensional plot; so we'll take advantage of a mathematical technique called *Principal Component Analysis* (PCA) to analyze the relationships between the features and summarize each observation as coordinates for two principal components - in other words, we'll translate the six-dimensional feature values into two-dimensional coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Normalize the numeric features so they're on the same scale\n",
    "scaled_features = MinMaxScaler().fit_transform(features[data.columns[0:6]])\n",
    "\n",
    "# Get two principal components\n",
    "pca = PCA(n_components=2).fit(scaled_features)\n",
    "features_2d = pca.transform(scaled_features)\n",
    "features_2d[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the data points translated to two dimensions, we can visualize them in a plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "plt.scatter(features_2d[:,0],features_2d[:,1])\n",
    "plt.xlabel('Dimension 1')\n",
    "plt.ylabel('Dimension 2')\n",
    "plt.title('Data')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully you can see at least two, arguably three, reasonably distinct groups of data points; but here lies one of the fundamental problems with clustering - without known class labels, how do you know how many clusters to separate your data into?\n",
    "\n",
    "One way we can try to find out is to use a data sample to create a series of clustering models with an incrementing number of clusters, and measure how tightly the data points are grouped within each cluster. A metric often used to measure this tightness is the *within cluster sum of squares* (WCSS), with lower values meaning that the data points are closer. You can then plot the WCSS for each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing the libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.cluster import KMeans\n",
    "%matplotlib inline\n",
    "\n",
    "# Create 10 models with 1 to 10 clusters\n",
    "wcss = []\n",
    "for i in range(1, 11):\n",
    "    kmeans = KMeans(n_clusters = i)\n",
    "    # Fit the data points\n",
    "    kmeans.fit(features.values)\n",
    "    # Get the WCSS (inertia) value\n",
    "    wcss.append(kmeans.inertia_)\n",
    "    \n",
    "#Plot the WCSS values onto a line graph\n",
    "plt.plot(range(1, 11), wcss)\n",
    "plt.title('WCSS by Clusters')\n",
    "plt.xlabel('Number of clusters')\n",
    "plt.ylabel('WCSS')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot shows a large reduction in WCSS (so greater *tightness*) as the number of clusters increases from one to two, and a further noticable reduction from two to three clusters. After that, the reduction is less pronounced, resulting in an \"elbow\" in the chart at around three clusters. This is a good indication that there are two to three reasonably well separated clusters of data points.\n",
    "\n",
    "## K-Means Clustering\n",
    "\n",
    "The algorithm we used to create our test clusters is *K-Means*. This is a commonly used clustering algorithm that separates a dataset into *K* clusters of equal variance. The number of clusters, *K*, is user defined. The basic algorithm has the following steps:\n",
    "\n",
    "1. A set of K centroids are randomly chosen.\n",
    "2. Clusters are formed by assigning the data points to their closest centroid.\n",
    "3. The means of each cluster is computed and the centroid is moved to the mean.\n",
    "4. Steps 2 and 3 are repeated until a stopping criteria is met. Typically, the algorithm terminates when each new iteration results in negligable movement of centroids and the clusters become static.\n",
    "5. When the clusters stop changing, the algorithm has *converged*, defining the locations of the clusters - note that the random starting point for the centroids means that re-running the algorithm could result in slightly different clusters, so training usually involves multiple iterations, reinitializing the centroids each time, and the model with the best WCSS is selected.\n",
    "\n",
    "Let's try using K-Means on our seeds data with a K value of 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "# Create a model based on 3 centroids\n",
    "model = KMeans(n_clusters=3, init='k-means++', n_init=100, max_iter=1000)\n",
    "# Fit to the data and predict the cluster assignments for each data point\n",
    "km_clusters = model.fit_predict(features.values)\n",
    "# View the cluster assignments\n",
    "km_clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see those cluster assignments with the two-dimensional data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_clusters(samples, clusters):\n",
    "    col_dic = {0:'blue',1:'green',2:'orange'}\n",
    "    mrk_dic = {0:'*',1:'x',2:'+'}\n",
    "    colors = [col_dic[x] for x in clusters]\n",
    "    markers = [mrk_dic[x] for x in clusters]\n",
    "    for sample in range(len(clusters)):\n",
    "        plt.scatter(samples[sample][0], samples[sample][1], color = colors[sample], marker=markers[sample], s=100)\n",
    "    plt.xlabel('Dimension 1')\n",
    "    plt.ylabel('Dimension 2')\n",
    "    plt.title('Assignments')\n",
    "    plt.show()\n",
    "\n",
    "plot_clusters(features_2d, km_clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully, the data has been separated into three distinct clusters.\n",
    "\n",
    "So what's the practical use of clustering? In some cases, you may have data that you need to group into distict clusters without knowing how many clusters there are or what they indicate. For example a marketing organization might want to separate customers into distinct segments, and then investigate how those segments exhibit different purchasing behaviors.\n",
    "\n",
    "Sometimes, clustering is used as an initial step towards creating a classification model. You start by identifying distinct groups of data points, and then assign class labels to those clusters. You can then use this labelled data to train a classification model.\n",
    "\n",
    "In the case of the seeds data, the different species of seed are already known and encoded as 0 (*Kama*), 1 (*Rosa*), or 2 (*Canadian*), so we can use these identifiers to compare the species classifications to the clusters identified by our unsupervised algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed_species = data[data.columns[7]]\n",
    "plot_clusters(features_2d, seed_species.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There may be some differences between the cluster assignments and class labels, but the K-Means model should have done a reasonable job of clustering the observations so that seeds of the same species are generally in the same cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hierarchical Clustering\n",
    "\n",
    "Hierarchical clustering methods make fewer distributional assumptions when compared to K-means methods. However, K-means methods are generally more scalable, sometimes very much so.\n",
    "\n",
    "Hierarchical clustering creates clusters by either a *divisive* method or *agglomerative* method. The divisive method is a \"top down\" approach starting with the entire dataset and then finding partitions in a stepwise manner. Agglomerative clustering is a \"bottom up** approach. In this lab you will work with agglomerative clustering which roughly works as follows:\n",
    "\n",
    "1. The linkage distances between each of the data points is computed.\n",
    "2. Points are clustered pairwise with their nearest neighbor.\n",
    "3. Linkage distances between the clusters are computed.\n",
    "4. Clusters are combined pairwise into larger clusters.\n",
    "5. Steps 3 and 4 are repeated until all data points are in a single cluster.\n",
    "\n",
    "The linkage function can be computed in a number of ways:\n",
    "- Ward linkage measures the increase in variance for the clusters being linked,\n",
    "- Average linkage uses the mean pairwise distance between the members of the two clusters,\n",
    "- Complete or Maximal linkage uses the maximum distance between the members of the two clusters.\n",
    "\n",
    "Several different distance metrics are used to compute linkage functions:\n",
    "- Euclidian or l2 distance is the most widely used. This is the only metric for the Ward linkage method.\n",
    "- Manhattan or l1 distance is robust to outliers and has other interesting properties.\n",
    "- Cosine similarity, is the dot product between the location vectors divided by the magnitudes of the vectors. Notice that this metric is a measure of similarity, whereas the other two metrics are measures of difference. Similarity can be quite useful when working with data such as images or text documents.\n",
    "\n",
    "### Agglomerative Clustering\n",
    "\n",
    "Let's see an example of clustering the seeds data using an agglomerative clustering algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import AgglomerativeClustering\n",
    "\n",
    "agg_model = AgglomerativeClustering(n_clusters=3)\n",
    "agg_clusters = agg_model.fit_predict(features.values)\n",
    "agg_clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what do the agglomerative cluster assignments look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "def plot_clusters(samples, clusters):\n",
    "    col_dic = {0:'blue',1:'green',2:'orange'}\n",
    "    mrk_dic = {0:'*',1:'x',2:'+'}\n",
    "    colors = [col_dic[x] for x in clusters]\n",
    "    markers = [mrk_dic[x] for x in clusters]\n",
    "    for sample in range(len(clusters)):\n",
    "        plt.scatter(samples[sample][0], samples[sample][1], color = colors[sample], marker=markers[sample], s=100)\n",
    "    plt.xlabel('Dimension 1')\n",
    "    plt.ylabel('Dimension 2')\n",
    "    plt.title('Assignments')\n",
    "    plt.show()\n",
    "\n",
    "plot_clusters(features_2d, agg_clusters)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Reading\n",
    "\n",
    "To learn more about clustering with scikit-learn, see the [scikit-learn documentation](https://scikit-learn.org/stable/modules/clustering.html).\n",
    "\n",
    "## Further Reading\n",
    "\n",
    "To learn more about the Python packages you explored in this notebook, see the following documentation:\n",
    "\n",
    "- [NumPy](https://numpy.org/doc/stable/)\n",
    "- [Pandas](https://pandas.pydata.org/pandas-docs/stable/)\n",
    "- [Matplotlib](https://matplotlib.org/contents.html)\n",
    "\n",
    "## Challenge: Cluster Unlabelled Data\n",
    "\n",
    "Now that you've seen how to create a clustering model, why not try for yourself? You'll find a clustering challenge in the [/challenges/04 - Clustering Challenge.ipynb](./challenges/04%20-%20Clustering%20Challenge.ipynb) notebook!\n",
    "\n",
    "> **Note**: The time to complete this optional challenge is not included in the estimated time for this exercise - you can spend as little or as much time on it as you like!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}