Custom_KNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom K-Nearest-Neighbor Algorithm Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import the libraries needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from mlxtend.data import loadlocal_mnist\n",
    "from sklearn.model_selection import train_test_split\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's load the dataset and store in X and y the data samples and data labels, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = loadlocal_mnist(\n",
    "    images_path='dataset/train-images.idx3-ubyte',\n",
    "    labels_path='dataset/train-labels.idx1-ubyte')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split dataset in training and testing samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's have a look at the dimensions of our train and test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Dimensions of X_train: %s x %s' % (X_train.shape[0], X_train.shape[1]))\n",
    "print('Dimensions of y_train: %s' % (len(y_train)))\n",
    "print('Dimensions of X_test: %s x %s' % (X_test.shape[0], X_test.shape[1]))\n",
    "print('Dimensions of y_test: %s' % (len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the K-nearest-neighbor algorithm, we can use different measurements of \"distance\" from the dataset and the unknown data. For this example, we will be using the Euclidean distance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def euclidean_distance(row1, row2):\n",
    "    \"\"\"\n",
    "    Calculate the Euclidean distance between two vectors\n",
    "    :param row1: numpy array with training samples\n",
    "    :param row2: numpy array with testing samples\n",
    "    :return: euclidean distance of row2 to with respect to row1\n",
    "    \"\"\"\n",
    "    distance = 0.0\n",
    "    for i in range(len(row1) - 1):\n",
    "        distance += (int(row1[i]) - int(row2[i])) ** 2\n",
    "    return np.sqrt(distance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, as we have seen in the course, let's implement the algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nearest_neighbor(x_trainset, y_trainset, unknown, k=1):\n",
    "    \"\"\"\n",
    "    K-nearest-neighbor implementation with euclidean distance\n",
    "    :param x_trainset: numpy array of training samples\n",
    "    :param y_trainset: numpy array of training labels\n",
    "    :param unknown: numpy array of unknown/unlabeled sample\n",
    "    :param k: number of neighbors to consider in evaluation\n",
    "    :return: the class/number given to unknown\n",
    "    \"\"\"\n",
    "    # store distances from all samples to the unknown\n",
    "    distances = []\n",
    "    for i, sample in enumerate(x_trainset):\n",
    "        distances.append((y_trainset[i], euclidean_distance(sample, unknown)))\n",
    "\n",
    "    # sort the distances in ascending order\n",
    "    distances.sort(key=lambda x: x[1])\n",
    "\n",
    "    # look only at the k entries on distances\n",
    "    kn = distances[:k]\n",
    "    # take all the classes\n",
    "    classes = [x[0] for x in kn]\n",
    "    # create a Counter object with the list of classes\n",
    "    occurence_count = Counter(classes)\n",
    "    # return the class with more occurences\n",
    "    return occurence_count.most_common(1)[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now create a simple function to call the K-nearest-neighbor algorithm in chunks of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_knn(X_train, X_test, y_train, y_test, k=1):\n",
    "    \"\"\"\n",
    "    Create a classification model with customised\n",
    "    implementation and evaluate the test set.\n",
    "    :param X_train: numpy array with training samples\n",
    "    :param X_test: numpy array with testing samples\n",
    "    :param y_train: numpy array with training labels\n",
    "    :param y_test: numpy array with training labels\n",
    "    :param k: number of k neighbors\n",
    "    :return: print accuracy of model\n",
    "    \"\"\"\n",
    "    # number of samples to be considered in evaluation\n",
    "    trainset_size = 100\n",
    "    # number of tests to evaluate\n",
    "    testset_size = 100\n",
    "\n",
    "    # keep track of correctly classified digits\n",
    "    correct = 0\n",
    "    for i in range(testset_size):\n",
    "        # classify digit based on our KNN implementation\n",
    "        estimation = nearest_neighbor(X_train[:trainset_size], y_train[:trainset_size], X_test[i], k=k)\n",
    "        # retrieve the correct value\n",
    "        truth = y_test[i]\n",
    "        if truth == estimation:\n",
    "            correct += 1\n",
    "\n",
    "    # compute accuracy\n",
    "    accuracy = correct / testset_size * 100\n",
    "    print(\"Our algorithm correctly classified a number %s / %s times\" % (str(correct), str(testset_size)))\n",
    "    print(\"Accuracy of our model: %s %%\" % str(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now evaluate our algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate our KNN implementation\n",
    "evaluate_knn(X_train, X_test, y_train, y_test, k=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, feel free to go back and edit parts of the code to try and improve the accuracy."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}