waywt_app.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What Are You Wearing Today (WAYWT)\n",
    "\n",
    "## HackGT Project \n",
    "\n",
    "---\n",
    "### Introduction \n",
    "\n",
    "In this notebook, we will develop a recommender system that will be used as part of a web app. The goal of this project is to accept a user-supplied image of clothing (only tops for now) as input, and score it against the user's 'style vector' (generated via user preferences during initialization of the app). All image files used to train the model for this project are from the DeepFashion dataset.\n",
    "\n",
    "> Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. [DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Liu_DeepFashion_Powering_Robust_CVPR_2016_paper.pdf). In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, 2016.\n",
    "\n",
    "### Machine Learning Pipeline\n",
    "\n",
    "We break the notebook into separate steps.  Feel free to use the links below to navigate the notebook.\n",
    "\n",
    "* [Step 1](#step1): Load dataset\n",
    "* [Step 2](#step2): Pre-process and build training, testing, and validation data loaders\n",
    "* [Step 3](#step3): Train the model (using Transfer Learning)\n",
    "* [Step 4](#step4): Create style vectors for each user\n",
    "* [Step 5](#step5): Test out recommendations\n",
    "\n",
    "---\n",
    "<a id='step1'></a>\n",
    "## Step 1: Load dataset\n",
    "\n",
    "The DeepFashion dataset used in this project is open-source and freely available:\n",
    "* Download the [DeepFashion dataset](https://drive.google.com/file/d/0B7EVK8r0v71pa2EyNEJ0dE9zbU0/view?usp=sharing).  Unzip the folder and place it in this project's home directory, at the location `/img`.\n",
    "\n",
    "In the code cell below, we will write the file paths for the DeepFashion dataset in the numpy array `img_files` and check the size of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from glob import glob\n",
    "\n",
    "# !unzip img\n",
    "\n",
    "# load filenames for clothing images\n",
    "img_files = np.array(glob(\"img/*/*\"))\n",
    "\n",
    "# print number of images in each dataset\n",
    "print('There are %d total clothing images.' % len(img_files))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step2'></a>\n",
    "## Step 2: Pre-process and build training, testing, and validation data loaders\n",
    "\n",
    "The data has already been randomly partitioned off into training, testing, and validation datasets so all we need to do is load it into a dataframe and validate that the data is split in correct proportions.\n",
    "\n",
    "The images are then resized to 150 x 150 and centercropped to create an image tensor of size 150 x 150 x 3. They are initially 300 pixels in height and the aspect ratio is not altered. In the interest of time, this dataset will not be augmented by adding flipped/rotated images to the training set; although, that is an effective method to increase the size of the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df_full = pd.open_csv(\"data_attributes.csv\")\n",
    "\n",
    "df_train = df_full.loc[df_full['evaluation_status'] == 'train']][['img_path', 'category_values', 'attribute_values']]\n",
    "df_test = df_full.loc[df_full['evaluation_status'] == 'test']][['img_path', 'category_values', 'attribute_values']]\n",
    "df_val = df_full.loc[df_full['evaluation_status'] == 'val']][['img_path', 'category_values', 'attribute_values']]\n",
    "\n",
    "print('The training set has %d records.' % len(df_train))\n",
    "print('The testing set has %d records.' % len(df_test))\n",
    "print('The validation set has %d records.' % len(df_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from PIL import Image\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms as T\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# Set PIL to be tolerant of image files that are truncated.\n",
    "from PIL import ImageFile\n",
    "ImageFile.LOAD_TRUNCATED_IMAGES = True\n",
    "\n",
    "### DONE: Write data loaders for training, validation, and test sets\n",
    "## Specify appropriate transforms, and batch_sizes\n",
    "transform = T.Compose([T.Resize(150), T.CenterCrop(150), T.ToTensor()]) \n",
    "\n",
    "dataset_train = datasets.ImageFolder('img/train', transform=transform)\n",
    "dataset_valid = datasets.ImageFolder('img/valid', transform=transform)\n",
    "dataset_test = datasets.ImageFolder('img/test', transform=transform)\n",
    "\n",
    "loader_train = DataLoader(dataset_train, batch_size=1, shuffle=False)\n",
    "loader_valid = DataLoader(dataset_valid, batch_size=1, shuffle=False)\n",
    "loader_test = DataLoader(dataset_test, batch_size=1, shuffle=False)\n",
    "\n",
    "loaders_transfer = {'train': loader_train, 'valid': loader_valid, 'test': loader_test}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step3'></a>\n",
    "## Step 3: Train the model (using Transfer Learning)\n",
    "\n",
    "The FashionNet model is nearly identical to the VGG-16 model architecture, with the exception of the last convolutional layer. However, instead of introducing the additional complexities of the FashionNet model, this model can be simplified by simply retaining the attributes embedding from the dataset. The data will be filtered into 1,000, potentially relevant buckets across 5 attributes of clothing, namely its pattern, material, fit, cut, and style. All layers use Rectified Linear Units (ReLUs) for the reduction in training times as documented by Nair and Hinton. It will be interesting to test the trained model to see how the the training and validation loss function perform.\n",
    "\n",
    "> Vinod Nair and Geoffrey Hinton. [Rectified Linear Units Improve Restricted Boltzmann Machines](https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf). In _Proceedings of ICML_, 2010.\n",
    "\n",
    "An alternative could have been to use a pretrained VGG-19 model, which would yield an architecture similar to that described by Simonyan and Zisserman. The results attained by their model showed great promise for a similar image classification problem and it could have made sense to reuse the same architecture, and only modifying the final fully connected layer as done for the VGG-16 model in the cells below.\n",
    "\n",
    "> Karen Simonyan and Andrew Zisserman. [Very Deep Convolutional Neural Network Based Image Classification Using Small Training Sample Size](https://arxiv.org/pdf/1409.1556.pdf). In _Proceedings of ICLR_, 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision.models as models\n",
    "import torch.nn as nn\n",
    "import torch\n",
    "\n",
    "# The underlying network structure of FashionNet is identical to VGG-16\n",
    "model_transfer = models.vgg19(pretrained=True)\n",
    "\n",
    "for param in model_transfer.parameters():\n",
    "    param.requires_grad = False\n",
    "\n",
    "# The sixth, final convolutional layer will be adjusted to 1,000\n",
    "model_transfer.classifier[6] = nn.Linear(1000, 133)\n",
    "\n",
    "# check if CUDA is available\n",
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "# move to GPU\n",
    "if use_cuda:\n",
    "    model_transfer = model_transfer.cuda()\n",
    "\n",
    "# create a complete CNN\n",
    "model_transfer = Net()\n",
    "print(model_transfer)\n",
    "\n",
    "# check if CUDA is available\n",
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "# move tensors to GPU if CUDA is available\n",
    "if use_cuda:\n",
    "    model_transfer.cuda()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify Loss Function and Optimizer\n",
    "\n",
    "Use the next code cell to specify a [loss function](http://pytorch.org/docs/master/nn.html#loss-functions) and [optimizer](http://pytorch.org/docs/master/optim.html).  Save the chosen loss function as `criterion_transfer`, and the optimizer as `optimizer_transfer` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "## select loss function\n",
    "criterion_transfer = nn.CrossEntropyLoss()\n",
    "\n",
    "# check if CUDA is available\n",
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "# move loss function to GPU if CUDA is available\n",
    "if use_cuda:\n",
    "    criterion_transfer = criterion_transfer.cuda()\n",
    "\n",
    "## select optimizer\n",
    "optimizer_transfer = optim.SGD(model_transfer.parameters(), lr=0.001)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and Validate the Model\n",
    "\n",
    "The model is to be trained and validated below, with [the final model parameters](http://pytorch.org/docs/master/notes/serialization.html) to be saved at the filepath `'model_transfer.pt'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 25\n",
    "\n",
    "# train the model\n",
    "model_transfer = train(n_epochs, loaders_transfer, model_transfer, optimizer_transfer, criterion_transfer, use_cuda, 'model_transfer.pt')\n",
    "\n",
    "# load the model that got the best validation accuracy\n",
    "model_transfer.load_state_dict(torch.load('model_transfer.pt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Model\n",
    "\n",
    "The model can be validated against test data to calculate and print the test loss and accuracy. We should ensure that the test accuracy is greater than 80%, as the implementation in the FashionNet paper yielded an accuracy of 85%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(loaders, model, criterion, use_cuda):\n",
    "\n",
    "    # monitor test loss and accuracy\n",
    "    test_loss = 0.\n",
    "    correct = 0.\n",
    "    total = 0.\n",
    "\n",
    "    model.eval()\n",
    "    for batch_idx, (data, target) in enumerate(loaders['test']):\n",
    "        # move to GPU\n",
    "        if use_cuda:\n",
    "            data, target = data.cuda(), target.cuda()\n",
    "        # forward pass: compute predicted outputs by passing inputs to the model\n",
    "        output = model(data)\n",
    "        # calculate the loss\n",
    "        loss = criterion(output, target)\n",
    "        # update average test loss \n",
    "        test_loss = test_loss + ((1 / (batch_idx + 1)) * (loss.data - test_loss))\n",
    "        # convert output probabilities to predicted class\n",
    "        pred = output.data.max(1, keepdim=True)[1]\n",
    "        # compare predictions to true label\n",
    "        correct += np.sum(np.squeeze(pred.eq(target.data.view_as(pred))).cpu().numpy())\n",
    "        total += data.size(0)\n",
    "            \n",
    "    print('Test Loss: {:.6f}\\n'.format(test_loss))\n",
    "\n",
    "    print('\\nTest Accuracy: %2d%% (%2d/%2d)' % (\n",
    "        100. * correct / total, correct, total))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test(loaders_transfer, model_transfer, criterion_transfer, use_cuda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step4'></a>\n",
    "## Step 4: Create style vectors for each user\n",
    "\n",
    "This capability is the crux of a recommendation engine; it generates a feature vector for a particular user, based on images they have previously selected or liked, and subsequently compares future images to ascertain the similarity, or distance, from previous selections to recommend items that would be a good fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## load attribute labels and their mappings\n",
    "\n",
    "df_attributes = pd.read_csv('labels_attributes.csv')\n",
    "\n",
    "# list of attribute names and their corresponding indices\n",
    "attr_pattern = []\n",
    "attr_material = []\n",
    "attr_fit = []\n",
    "attr_cut = []\n",
    "attr_style = []\n",
    "\n",
    "for i in range(len(df_attributes)):\n",
    "    if df_attributes[['attribute_type_id']][i] == 1:\n",
    "        attr_pattern.append(df_attributes[['attribute_id']][i])\n",
    "    if df_attributes[['attribute_type_id']][i] == 2:\n",
    "        attr_material.append(df_attributes[['attribute_id']][i])\n",
    "    if df_attributes[['attribute_type_id']][i] == 3:\n",
    "        attr_fit.append(df_attributes[['attribute_id']][i])\n",
    "    if df_attributes[['attribute_type_id']][i] == 4:\n",
    "        attr_cut.append(df_attributes[['attribute_id']][i])\n",
    "    if df_attributes[['attribute_type_id']][i] == 5:\n",
    "        attr_style.append(df_attributes[['attribute_id']][i])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='step5'></a>\n",
    "## Step 5: Test out recommendations\n",
    "\n",
    "Test the recommender system on sample images. It would be good to understand the output and gauge its performance - regardless of which, it can tangibly be improved by:\n",
    "* data augmentation of the training dataset by adding flipped/rotated images would yield a much larger training set and ultimately give better results\n",
    "* further experimentation with CNN architectures could potentially lead to a more effective architecture with less overfitting\n",
    "* an increase in training epochs, given more time, would both grant the training algorithms more time to converge at the local minimum and help discover patterns in training that could aid in identifying points of improvement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "img = Image.open(urllib.request.urlopen('https://images.footballfanatics.com/FFImage/thumb.aspx?i=/productimages/_2510000/altimages/ff_2510691alt1_full.jpg'))\n",
    "\n",
    "plt.imshow(img)\n",
    "plt.show()\n",
    "\n",
    "transform = T.Compose([T.Resize(150), T.CenterCrop(150), T.ToTensor()])\n",
    "transformed_img = transform(img)\n",
    "\n",
    "# the images have to be loaded in to a range of [0, 1]\n",
    "# then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]\n",
    "normalize = T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
    "normalized_img = normalize(transformed_img)\n",
    "\n",
    "# model loading\n",
    "tensor_img = normalized_img.unsqueeze(0)\n",
    "\n",
    "# check if CUDA is available\n",
    "use_cuda = torch.cuda.is_available()\n",
    "\n",
    "# move image tensor to GPU if CUDA is available\n",
    "if use_cuda:\n",
    "    tensor_img = tensor_img.cuda()\n",
    "\n",
    "# make prediction by passing image tensor to model\n",
    "prediction = model_transfer(tensor_img)\n",
    "# convert predicted probabilities to class index\n",
    "tensor_prediction = torch.argmax(prediction)\n",
    "\n",
    "# move prediction tensor to CPU if CUDA is available\n",
    "if use_cuda:\n",
    "    tensor_prediction = tensor_prediction.cpu()\n",
    "\n",
    "predicted_class_index = int(np.squeeze(tensor_prediction.numpy()))\n",
    "\n",
    "class_out = class_names[predicted_class_index] # predicted class index\n",
    "\n",
    "# The output would then be compared against the user's style vector to rank against other potential outfits"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}