Intro_to_Recommendation_Systems/Intro_Recommender.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Recommendation Systems\n",
    "\n",
    "In this tutorial we used a [deep autoencoder](https://arxiv.org/abs/1708.01715) to create a recommender system with the [Netflix dataset](https://netflixprize.com/). We show how to create an API to wrap the autoencoder and operationalize the recommendation system.\n",
    "\n",
    "The deep autoencoder in this tutorial is done with [PyTorch](http://pytorch.org/) and is based on [this repo](https://github.com/NVIDIA/DeepRecommender) by NVIDIA. For the API, we used [Flask](http://flask.pocoo.org/) and [CherryPy](http://cherrypy.org/).\n",
    "\n",
    "## Overview of Recommendation Systems\n",
    "\n",
    "A [recommendation system](https://en.wikipedia.org/wiki/Recommender_system) seeks to understand the user preferences with the objective of recommending him or her items. These systems has become increasingly popular in recent years, in parallel with the growth of internet retailers like Amazon, Netflix or Spotify. Recommender systems are used in a variety of areas including movies, music, news, books, research articles, search queries, social tags, and products in general. In terms of business impact, according to a recent [study](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.895.3477&rep=rep1&type=pdf) from Wharton School, recommendation engines can cause a 25% lift in number of views and 35% lift in number of items purchased. So it is worth to understand these systems.   \n",
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"data/shut_up.gif\" width=300px/>\n",
    "</p>\n",
    "\n",
    "Generally speaking, there are 3 methodologies for recommendation systems: collaborative filtering, content-based filtering and hybrid recommender systems.\n",
    "\n",
    "[**Collaborative filtering**](https://en.wikipedia.org/wiki/Collaborative_filtering) collects large amounts of information on users’ behaviors, activities or preferences in order to predict what users will like based on their similarity to other users. This information can be explicit, where the user provides directly the ratings of the items, or implicit, where the ratings have to be extracted for the implicit user behavior, like number of views, likes, purchases, etc. \n",
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"https://upload.wikimedia.org/wikipedia/commons/5/52/Collaborative_filtering.gif\" width=300px/>\n",
    "</p>\n",
    "\n",
    "Mathematically, it is based on inferring the missing entries in an `mxn` matrix, `R`, whose `(i, j)` entry describes the ratings given by the `ith` user to the `jth` item. The performance is then measured using Root Mean Squared Error (RMSE). This problem has been addressed in a variaety of ways. Traditional methods include low rank matrix factorization like [Alternating Least Squares](https://datajobs.com/data-science-repo/Recommender-Systems-[Netflix].pdf), which became popular due to its implementation in [Spark](https://spark.apache.org/docs/latest/mllib-collaborative-filtering.html). Methods based on Deep Learning have grown in popularity recently, some techniques use embbedings and RELU activations like in [Youtube](http://dl.acm.org/citation.cfm?doid=2959100.2959190), recurrent neural networks like in [Hedasi et al., 2016](http://arxiv.org/abs/1511.06939) or deep autoencoders like the present tutorial.\n",
    "\n",
    "**Content-based filtering** take into account contextual user factors such as location, date of purchase, user demographics and item factors like price, brand, type of item, etc, to recommend items that are similar to those that a user liked in the past.\n",
    "\n",
    "The system creates a content-based profile of users based on a weighted vector of item features. The weights denote the importance of each feature to the user and can be computed from individually rated content vectors using a variety of techniques. Simple approaches use the average values of the rated item vector while other sophisticated methods use machine learning techniques such as Bayesian Classifiers, cluster analysis, decision trees, and neural networks in order to estimate the probability that the user is going to like the item.\n",
    "\n",
    "**Hybrid recommender systems** combine multiple techniques together to achieve some synergy between them. They can use aspects from collaborative filtering, content-based filtering, knowledge based and demographics. They have proved to be very effective in some cases, like the [Bellkor solution](https://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf), winner of the Netflix prize, the [Netflix Recommender System](http://delivery.acm.org/10.1145/2850000/2843948/a13-gomez-uribe.pdf) or [LightFM](https://github.com/lyst/lightfm). \n",
    "\n",
    "In this tutorial we focused on collaborative filtering with autoencoders. We used data of the Netflix prize and \n",
    " created an API so the model can be operationalized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OS:  linux\n",
      "Python:  3.5.4 | packaged by conda-forge | (default, Nov  4 2017, 10:11:29) \n",
      "[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]\n",
      "PyTorch:  0.3.0.post4\n",
      "Numpy:  1.14.0\n",
      "Number of CPU processors:  24\n",
      "GPU:  ['Tesla M60', 'Tesla M60', 'Tesla M60', 'Tesla M60']\n",
      "GPU memory:  ['8123 MiB', '8123 MiB', '8123 MiB', '8123 MiB']\n",
      "CUDA:  CUDA Version 8.0.61\n",
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import aiohttp\n",
    "import asyncio\n",
    "import json\n",
    "import requests\n",
    "from utils import get_gpu_name, get_number_processors, get_gpu_memory, get_cuda_version\n",
    "from parameters import *\n",
    "from load_test import run_load_test\n",
    "\n",
    "print(\"OS: \", sys.platform)\n",
    "print(\"Python: \", sys.version)\n",
    "print(\"PyTorch: \", torch.__version__)\n",
    "print(\"Numpy: \", np.__version__)\n",
    "print(\"Number of CPU processors: \", get_number_processors())\n",
    "print(\"GPU: \", get_gpu_name())\n",
    "print(\"GPU memory: \", get_gpu_memory())\n",
    "print(\"CUDA: \", get_cuda_version())\n",
    "\n",
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset: Netflix\n",
    "\n",
    "This dataset was constructed to support participants in the [Netflix Prize](http://www.netflixprize.com). The movie rating files contain over 100 million ratings from 480 thousand randomly-chosen, anonymous Netflix customers over 17 thousand movie titles.  The data were collected between October, 1998 and December, 2005 and reflect the distribution of all ratings received during this period.  The ratings are on a scale from 1 to 5 (integral) stars.\n",
    "\n",
    "The dataset can be [downloaded here](http://academictorrents.com/details/9b13183dc4d60676b773c9e2cd6de5e5542cee9a). To uncompress it:\n",
    "\n",
    "```bash\n",
    "tar -xvf nf_prize_dataset.tar.gz\n",
    "tar -xf download/training_set.tar\n",
    "```\n",
    "\n",
    "When we download the data, there are two important files:\n",
    "\n",
    "1) The file `training_set.tar` is a tar of a directory containing 17770 files, one per movie.  The first line of each file contains the movie id followed by a colon.  Each subsequent line in the file corresponds to a rating from a customer and its date in the following format:\n",
    "\n",
    "`CustomerID, Rating, Date`\n",
    "- MovieIDs range from 1 to 17770 sequentially.\n",
    "- CustomerIDs range from 1 to 2649429, with gaps. There are 480189 users.\n",
    "- Ratings are on a five star (integral) scale from 1 to 5.\n",
    "- Dates have the format YYYY-MM-DD.\n",
    "\n",
    "2) Movie information in [`movie_titles.txt`](data/movie_titles.txt) is in the following format:\n",
    "\n",
    "`MovieID, YearOfRelease, Title`\n",
    "\n",
    "- MovieID do not correspond to actual Netflix movie ids or IMDB movie ids.\n",
    "- YearOfRelease can range from 1890 to 2005 and may correspond to the release of corresponding DVD, not necessarily its theaterical release.\n",
    "- Title in English is the Netflix movie.\n",
    "\n",
    "### Data prep\n",
    "\n",
    "The first step is to covert the data to the correct format for the autoencoder to read. This can take between 1 to 2 hours.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "%run ./DeepRecommender/data_utils/netflix_data_convert.py $NF_PRIZE_DATASET $NF_DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script splitted the data into train, test and validation set, creating files with three columns: `CustomerID,MovieID,Rating`. The data is splitted over time generating 4 datasets: Netflix 3months, Netflix 6 months, Netflix 1 year and Netflix full. Here there is a table with some details of each dataset:\n",
    "\n",
    "| Dataset  | Netflix 3 months | Netflix 6 months | Netflix 1 year | Netflix full |\n",
    "| -------- | ---------------- | ---------------- | ----------- |  ------------ |\n",
    "| Ratings train | 13,675,402 | 29,179,009 | 41,451,832 | 98,074,901 |\n",
    "| Users train | 311,315 |390,795  | 345,855 | 477,412 |\n",
    "| Items train | 17,736 |17,757  | 16,907 | 17,768 |\n",
    "| Time range train | 2005-09-01 to 2005-11-31 | 2005-06-01 to 2005-11-31 | 2004-06-01 to 2005-05-31 | 1999-12-01 to 2005-11-31\n",
    "|  |  |  |   | |\n",
    "| Ratings test | 2,082,559 | 2,175,535  | 3,888,684| 2,250,481 |\n",
    "| Users test | 160,906 | 169,541  | 197,951| 173,482 |\n",
    "| Items test | 17,261 | 17,290  | 16,506| 17,305 |\n",
    "| Time range test | 2005-12-01 to 2005-12-31 | 2005-12-01 to 2005-12-31 | 2005-06-01 to 2005-06-31 | 2005-12-01 to 2005-12-31\n",
    "\n",
    "Let's take a look at some of the files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1041739, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CustomerID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1549</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>5144</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>7716</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>8348</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>4635</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CustomerID  MovieID  Rating\n",
       "0           0     1549     1.0\n",
       "1           0     5144     2.0\n",
       "2           0     7716     3.0\n",
       "3           0     8348     3.0\n",
       "4           0     4635     2.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nf_3m_valid = os.path.join(NF_DATA, 'N3M_VALID', 'n3m.valid.txt')\n",
    "df = pd.read_csv(nf_3m_valid, names=['CustomerID','MovieID','Rating'], sep='\\t')\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1040820, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CustomerID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>159</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>4830</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1261</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>12058</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>13412</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CustomerID  MovieID  Rating\n",
       "0           0      159     4.0\n",
       "1           0     4830     1.0\n",
       "2           0     1261     3.0\n",
       "3           0    12058     3.0\n",
       "4           0    13412     2.0"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nf_3m_test = os.path.join(NF_DATA, 'N3M_TEST', 'n3m.test.txt')\n",
    "df2 = pd.read_csv(nf_3m_test, names=['CustomerID','MovieID','Rating'], sep='\\t')\n",
    "print(df2.shape)\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Autoencoder for Collaborative Filtering\n",
    "\n",
    "Once we have the data, let's explain in some detail the model that we are going to use. The [model](https://arxiv.org/abs/1708.01715), developed by NVIDIA folks, is a Deep autoencoder with 6 layers with non-linear activation function SELU (scaled exponential linear units), dropout and iterative dense refeeding.\n",
    "\n",
    "An autoencoder is a network which implements two transformations: $encode(x): R^n \\Rightarrow R^d$ and $decoder(z): R^d \\Rightarrow R^n$. The “goal” of autoencoder is to obtain a $d$ dimensional representation of data such that an error measure between $x$ and $f(x) = decode(encode(x))$ is minimized. In the next figure, the autocoder architecture proposed in the [paper](https://arxiv.org/abs/1708.01715) is showed. Encoder has 2 layers $e_1$ and $e_2$ and decoder has 2 layers $d_1$ and $d_2$. Dropout may be applied to coding layer $z$. In the paper, the authors show experiments with different number of layers, from 2 to 12 (see Table 2 in the original paper).\n",
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"./data/AutoEncoder.png\" width=350px/>\n",
    "</p>\n",
    "\n",
    "During the forward pass the model takes a user representation by his vector of ratings from the training set $x \\in R^n$, where $n$ is number of items. Note that $x$ is very sparse, while the output of the decoder, $y=f(x) \\in R^n$ is dense and contains the rating predictions for all items in the corpus. The loss is the root mean squared error (RMSE).\n",
    "\n",
    "One of the key ideas of the paper is dense re-feeding. Let's consider an idealized scenario with a perfect $f$. Then $f(x)_i = x_i ,\\forall i : x_i \\ne 0$ and $f(x)_i$ accurately predicts all user's future ratings. This means that if a user rates a new item $k$ (thereby creating a new vector $x'$) then $f(x)_k = x'_k$ and $f(x) = f(x')$. Therefore, the authors refeed the input in the autoencoder to augment the dataset. The method consists of the following steps:\n",
    "\n",
    "1. Given a sparse $x$, compute the forward pass to get $f(x)$ and the loss.\n",
    "\n",
    "2. Backpropagate the loss and update the weights.\n",
    "\n",
    "3. Treat $f(x)$ as a new example and compute $f(f(x))$\n",
    "\n",
    "4. Compute a second backward pass.\n",
    "\n",
    "Steps 3 and 4 can be repeated several times.\n",
    "\n",
    "Finally, the authors explore different non-linear [activation functions](https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py). They found that on this task ELU, SELU and LRELU, which have non-zero negative parts, perform much better than SIGMOID, RELU, RELU6, and TANH.\n",
    "\n",
    "Now, let's compute the training. The model parameters can be found in [parameters.py](parameters.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namespace(aug_step=1, batch_size=128, constrained=False, drop_prob=0.8, gpu_ids='0', hidden_layers='512,512,1024', logdir='model_save', lr=0.005, noise_prob=0.0, non_linearity_type='selu', num_epochs=10, optimizer='momentum', path_to_eval_data='Netflix/N3M_VALID', path_to_train_data='Netflix/N3M_TRAIN', skip_last_layer_nl=False, weight_decay=0.0)\n",
      "Loading training data from Netflix/N3M_TRAIN\n",
      "Data loaded\n",
      "Total items found: 311315\n",
      "Vector dim: 17736\n",
      "Loading eval data from Netflix/N3M_VALID\n",
      "******************************\n",
      "******************************\n",
      "[17736, 512, 512, 1024]\n",
      "Dropout drop probability: 0.8\n",
      "Encoder pass:\n",
      "torch.Size([512, 17736])\n",
      "torch.Size([512])\n",
      "torch.Size([512, 512])\n",
      "torch.Size([512])\n",
      "torch.Size([1024, 512])\n",
      "torch.Size([1024])\n",
      "Decoder pass:\n",
      "torch.Size([512, 1024])\n",
      "torch.Size([512])\n",
      "torch.Size([512, 512])\n",
      "torch.Size([512])\n",
      "torch.Size([17736, 512])\n",
      "torch.Size([17736])\n",
      "******************************\n",
      "******************************\n",
      "Using GPUs: [0]\n",
      "Doing epoch 0 of 10\n",
      "Total epoch 0 finished in 69.39120125770569 seconds with TRAINING RMSE loss: 1.1183533288603893\n",
      "Epoch 0 EVALUATION LOSS: 0.997187149064757\n",
      "Doing epoch 1 of 10\n",
      "Total epoch 1 finished in 69.05308318138123 seconds with TRAINING RMSE loss: 0.9789836858425376\n",
      "Epoch 1 EVALUATION LOSS: 0.9830844731127444\n",
      "Doing epoch 2 of 10\n",
      "Total epoch 2 finished in 68.9950942993164 seconds with TRAINING RMSE loss: 0.9593065493420863\n",
      "Epoch 2 EVALUATION LOSS: 0.98861261075587\n",
      "Doing epoch 3 of 10\n",
      "Total epoch 3 finished in 69.13668775558472 seconds with TRAINING RMSE loss: 0.9464339257342224\n",
      "Epoch 3 EVALUATION LOSS: 0.9762008394628371\n",
      "Doing epoch 4 of 10\n",
      "Total epoch 4 finished in 69.09636783599854 seconds with TRAINING RMSE loss: 0.9355610656442712\n",
      "Epoch 4 EVALUATION LOSS: 0.9808872814542353\n",
      "Doing epoch 5 of 10\n",
      "Total epoch 5 finished in 69.17095017433167 seconds with TRAINING RMSE loss: 0.9258845933913559\n",
      "Epoch 5 EVALUATION LOSS: 0.9842890565637975\n",
      "Doing epoch 6 of 10\n",
      "Total epoch 6 finished in 69.08278322219849 seconds with TRAINING RMSE loss: 0.9622985019130311\n",
      "Epoch 6 EVALUATION LOSS: 0.9838701840431763\n",
      "Doing epoch 7 of 10\n",
      "Total epoch 7 finished in 69.04743361473083 seconds with TRAINING RMSE loss: 0.9337027765558531\n",
      "Epoch 7 EVALUATION LOSS: 0.984365107262594\n",
      "Doing epoch 8 of 10\n",
      "Total epoch 8 finished in 69.1117844581604 seconds with TRAINING RMSE loss: 0.9205498934527924\n",
      "Epoch 8 EVALUATION LOSS: 0.9750949946703523\n",
      "Doing epoch 9 of 10\n",
      "Total epoch 9 finished in 69.13744783401489 seconds with TRAINING RMSE loss: 0.9099730840527471\n",
      "Epoch 9 EVALUATION LOSS: 0.9742177051690417\n",
      "Saving model to model_save/model.epoch_9\n",
      "Routine finished. Process time 2747.513389825821 s\n"
     ]
    }
   ],
   "source": [
    "%run ./DeepRecommender/run.py --gpu_ids $GPUS \\\n",
    "    --path_to_train_data $TRAIN \\\n",
    "    --path_to_eval_data $EVAL \\\n",
    "    --hidden_layers $HIDDEN \\\n",
    "    --non_linearity_type $ACTIVATION \\\n",
    "    --batch_size $BATCH_SIZE \\\n",
    "    --logdir $MODEL_OUTPUT_DIR \\\n",
    "    --drop_prob $DROPOUT \\\n",
    "    --optimizer $OPTIMIZER \\\n",
    "    --lr $LR \\\n",
    "    --weight_decay $WD \\\n",
    "    --aug_step $AUG_STEP \\\n",
    "    --num_epochs $EPOCHS "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "Now we are going to evaluate the model on the test set and compute the final loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namespace(constrained=False, drop_prob=0.8, hidden_layers='512,512,1024', non_linearity_type='selu', path_to_eval_data='Netflix/N3M_TEST', path_to_train_data='Netflix/N3M_TRAIN', predictions_path='preds.txt', save_path='model_save/model.epoch_9', skip_last_layer_nl=False)\n",
      "Loading training data\n",
      "Data loaded\n",
      "Total items found: 311315\n",
      "Vector dim: 17736\n",
      "Loading eval data\n",
      "******************************\n",
      "******************************\n",
      "[17736, 512, 512, 1024]\n",
      "Dropout drop probability: 0.8\n",
      "Encoder pass:\n",
      "torch.Size([512, 17736])\n",
      "torch.Size([512])\n",
      "torch.Size([512, 512])\n",
      "torch.Size([512])\n",
      "torch.Size([1024, 512])\n",
      "torch.Size([1024])\n",
      "Decoder pass:\n",
      "torch.Size([512, 1024])\n",
      "torch.Size([512])\n",
      "torch.Size([512, 512])\n",
      "torch.Size([512])\n",
      "torch.Size([17736, 512])\n",
      "torch.Size([17736])\n",
      "******************************\n",
      "******************************\n",
      "Loading model from: model_save/model.epoch_9\n",
      "Done: 0\n",
      "Done: 10000\n",
      "Done: 20000\n",
      "Done: 30000\n",
      "Done: 40000\n",
      "Done: 50000\n",
      "Done: 60000\n",
      "Done: 70000\n",
      "Done: 80000\n",
      "Done: 90000\n",
      "Done: 100000\n",
      "Done: 110000\n",
      "Done: 120000\n",
      "Routine finished. Process time 228.1412889957428 s\n"
     ]
    }
   ],
   "source": [
    "%run ./DeepRecommender/infer.py \\\n",
    "--path_to_train_data $TRAIN \\\n",
    "--path_to_eval_data $TEST \\\n",
    "--hidden_layers $HIDDEN \\\n",
    "--non_linearity_type $ACTIVATION \\\n",
    "--save_path  $MODEL_PATH \\\n",
    "--drop_prob $DROPOUT \\\n",
    "--predictions_path $INFER_OUTPUT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namespace(path_to_predictions='preds.txt', round=False)\n",
      "####################\n",
      "RMSE: 0.9746437597050387\n",
      "####################\n"
     ]
    }
   ],
   "source": [
    "%run ./DeepRecommender/compute_RMSE.py --path_to_predictions=$INFER_OUTPUT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API\n",
    "\n",
    "Let's create an API for the recommender. The first step is to take data fom a customer and make it available as a json to perform the query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Year</th>\n",
       "      <th>Title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>Dinosaur Planet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>Isle of Man TT 2004 Review</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>Character</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1994.0</td>\n",
       "      <td>Paula Abdul's Get Up &amp; Dance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>The Rise and Fall of ECW</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID    Year                         Title\n",
       "0        1  2003.0               Dinosaur Planet\n",
       "1        2  2004.0    Isle of Man TT 2004 Review\n",
       "2        3  1997.0                     Character\n",
       "3        4  1994.0  Paula Abdul's Get Up & Dance\n",
       "4        5  2004.0      The Rise and Fall of ECW"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles = pd.read_csv(MOVIE_TITLES, names=['MovieID','Year','Title'], encoding = \"latin\")\n",
    "titles.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CustomerID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>159</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>4830</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1261</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>12058</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>13412</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>15887</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>16111</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>16445</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CustomerID  MovieID  Rating\n",
       "0           0      159     4.0\n",
       "1           0     4830     1.0\n",
       "2           0     1261     3.0\n",
       "3           0    12058     3.0\n",
       "4           0    13412     2.0\n",
       "5           0    15887     3.0\n",
       "6           0    16111     3.0\n",
       "7           0    16445     2.0"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = df2[df2['CustomerID'] == 0]\n",
    "target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CustomerID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>159</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Gentlemen of Fortune</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>4830</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Poirot: Peril at End House</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1261</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Mojados: Through the Night</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>12058</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Being Mick: You Would If You Could</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>13412</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Billy Elliot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>15887</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Van Helsing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>16111</td>\n",
       "      <td>3.0</td>\n",
       "      <td>The First World War: The Complete Series</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>16445</td>\n",
       "      <td>2.0</td>\n",
       "      <td>House of 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CustomerID  MovieID  Rating                                     Title\n",
       "0           0      159     4.0                      Gentlemen of Fortune\n",
       "1           0     4830     1.0                Poirot: Peril at End House\n",
       "2           0     1261     3.0                Mojados: Through the Night\n",
       "3           0    12058     3.0        Being Mick: You Would If You Could\n",
       "4           0    13412     2.0                              Billy Elliot\n",
       "5           0    15887     3.0                               Van Helsing\n",
       "6           0    16111     3.0  The First World War: The Complete Series\n",
       "7           0    16445     2.0                                House of 1"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_customer = pd.merge(target, titles, on='MovieID', how='left', suffixes=('_',''))\n",
    "df_customer.drop(['Title_','Year'], axis=1, inplace=True)\n",
    "df_customer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input of the API is a json of one customer, right now it's for `CustomerID` 0, containing `MovieID` and its rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{159: 4.0,\n",
       " 1261: 3.0,\n",
       " 4830: 1.0,\n",
       " 12058: 3.0,\n",
       " 13412: 2.0,\n",
       " 15887: 3.0,\n",
       " 16111: 3.0,\n",
       " 16445: 2.0}"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_query = df_customer.drop(['CustomerID','Title'], axis=1).set_index('MovieID')\n",
    "dict_query = df_query.to_dict()['Rating']\n",
    "dict_query"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The API is defined in [api.py](api.py). The model is loaded when the server starts and stays in memory. The main function `/recommend` just receives the request `dict_query` as a json, computes the inference with the autoencoder and returns another json.\n",
    "\n",
    "Let's test the end point. For that, in a command line start the api:\n",
    "```bash\n",
    "python api.py\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "end_point = 'http://127.0.0.1:5000/'\n",
    "end_point_recommend = \"http://127.0.0.1:5000/recommend\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Yeah, yeah, I highly recommend it"
     ]
    }
   ],
   "source": [
    "!curl $end_point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a call with the `dict_query` to the recommender API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "{\n",
      "  \"16445\": \"1.830115\",\n",
      "  \"1261\": \"3.0182471\",\n",
      "  \"4830\": \"2.5257375\",\n",
      "  \"13412\": \"3.113011\",\n",
      "  \"15887\": \"1.8397967\",\n",
      "  \"16111\": \"2.7220972\",\n",
      "  \"12058\": \"0.794388\",\n",
      "  \"159\": \"3.4304914\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "headers = {'Content-type':'application/json'}\n",
    "res = requests.post(end_point_recommend, data=json.dumps(dict_query), headers=headers)\n",
    "print(res.ok)\n",
    "print(json.dumps(res.json(), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load test\n",
    "We are going to test the load stability of the API, for it we create an asynchronous client to handle the queries. The code can be found in [load_test.py](load_test.py). We can play with the number of calls `NUM` and number of concurrent calls `CONCURRENT`. In my machine using 10 calls with 2 concurrent, I get a response time of 4ms. I got the same result making 100 calls with 2 concurrent. If I change to 100 calls and too many concurrent concurrent calls, 20 for example, the time gets to 12ms.\n",
    "\n",
    "In a normal application, the requests don't usually come from the same computer where the API is hosted. So to the response time, you'll have to add the latency of communicating client and server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM = 10\n",
    "CONCURRENT = 2\n",
    "VERBOSE = True\n",
    "payload = {13:5.0, 191:5.0, 209:5.0}\n",
    "payload_list = [payload]*NUM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating a client session outside of coroutine\n",
      "client_session: <aiohttp.client.ClientSession object at 0x7f3faeba0278>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "Response status: 200\n",
      "{'191': '4.1924996', '13': '3.1280162', '209': '3.3941908'}\n",
      "CPU times: user 20 ms, sys: 4 ms, total: 24 ms\n",
      "Wall time: 35.5 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Run:\n",
    "with aiohttp.ClientSession() as session:  # We create a persistent connection\n",
    "    loop = asyncio.get_event_loop()\n",
    "    calc_routes = loop.run_until_complete(run_load_test(end_point_recommend, payload_list, session, CONCURRENT, VERBOSE))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this notebook we showed how to create a recommendation system with a deep autoencoder and how to make the model accessible through an API. We see that this model is fast enough to be served in any production system with an inference time of 4ms and that can be easily trained in a GPU in about 45min. \n",
    "\n",
    "Would you recommend this tutorial? Well, now you don't need to waist your time thinking about this, it's funnier to gather some data, compute the autoencoder and get the result.\n",
    "\n",
    "Happy coding!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}