notebook/ML_movie_recommendation/movie_recommendation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 电影推荐案例\n",
    "\n",
    "在本案例中我们将会学习使用人工智能技术分析用户对电影的评分数据，并基于这个数据建立一个推荐系统，根据用户输入的一部感兴趣的电影，为其推荐其他可能感兴趣的电影。此案例中，我们使用的数据集是用户对电影的评分数据，包含用户数据、评分数据、电影数据。\n",
    "\n",
    "本案例将掌握如何使用机器学习算法全流程构建一个电影推荐系统的方案；掌握如何使用华为云ModelArts Notebook上传数据、执行Python代码；掌握如何载入、查阅、清洗、合并用户的数据，并计算物品相似度矩阵；掌握如何使用物品的相似度矩阵，为客户进行推荐其他物品。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 进入人工智能开发平台ModelArts界面\n",
    "\n",
    "这步教大家如何进入人工智能开发平台华为云ModelArts服务。\n",
    "\n",
    "第一步：点击“控制台”，如下图所示\n",
    "![title](img/enter_modelarts_step1.png)\n",
    "\n",
    "第二步：点击“所有服务”，如下图所示\n",
    "![title](img/enter_modelarts_step2.png)\n",
    "\n",
    "第三步：在“EI企业智能”大类下找到“ModelArts”，点击“ModelArts”，进入ModelArts服务主界面，如下图所示\n",
    "![title](img/enter_modelarts_step3.png)\n",
    "\n",
    "第四步：看到以下界面，说明成功进入了ModelArts服务主界面\n",
    "![title](img/ModelArts_0.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建ModelArts Notebook\n",
    "\n",
    "此步教大家如何在ModelArts中创建一个Notebook开发环境。ModelArts Notebook提供网页版的Python开发环境，无需用户自己搭建Python开发环境。\n",
    "\n",
    "第一步：点击ModelArts服务主界面中的“开发环境”，如下图所示\n",
    "![title](img/ModelArts_1.png)\n",
    "\n",
    "第二步：点击下图中的“创建”按钮\n",
    "![title](img/create_notebook_step2.png)\n",
    "\n",
    "第三步：填写创建Notebook所需的参数，并点击下一步，参数填写请参考下表：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|参数|说明|\n",
    "|-|-|\n",
    "|计费方式|按需计费|\n",
    "|名称|notebook的名称，如ai-course|\n",
    "|工作环境 | Python3|\n",
    "|资源池|选择“公共资源池”即可|\n",
    "|类型|本案例选择CPU环境即可|\n",
    "|规格|选择“2核&#124;8GiB”|\n",
    "|存储配置|选择EVS，磁盘规格5GB|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第四步：点击下图中的“立即创建”\n",
    "![title](img/create_notebook_step4.png)\n",
    "\n",
    "第五步：点击下图中的“返回Notebook列表”\n",
    "![title](img/create_notebook_step5.png)\n",
    "\n",
    "第六步：等待Notebook创建成功，创建成功后状态会变成“运行中”，如下图所示\n",
    "![title](img/create_notebook_step6.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 在ModelArts Notebook中创建一个Notebook Python开发环境\n",
    "\n",
    "第一步：点击下图所示的“打开”按钮，进入刚刚创建的Notebook\n",
    "![title](img/create_notebook_dev_step1.png)\n",
    "\n",
    "第二步：创建一个Notebook Python语言开发环境。先点击“New”按钮，然后创建 XGBoost-Sklearn 开发环境。\n",
    "\n",
    "第三步：重命名刚刚创建的Notebook Python开发环境。点击“Untitle”，如下图所示\n",
    "![title](img/create_notebook_dev_step2.png)\n",
    "\n",
    "第四步：填写名称。我们可以填写一个跟本实验相关的名称，然后点击“Rename”按钮，如下图所示\n",
    "![title](img/create_notebook_dev_step3.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 如何在Notebook Python开发环境中写代码并执行代码\n",
    "\n",
    "第一步：输入代码。我们打印一行字符串，如下图所示\n",
    "![title](img/type_code_step1.png)\n",
    "\n",
    "第二步：执行代码。代码输入完成后，点击Notebook界面上的“Run”按钮，就可以执行代码，如下图所示\n",
    "![title](img/type_code_step2.png)\n",
    "\n",
    "第三步：查看代码执行结果。在代码输入框下面，可以看到代码执行结果，如下图所示\n",
    "![title](img/type_code_step3.png)\n",
    "\n",
    "第四步：保存代码。代码编写完之后，我们点击下图所示的“保存”按钮，保存代码和代码执行结果，如下图所示\n",
    "![title](img/type_code_step4.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notebook Python开发环境终于准备好了，现在可以在Notebook Python开发环境写代码啦**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 案例配置信息填写\n",
    "\n",
    "案例中需要将运行结果上传至OBS中，我们需要设置以下相关参数（使用自己真实的桶名和唯一ID替换掉*号）：\n",
    "\n",
    "* BUCKET_NAME ： 自己的OBS桶名\n",
    "* UNIQUE_ID : 唯一ID，填写自己的学号或者IAM子账号名称"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET_NAME = '*' \n",
    "UNIQUE_ID = '*' \n",
    "OBS_BASE_PATH = BUCKET_NAME + '/' + UNIQUE_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 初始化ModelArts SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modelarts.session import Session\n",
    "session = Session()\n",
    "if session.region_name == 'cn-north-1':\n",
    "    bucket_path=\"ai-course-common-20/movie_recommendation/movie_recommendation.tar.gz\"\n",
    "elif session.region_name == 'cn-north-4':\n",
    "    bucket_path=\"ai-course-common-20-bj4/movie_recommendation/movie_recommendation.tar.gz\"\n",
    "else:\n",
    "    print(\"请更换地区到北京一或北京四\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  准备源代码和数据\n",
    "\n",
    "这一步准备案例所需的源代码和数据，相关资源已经保存在OBS中，我们通过ModelArts SDK将资源下载到本地，并解压到当前目录下。解压后，当前目录包含ml-100k目录，存有数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully download file ai-course-common-20-bj4/movie_recommendation/movie_recommendation.tar.gz from OBS to local ./movie_recommendation.tar.gz\n"
     ]
    }
   ],
   "source": [
    "session.download_data(bucket_path=bucket_path, path=\"./movie_recommendation.tar.gz\")\n",
    "# 使用tar命令解压资源包\n",
    "!tar xf movie_recommendation.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入基本工具库\n",
    "执行下面方框中的这段代码，可以导入本次实验中使用的Python开发基本工具库。\n",
    "\n",
    "numpy是数据分处理工具,pandas是文件读取和数据处理工具，scipy是一个科学计算库，这里导入了cosine, correlation两种距离计算方法。\n",
    "\n",
    "此段代码只是引入Python包，无回显（代码执行输出）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import same usefull libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.metrics import pairwise_distances\n",
    "from scipy.spatial.distance import cosine, correlation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入并展示样本数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用pandas库导入用户的个人信息\n",
    "\n",
    "用户数据的字段描述如下：\n",
    "* user_id：用户ID\n",
    "* age：用户年龄\n",
    "* sex：性别\n",
    "* occupation：职业\n",
    "* zip_code：邮编\n",
    "\n",
    "评分数据的字段描述如下：\n",
    "* user_id：用户ID\n",
    "* movide_id：电影ID\n",
    "* rating：评分\n",
    "* unix_tiemstamp：评分时间\n",
    "\n",
    "电影数据的字段描述如下：\n",
    "* user_id：用户ID\n",
    "* movide_id：电影ID\n",
    "* rating：评分\n",
    "* unix_tiemstamp：评分时间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用户信息\n",
    "users_cols = ['user_id', 'age', 'sex', 'occupation', 'zip_code']\n",
    "users = pd.read_csv('./ml-100k/u.user', sep='|', names=users_cols, parse_dates=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印前5个用户的个人信息，可以看到用户个人信息包含用户ID（user_id）、年龄(age)、性别(sex)、职业(occupation)、邮编(zip_code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>M</td>\n",
       "      <td>technician</td>\n",
       "      <td>85711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>F</td>\n",
       "      <td>other</td>\n",
       "      <td>94043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>23</td>\n",
       "      <td>M</td>\n",
       "      <td>writer</td>\n",
       "      <td>32067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>M</td>\n",
       "      <td>technician</td>\n",
       "      <td>43537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>33</td>\n",
       "      <td>F</td>\n",
       "      <td>other</td>\n",
       "      <td>15213</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  age sex  occupation zip_code\n",
       "0        1   24   M  technician    85711\n",
       "1        2   53   F       other    94043\n",
       "2        3   23   M      writer    32067\n",
       "3        4   24   M  technician    43537\n",
       "4        5   33   F       other    15213"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印数据表格的大小，可以看到这是一个 943x5的矩阵， 其中943代表有943个用户，5代表每个用户有5项信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(943, 5)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用pandas库导入用户的评分信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Ratings\n",
    "rating_cols = ['user_id', 'movie_id', 'rating', 'unix_timestamp']\n",
    "ratings = pd.read_csv('./ml-100k/u.data', sep='\\t', names=rating_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印前5个评分信息，可以看到评分信息包含用户ID（user_id）、电影ID(movide_id)、评分(rating)、评分时间(unix_tiemstamp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>unix_timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  unix_timestamp\n",
       "0      196       242       3       881250949\n",
       "1      186       302       3       891717742\n",
       "2       22       377       1       878887116\n",
       "3      244        51       2       880606923\n",
       "4      166       346       1       886397596"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印数据表格的大小，可以看到这是一个 10000x4的矩阵， 其中10000代表有10000条评论，4代表每个评论有5项信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 4)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用pandas库导入电影的信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Movies\n",
    "movie_cols = ['movie_id', 'title', 'release_date', 'video_release_date', 'imdb_url']\n",
    "movies = pd.read_csv('./ml-100k/u.item', sep='|', names=movie_cols, usecols=range(5), encoding='latin-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印前5个电影信息，可以看到电影信息包含电影ID(movide_id)、电影名称(title)、发布时间(release_date)、视频发布时间(video_release_date)、评论网站URL链接(imdb_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>release_date</th>\n",
       "      <th>video_release_date</th>\n",
       "      <th>imdb_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Toy%20Story%2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>GoldenEye (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?GoldenEye%20(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Four Rooms (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Four%20Rooms%...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Get Shorty (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Get%20Shorty%...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Copycat (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Copycat%20(1995)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id              title release_date  video_release_date  \\\n",
       "0         1   Toy Story (1995)  01-Jan-1995                 NaN   \n",
       "1         2   GoldenEye (1995)  01-Jan-1995                 NaN   \n",
       "2         3  Four Rooms (1995)  01-Jan-1995                 NaN   \n",
       "3         4  Get Shorty (1995)  01-Jan-1995                 NaN   \n",
       "4         5     Copycat (1995)  01-Jan-1995                 NaN   \n",
       "\n",
       "                                            imdb_url  \n",
       "0  http://us.imdb.com/M/title-exact?Toy%20Story%2...  \n",
       "1  http://us.imdb.com/M/title-exact?GoldenEye%20(...  \n",
       "2  http://us.imdb.com/M/title-exact?Four%20Rooms%...  \n",
       "3  http://us.imdb.com/M/title-exact?Get%20Shorty%...  \n",
       "4  http://us.imdb.com/M/title-exact?Copycat%20(1995)  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印数据表格的大小，可以看到这是一个 1682x5的矩阵， 其中1682代表有1682部电影，5代表每部电影有5项信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1682, 5)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据合并\n",
    "\n",
    "把电影数据表、评论数据表、用户信息数据表进行合并，最后得到一张数据信息总表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merging movie data with their ratings\n",
    "movie_ratings = pd.merge(movies, ratings)\n",
    "# merging movie_ratings data with the User's dataframe\n",
    "df = pd.merge(movie_ratings, users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看数据信息总表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>release_date</th>\n",
       "      <th>video_release_date</th>\n",
       "      <th>imdb_url</th>\n",
       "      <th>user_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>unix_timestamp</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Toy%20Story%2...</td>\n",
       "      <td>308</td>\n",
       "      <td>4</td>\n",
       "      <td>887736532</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Get Shorty (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Get%20Shorty%...</td>\n",
       "      <td>308</td>\n",
       "      <td>5</td>\n",
       "      <td>887737890</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>Copycat (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Copycat%20(1995)</td>\n",
       "      <td>308</td>\n",
       "      <td>4</td>\n",
       "      <td>887739608</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>Twelve Monkeys (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Twelve%20Monk...</td>\n",
       "      <td>308</td>\n",
       "      <td>4</td>\n",
       "      <td>887738847</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>Babe (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://us.imdb.com/M/title-exact?Babe%20(1995)</td>\n",
       "      <td>308</td>\n",
       "      <td>5</td>\n",
       "      <td>887736696</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                  title release_date  video_release_date  \\\n",
       "0         1       Toy Story (1995)  01-Jan-1995                 NaN   \n",
       "1         4      Get Shorty (1995)  01-Jan-1995                 NaN   \n",
       "2         5         Copycat (1995)  01-Jan-1995                 NaN   \n",
       "3         7  Twelve Monkeys (1995)  01-Jan-1995                 NaN   \n",
       "4         8            Babe (1995)  01-Jan-1995                 NaN   \n",
       "\n",
       "                                            imdb_url  user_id  rating  \\\n",
       "0  http://us.imdb.com/M/title-exact?Toy%20Story%2...      308       4   \n",
       "1  http://us.imdb.com/M/title-exact?Get%20Shorty%...      308       5   \n",
       "2  http://us.imdb.com/M/title-exact?Copycat%20(1995)      308       4   \n",
       "3  http://us.imdb.com/M/title-exact?Twelve%20Monk...      308       4   \n",
       "4     http://us.imdb.com/M/title-exact?Babe%20(1995)      308       5   \n",
       "\n",
       "   unix_timestamp  age sex occupation zip_code  \n",
       "0       887736532   60   M    retired    95076  \n",
       "1       887737890   60   M    retired    95076  \n",
       "2       887739608   60   M    retired    95076  \n",
       "3       887738847   60   M    retired    95076  \n",
       "4       887736696   60   M    retired    95076  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印数据总表的大小，可以看到这是一个 10000x12的矩阵， 其中10000代表有10000条评论，12代表每条评论有12项属性，包括电影ID，电影信息，用户ID，评分，用户信息等"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 12)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据清洗\n",
    "\n",
    "去除一些无效或不需要的信息，比如video_release_date、imdb_url、unix_timestamp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pre-processing\n",
    "# dropping colums that aren't needed\n",
    "df.drop(df.columns[[3, 4, 7]], axis=1, inplace=True)\n",
    "ratings.drop(\"unix_timestamp\", inplace=True, axis=1)\n",
    "movies.drop(movies.columns[[3, 4]], inplace=True, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看新的数据信息总表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>release_date</th>\n",
       "      <th>user_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>308</td>\n",
       "      <td>4</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>Get Shorty (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>308</td>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>Copycat (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>308</td>\n",
       "      <td>4</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>Twelve Monkeys (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>308</td>\n",
       "      <td>4</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>Babe (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>308</td>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>retired</td>\n",
       "      <td>95076</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                  title release_date  user_id  rating  age sex  \\\n",
       "0         1       Toy Story (1995)  01-Jan-1995      308       4   60   M   \n",
       "1         4      Get Shorty (1995)  01-Jan-1995      308       5   60   M   \n",
       "2         5         Copycat (1995)  01-Jan-1995      308       4   60   M   \n",
       "3         7  Twelve Monkeys (1995)  01-Jan-1995      308       4   60   M   \n",
       "4         8            Babe (1995)  01-Jan-1995      308       5   60   M   \n",
       "\n",
       "  occupation zip_code  \n",
       "0    retired    95076  \n",
       "1    retired    95076  \n",
       "2    retired    95076  \n",
       "3    retired    95076  \n",
       "4    retired    95076  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建用户-电影评分矩阵\n",
    "\n",
    "根据评分数据表(ratings),创建用户-电影评分矩阵 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pivot Table(This creates a matrix of users and movie_ratings)\n",
    "ratings_matrix = ratings.pivot_table(index=['movie_id'], columns=['user_id'], values='rating').reset_index(drop=True)\n",
    "ratings_matrix.fillna(0, inplace=True)\n",
    "cmu = ratings_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看用户-电影评分矩阵，其中每一行代表每部电影来自所有用户的评分；每一列代表每个用户对所有电影的评分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>user_id</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>934</th>\n",
       "      <th>935</th>\n",
       "      <th>936</th>\n",
       "      <th>937</th>\n",
       "      <th>938</th>\n",
       "      <th>939</th>\n",
       "      <th>940</th>\n",
       "      <th>941</th>\n",
       "      <th>942</th>\n",
       "      <th>943</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 943 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "user_id  1    2    3    4    5    6    7    8    9    10   ...  934  935  936  \\\n",
       "0        5.0  4.0  0.0  0.0  4.0  4.0  0.0  0.0  0.0  4.0  ...  2.0  3.0  4.0   \n",
       "1        3.0  0.0  0.0  0.0  3.0  0.0  0.0  0.0  0.0  0.0  ...  4.0  0.0  0.0   \n",
       "2        4.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0  4.0   \n",
       "3        3.0  0.0  0.0  0.0  0.0  0.0  5.0  0.0  0.0  4.0  ...  5.0  0.0  0.0   \n",
       "4        3.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0   \n",
       "\n",
       "user_id  937  938  939  940  941  942  943  \n",
       "0        0.0  4.0  0.0  0.0  5.0  0.0  0.0  \n",
       "1        0.0  0.0  0.0  0.0  0.0  0.0  5.0  \n",
       "2        0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "3        0.0  0.0  0.0  2.0  0.0  0.0  0.0  \n",
       "4        0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "\n",
       "[5 rows x 943 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cmu.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印数据总表的大小，可以看到这是一个 1682x943的矩阵， 其中1682代表有1682部电影，943代表有943个用户"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建电影的相似矩阵\n",
    "\n",
    "根据943位用户对每部电影的评分，创建1682部电影的相似矩阵，矩阵大小为1682x1682的形状。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cosine Similarity(Creates a cosine matrix of similaraties ..... which is the pairwise distances\n",
    "# between two items )\n",
    "\n",
    "movie_similarity = 1 - pairwise_distances(ratings_matrix.values, metric=\"cosine\")\n",
    "np.fill_diagonal(movie_similarity, 0)\n",
    "ratings_matrix = pd.DataFrame(movie_similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看电影相似矩阵，以第三行，第二列为例，数值为0.273069，这个值代表第二部电影与第三部电影的相似度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1672</th>\n",
       "      <th>1673</th>\n",
       "      <th>1674</th>\n",
       "      <th>1675</th>\n",
       "      <th>1676</th>\n",
       "      <th>1677</th>\n",
       "      <th>1678</th>\n",
       "      <th>1679</th>\n",
       "      <th>1680</th>\n",
       "      <th>1681</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.402382</td>\n",
       "      <td>0.330245</td>\n",
       "      <td>0.454938</td>\n",
       "      <td>0.286714</td>\n",
       "      <td>0.116344</td>\n",
       "      <td>0.620979</td>\n",
       "      <td>0.481114</td>\n",
       "      <td>0.496288</td>\n",
       "      <td>0.273935</td>\n",
       "      <td>...</td>\n",
       "      <td>0.035387</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.035387</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.047183</td>\n",
       "      <td>0.047183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.402382</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.273069</td>\n",
       "      <td>0.502571</td>\n",
       "      <td>0.318836</td>\n",
       "      <td>0.083563</td>\n",
       "      <td>0.383403</td>\n",
       "      <td>0.337002</td>\n",
       "      <td>0.255252</td>\n",
       "      <td>0.171082</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.078299</td>\n",
       "      <td>0.078299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.330245</td>\n",
       "      <td>0.273069</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.324866</td>\n",
       "      <td>0.212957</td>\n",
       "      <td>0.106722</td>\n",
       "      <td>0.372921</td>\n",
       "      <td>0.200794</td>\n",
       "      <td>0.273669</td>\n",
       "      <td>0.158104</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.032292</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.096875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.454938</td>\n",
       "      <td>0.502571</td>\n",
       "      <td>0.324866</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.334239</td>\n",
       "      <td>0.090308</td>\n",
       "      <td>0.489283</td>\n",
       "      <td>0.490236</td>\n",
       "      <td>0.419044</td>\n",
       "      <td>0.252561</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.094022</td>\n",
       "      <td>0.094022</td>\n",
       "      <td>0.037609</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.056413</td>\n",
       "      <td>0.075218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.286714</td>\n",
       "      <td>0.318836</td>\n",
       "      <td>0.212957</td>\n",
       "      <td>0.334239</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.037299</td>\n",
       "      <td>0.334769</td>\n",
       "      <td>0.259161</td>\n",
       "      <td>0.272448</td>\n",
       "      <td>0.055453</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.094211</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1682 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       0         1         2         3         4         5         6     \\\n",
       "0  0.000000  0.402382  0.330245  0.454938  0.286714  0.116344  0.620979   \n",
       "1  0.402382  0.000000  0.273069  0.502571  0.318836  0.083563  0.383403   \n",
       "2  0.330245  0.273069  0.000000  0.324866  0.212957  0.106722  0.372921   \n",
       "3  0.454938  0.502571  0.324866  0.000000  0.334239  0.090308  0.489283   \n",
       "4  0.286714  0.318836  0.212957  0.334239  0.000000  0.037299  0.334769   \n",
       "\n",
       "       7         8         9     ...      1672  1673      1674      1675  \\\n",
       "0  0.481114  0.496288  0.273935  ...  0.035387   0.0  0.000000  0.000000   \n",
       "1  0.337002  0.255252  0.171082  ...  0.000000   0.0  0.000000  0.000000   \n",
       "2  0.200794  0.273669  0.158104  ...  0.000000   0.0  0.000000  0.000000   \n",
       "3  0.490236  0.419044  0.252561  ...  0.000000   0.0  0.094022  0.094022   \n",
       "4  0.259161  0.272448  0.055453  ...  0.000000   0.0  0.000000  0.000000   \n",
       "\n",
       "       1676  1677  1678  1679      1680      1681  \n",
       "0  0.035387   0.0   0.0   0.0  0.047183  0.047183  \n",
       "1  0.000000   0.0   0.0   0.0  0.078299  0.078299  \n",
       "2  0.032292   0.0   0.0   0.0  0.000000  0.096875  \n",
       "3  0.037609   0.0   0.0   0.0  0.056413  0.075218  \n",
       "4  0.000000   0.0   0.0   0.0  0.000000  0.094211  \n",
       "\n",
       "[5 rows x 1682 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_matrix.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1682, 1682)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 根据电影的相似矩阵，推荐电影\n",
    "\n",
    "当用户查看了 Copycat (1995)，那么根据电影的相似矩阵，推荐与 Copycat (1995) 近似分数比较高的电影。\n",
    "\n",
    "具体如下：\n",
    "\n",
    "根据电影名 Copycat (1995)， 查询电影信息表 (movies)中的index序号"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# user_inp=input('Enter the reference movie title based on which recommendations are to be made: ')\n",
    "user_inp = \"Copycat (1995)\"\n",
    "inp = movies[movies['title'] == user_inp].index.tolist()\n",
    "inp = inp[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据电影index序号，去查电影的相似矩阵，得到1682部电影的相似值，并打印表格中的5部电影的相似值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>release_date</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>0.286714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>GoldenEye (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>0.318836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Four Rooms (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>0.212957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Get Shorty (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>0.334239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Copycat (1995)</td>\n",
       "      <td>01-Jan-1995</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id              title release_date  similarity\n",
       "0         1   Toy Story (1995)  01-Jan-1995    0.286714\n",
       "1         2   GoldenEye (1995)  01-Jan-1995    0.318836\n",
       "2         3  Four Rooms (1995)  01-Jan-1995    0.212957\n",
       "3         4  Get Shorty (1995)  01-Jan-1995    0.334239\n",
       "4         5     Copycat (1995)  01-Jan-1995    0.000000"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies['similarity'] = ratings_matrix.iloc[inp]\n",
    "movies.columns = ['movie_id', 'title', 'release_date', 'similarity']\n",
    "movies.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把相似值进行排序，并打印最相似的5部电影"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recommended movies based on your choice of  Copycat (1995) : \n",
      "      movie_id                              title release_date  similarity\n",
      "218       219  Nightmare on Elm Street, A (1984)  01-Jan-1984    0.472725\n",
      "53         54                    Outbreak (1995)  01-Jan-1995    0.472399\n",
      "233       234                        Jaws (1975)  01-Jan-1975    0.450780\n",
      "52         53        Natural Born Killers (1994)  01-Jan-1994    0.445242\n",
      "97         98   Silence of the Lambs, The (1991)  01-Jan-1991    0.440996\n"
     ]
    }
   ],
   "source": [
    "recommended_movies = movies.sort_values([\"similarity\"], ascending=False)[1:6]\n",
    "\n",
    "print(\"Recommended movies based on your choice of \", user_inp, \": \\n\", recommended_movies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 保存结果至OBS\n",
    "\n",
    "我们将推荐的前5部电影的信息保存到文本文件中，并上传到OBS，以便以后查看。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 写入本地文件\n",
    "\n",
    "将电影的信息写入到文本文件中。会打印成功保存的信息。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully saved!\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "if not os.path.exists('results'):\n",
    "    os.mkdir('results') # 创建本地保存路径\n",
    "\n",
    "with open('./results/recommended_movies.txt', 'w') as f:\n",
    "    f.write(str(recommended_movies)) # 写入本地文本文件\n",
    "    \n",
    "print('Successfully saved!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 上传文件至OBS\n",
    "\n",
    "使用ModelArts SDK上传本地文件至OBS。可以看到上传成功的日志。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully upload file ./results/recommended_movies.txt to OBS mr-l-bucket/hw40161597/movie_recommendation/results\n"
     ]
    }
   ],
   "source": [
    "session.upload_data(bucket_path=OBS_BASE_PATH + '/movie_recommendation/results/', path='./results/recommended_movies.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"font-family: Arial; font-size:1.4em;color:gold;\">总结：该案例所在的OBS存储路径下，results目录下，有模型文件recommended_movies.txt。</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "XGBoost-Sklearn",
   "language": "python",
   "name": "xgboost-sklearn"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}