PyLadiesDC-Books_to_Movies_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome, PyLadies!\n",
    "\n",
    "This tutorial is intended to demo a basic data mining pipeline:\n",
    "1. Define your problem\n",
    "2. Identify your data\n",
    "3.  ,-> Explore your data ------------------,\n",
    "4.  '-- Normalize and Clean your data <--'\n",
    "5. Extract information\n",
    "\n",
    "If you're a pro, feel free to modify as you go or carve your own path. Everything you need outside of open source can be found here: https://github.com/ravedawg/PyLadies/ If you share, please cite. Thanks!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the following libraries work\n",
    "# To troubleshoot: open the command line and check that it's installed, e.g. \"which numpy\"\n",
    "# if it is not installed, simply install e.g. \"pip install pandas\"\n",
    "# if it is installed, you might need to check your paths\n",
    "from IPython.display import Image\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.pyplot import figure\n",
    "plt.style.use('ggplot')\n",
    "import ast\n",
    "from sklearn import preprocessing\n",
    "from sklearn.preprocessing import scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Define your problem: What makes a good book-to-movie? \n",
    "Seriously though, why do so many good books become terrible movies? Take this conundrum, for example:\n",
    "\n",
    "Mystic River. Written by Dennis Lehane, starring Sean Penn and Tim Robbins, directed by Clint Eastwood. Award-winning book, and award-winning movie.\n",
    "\n",
    "Live By Night. Written by Dennis Lehane, starring Ben Affleck and Zoe Saldana, directed by Ben Affleck. Award-winning book, sh!t movie. Even after it was adapted to film by the author himself.\n",
    "\n",
    "I don't expect to solve all the mysteries here, but I'd like to get some general trends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cwd = os.getcwd()\n",
    "movie_posters = cwd+\"/img/MoviePosters.png\"\n",
    "Image(movie_posters, width=400, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Identify your data. \n",
    "\n",
    "Let's use a subset of a dataset that I created, using goodreads and imdb.\n",
    "\n",
    "For book data, I used the goodreads API https://www.goodreads.com/api, which I found pretty robust and easy-to-use, as far as API's go.\n",
    "\n",
    "For movie data, IMDb has a huge volume of public data. You can download it in large chunks, https://www.imdb.com/interfaces/, or go through the API like I did (http://www.omdbapi.com/).\n",
    "\n",
    "Holy crap there are a lot of books and movies! To link the two datasets, I went to wikipedia.\n",
    "\n",
    "https://en.wikipedia.org/w/index.php?title=Category:American_novels_adapted_into_films&pageuntil=Burnt+Offerings+%28Marasco+novel%29#mw-pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Explore your data.\n",
    "\n",
    "Start by importing the data and examining its structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data! Data! Data!\n",
    "film_adaptations = []\n",
    "with open('/Users/ravenholm/Books-to-Movies.txt') as f:\n",
    "    for line in f.read().splitlines():\n",
    "        d = (ast.literal_eval(line))\n",
    "        film_adaptations.append(d)\n",
    "    \n",
    "# let's take a look at the keys to get a sense of structure\n",
    "for key in film_adaptations[0].keys():\n",
    "    print(key,type(key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's shift to pandas for more flexible handling and better features (like sample!).\n",
    "df = pd.DataFrame(film_adaptations)\n",
    "\n",
    "# print a row, and print the top (head) of a column\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And lets examine some random samples and practice accessing values based on key.\n",
    "for i in range(5):\n",
    "    row=df.sample()\n",
    "    \n",
    "    print('{} was written by {} in {}, and had an average rating of {}\\n'.format(\n",
    "                        row['title'].values[0],\n",
    "                        row['author_name'].values[0],\n",
    "                        row['publication_date'].values[0][-4:],\n",
    "                        row['average_rating'].values[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For practice, look up the goodreads id of your favorite film-adapted book\n",
    "# https://www.goodreads.com/book/\n",
    "# The goodreads id preceeds the book title in the web address, highlighted below in blue\n",
    "Image(cwd+\"/img/how_to_find_gr_id.png\", width=500, height=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access the row of the desired film adaptation\n",
    "\n",
    "grID = '41865'  # <------ edit this variable with the goodreads ID you just looked up\n",
    "\n",
    "print('what the goodreads_id column looks like:', df['goodreads_id'].head())\n",
    "\n",
    "row = df.loc[df['goodreads_id'] == grID]\n",
    "\n",
    "# add a line to display other data about the film that might tell you whether or not it was \n",
    "# a good film adaptation.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What do other book ratings look like?\n",
    "df['average_rating'] = df['average_rating'].astype(float)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Plot goodreads ratings\n",
    "plt.rcParams[\"figure.figsize\"] = (18,3)\n",
    "df['average_rating'].plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Normalize and clean your data.\n",
    "\n",
    "Normalizing, cleanding, transforming... These terms are thrown used loosely but have very concrete meanings in practice. For more: https://www.statisticshowto.com/normalized/\n",
    "\n",
    "If you notice errors or something missing, please please please mention it in the comments so the quality of this dataset can be improved!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure numeric columns are formatted as numeric\n",
    "df['ratings_count'] = df['ratings_count'].astype(float)\n",
    "df['m_imdb_Rating'] = pd.to_numeric(df['m_imdb_Rating'],errors='coerce')\n",
    "df['m_imdb_Votes'] = pd.to_numeric(df['m_imdb_Votes'],errors='coerce')\n",
    "\n",
    "# convert date time objects\n",
    "df['publication_date'] = pd.to_datetime(df['publication_date'], errors='coerce')\n",
    "df['mReleased'] = pd.to_datetime(df['mReleased'], errors='coerce')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Plot goodreads ratings against imdb ratings\n",
    "df.plot(x='average_rating',y='m_imdb_Rating',style='o',figsize=(10,10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's explore the outliers\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Although these classics made ok movies, there are really too few book reviews for goodreads\n",
    "# scores to have any credibility. Let's establish a review count threshold for quality.\n",
    "\n",
    "df = df[df['ratings_count'].astype(float)>19] # for goodreads\n",
    "\n",
    "# add a similar threshold for imdb ratings, then replot\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# add a point to indicate your favorite adaptation to see where it falls\n",
    "grID = grID\n",
    "row = df.loc[df['goodreads_id'] == grID]\n",
    "plt.plot(row['average_rating'],row['m_imdb_Rating'], marker='x', markersize=15, color=\"blue\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before we move on to determining what is statistically significant across columns, we need to  \n",
    "# transform each numeric column to be reflective of what is statistically significant\n",
    "\n",
    "# normalize the data\n",
    "df['scaled_gr'] = scale(df['average_rating'].astype(float))\n",
    "\n",
    "# Plot goodreads ratings\n",
    "plt.rcParams[\"figure.figsize\"] = (18,3)\n",
    "df['scaled_gr'].plot(kind='bar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Repeat for the imdb ratings\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Replot with your point and see if it changes anything\n",
    "df.plot(x='scaled_gr',y='scaled_imdb',style='o',figsize=(10,10))\n",
    "row = df.loc[df['goodreads_id'] == grID]\n",
    "\n",
    "plt.plot(row['scaled_gr'],row['scaled_imdb'], marker='x', markersize=15, color=\"blue\")\n",
    "plt.ylabel('scaled imdb ratings')\n",
    "plt.legend(['Adaptations',str(row['title'])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Extract Information\n",
    "\n",
    "Let's start by analyzing what makes a good film adaptation, vice a bad one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create two more columns, labeling each row as either a good/bad book, and a good/bad movie\n",
    "df.loc[((df['scaled_imdb']>=0) & (df['scaled_gr']<0)),'Adaptation_Category'] = 1 # good movie, bad book\n",
    "df.loc[((df['scaled_imdb']>=0) & (df['scaled_gr']>=0)),'Adaptation_Category'] = 2 # good movie, good book\n",
    "df.loc[((df['scaled_imdb']<0) & (df['scaled_gr']>=0)),'Adaptation_Category'] = 3 # bad movie, good book\n",
    "df.loc[((df['scaled_imdb']<0) & (df['scaled_gr']<0)),'Adaptation_Category'] = 4 # bad movie, bad book\n",
    "\n",
    "print(df['Adaptation_Category'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Chart out the numeric columns\n",
    "df[['Adaptation_Category','ratings_count','m_imdb_Votes']].groupby(\"Adaptation_Category\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "goodreads_id <class 'str'>\n",
    "title <class 'str'>\n",
    "average_rating <class 'str'>\n",
    "ratings_count <class 'str'>\n",
    "publication_date <class 'str'>\n",
    "author_name <class 'str'>\n",
    "author_id <class 'str'>\n",
    "wiki_book_link <class 'str'>\n",
    "mTitle <class 'str'>\n",
    "mRated <class 'str'>\n",
    "mReleased <class 'str'>\n",
    "mLength <class 'str'>\n",
    "mGenre <class 'str'>\n",
    "mDirector <class 'str'>\n",
    "mWriters <class 'str'>\n",
    "mActors <class 'str'>\n",
    "mAwards <class 'str'>\n",
    "m_imdb_Rating <class 'str'>\n",
    "m_imdb_Votes <class 'str'>\n",
    "m_imdb_ID <class 'str'>\n",
    "mStudio <class 'str'>\n",
    "mPlot <class 'str'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks so much for attending! Please see my github https://github.com/ravedawg/PyLadies/ to leave a comment, to follow, or to collaborate!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}