slides.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 1 + 1 = 1 or Record Deduplication with Python\n",
    "Flávio Juvenal  \n",
    "[@flaviojuvenal](https://twitter.com/flaviojuvenal)  \n",
    "\n",
    "<p>\n",
    "    Partner at <a href=\"http://www.vinta.com.br?dedupe-slides=1\"><img class=\"vinta_logo\" style=\"display: inline-block; margin-left: 0;\" src=\"vinta.png\" width=300></a>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<p style=\"font-size: 2em; text-weight: bold\">\n",
    "    Slides available at: <a href=\"http://bit.ly/pybay-dupe\">bit.ly/pybay-dupe</a> (open as Desktop)\n",
    "</p>\n",
    "<p>\n",
    "    Jupyter Notebook source at: <a href=\"https://github.com/vintasoftware/deduplication-slides/\">github.com/vintasoftware/deduplication-slides/</a>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "Real world data is inputted by people and often it's:\n",
    "* Not reviewed\n",
    "* Not linked with related data\n",
    "* Not properly normalized by the input system\n",
    "* Or simply it's incorrectly inputted because people make mistakes: typos, mishearing, miscalculation, misinterpretation, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This causes the following problems on data:\n",
    "* lack of unique identifiers (making difficult detect duplicates in a dataset or to link with other datasets)\n",
    "* duplications (e.g. multiple records refer to a single person)\n",
    "* inconsistencies (e.g. a person appears with multiple addresses)\n",
    "* bad formatting (e.g. birth dates appear with multiple formats like DD/MM/YY and YYYY-MM-DD)\n",
    "\n",
    "All of that affects the ability to properly **extract knowledge** from one or more datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The solution is to perform **[Record Linkage](https://en.wikipedia.org/wiki/Record_linkage)**. It works by joining records in a **fuzzy way** using data like names, addresses, phone numbers, dates, etc.\n",
    "\n",
    "The term Record Linkage is most used when the linkage is applied to multiple datasets, like joining a Restaurant Food Inspection dataset with an Employee Wage dataset. In fact, [someone did just that](https://www.enigma.com/blog/on-wages-and-hygiene-surfacing-bad-management-in-public-data).\n",
    "\n",
    "Record Linkage is also known as Data Matching, Entity Resolution, and [other names](https://en.wikipedia.org/wiki/Record_linkage#Naming_conventions)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What we'll discuss here is a specific application of Record Linkage, called **Deduplication**, which is applying Record Linkage on a single dataset against itself to **find which records are duplicates**. Here's a very simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import warnings; warnings.simplefilter('ignore')\n",
    "import logging; logging.disable(level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = [\n",
    "    (\"Chin's\",\"3200 Las Vegas Boulevard\",\"New York\"),\n",
    "    (\"Chin Bistro\",\"3200 Las Vegas Blvd.\",\"New York\"),\n",
    "    (\"Bistro\",\"3400 Las Vegas Blvd.\",\"New York City\"),\n",
    "    (\"Bistro\",\"3400 Las Vegas B.\",\"NYC\"),\n",
    "]\n",
    "\n",
    "df = pd.DataFrame(data, columns=['restaurant', 'address', 'city'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>restaurant</th>\n",
       "      <th>address</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Chin's</td>\n",
       "      <td>3200 Las Vegas Boulevard</td>\n",
       "      <td>New York</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Chin Bistro</td>\n",
       "      <td>3200 Las Vegas Blvd.</td>\n",
       "      <td>New York</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Bistro</td>\n",
       "      <td>3400 Las Vegas Blvd.</td>\n",
       "      <td>New York City</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bistro</td>\n",
       "      <td>3400 Las Vegas B.</td>\n",
       "      <td>NYC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    restaurant                   address           city\n",
       "0       Chin's  3200 Las Vegas Boulevard       New York\n",
       "1  Chin Bistro      3200 Las Vegas Blvd.       New York\n",
       "2       Bistro      3400 Las Vegas Blvd.  New York City\n",
       "3       Bistro         3400 Las Vegas B.            NYC"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "A good deduplication on the data above would find that:\n",
    "  - `(0, 1)` are duplicates\n",
    "  - `(2, 3)` are duplicates \n",
    "  - but `(0, 1)` and `(2, 3)` are different, despite being similar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The process for deduplicating a dataset usually is:\n",
    "  0. Preprocessing\n",
    "    * input: dataset \n",
    "    * output: cleaned dataset\n",
    "  1. Indexing\n",
    "    * output: pairs to compare\n",
    "  2. Comparison\n",
    "    * output: comparison vectors\n",
    "  3. Classification\n",
    "    * output: matching/nonmatching pairs\n",
    "  4. Clustering \n",
    "    * output: unique record clusters\n",
    "    \n",
    "Let's explore each of those steps!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 0/4 - Preprocessing\n",
    "Without unique identifiers, we need to match records by fuzzy data like:\n",
    "- Names\n",
    "- Addresses\n",
    "- Phone Numbers\n",
    "- Dates\n",
    "\n",
    "So it's important to **clean** them for matching."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cleaning names (companies or people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's use string functions and regexes to **normalize** names and remove undesired variations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import re\n",
    "import pprint\n",
    "\n",
    "\n",
    "company_names = company_names_dirty = [\n",
    "    'APPLE COMPUTER INC',\n",
    "    'APPLE COMPUTER, INC.',\n",
    "    'APPLE INC',\n",
    "    'Apple Computer',\n",
    "    'Apple Computer Co.',\n",
    "    'Apple Computer Company',\n",
    "    'Apple Computer Inc',\n",
    "    'Apple Computer Incorporated',\n",
    "    'Apple Computer, Inc.',\n",
    "    'Apple Inc',\n",
    "    'Apple Inc.',\n",
    "    'Apple, Inc.'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lower case:\n",
      "['apple computer inc',\n",
      " 'apple computer, inc.',\n",
      " 'apple inc',\n",
      " 'apple computer',\n",
      " 'apple computer co.',\n",
      " 'apple computer company',\n",
      " 'apple computer inc',\n",
      " 'apple computer incorporated',\n",
      " 'apple computer, inc.',\n",
      " 'apple inc',\n",
      " 'apple inc.',\n",
      " 'apple, inc.']\n"
     ]
    }
   ],
   "source": [
    "print(\"Lower case:\")\n",
    "company_names = [c.lower() for c in company_names]\n",
    "pprint.pprint(company_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remove irrelevant separators:\n",
      "['apple computer inc',\n",
      " 'apple computer  inc ',\n",
      " 'apple inc',\n",
      " 'apple computer',\n",
      " 'apple computer co ',\n",
      " 'apple computer company',\n",
      " 'apple computer inc',\n",
      " 'apple computer incorporated',\n",
      " 'apple computer  inc ',\n",
      " 'apple inc',\n",
      " 'apple inc ',\n",
      " 'apple  inc ']\n"
     ]
    }
   ],
   "source": [
    "print(\"Remove irrelevant separators:\")\n",
    "irrelevant_regex = re.compile(r'[^a-z0-9\\s]')\n",
    "company_names = [irrelevant_regex.sub(' ', c) for c in company_names]\n",
    "pprint.pprint(company_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remove multi-spaces:\n",
      "['apple computer inc',\n",
      " 'apple computer inc',\n",
      " 'apple inc',\n",
      " 'apple computer',\n",
      " 'apple computer co',\n",
      " 'apple computer company',\n",
      " 'apple computer inc',\n",
      " 'apple computer incorporated',\n",
      " 'apple computer inc',\n",
      " 'apple inc',\n",
      " 'apple inc',\n",
      " 'apple inc']\n"
     ]
    }
   ],
   "source": [
    "print(\"Remove multi-spaces:\")\n",
    "multispace_regex = re.compile(r'\\s\\s+')\n",
    "company_names = [multispace_regex.sub(' ', c).strip() for c in company_names]\n",
    "pprint.pprint(company_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remove stopwords:\n",
      "['apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple',\n",
      " 'apple']\n"
     ]
    }
   ],
   "source": [
    "print(\"Remove stopwords:\")\n",
    "business_stopwords = {  # suppose we got this from somewhere\n",
    "    'computer',\n",
    "    'co',\n",
    "    'company',\n",
    "    'corp',\n",
    "    'corporation',\n",
    "    'inc',\n",
    "    'incorporated',\n",
    "    'llc',\n",
    "    #...\n",
    "}\n",
    "\n",
    "company_names = [\n",
    "    ' '.join([c_part for c_part in c.split() if c_part not in business_stopwords])\n",
    "    for c in company_names\n",
    "]\n",
    "pprint.pprint(company_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can also use the natural language processing library [**probablepeople**](https://github.com/datamade/probablepeople) to **parse company names** and extract just the parts we want (or break into parts and match by part on the comparison step later):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Apple', 'CorporationName'),\n",
       " ('Computer', 'CorporationName'),\n",
       " ('Incorporated', 'CorporationLegalType')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import probablepeople as pp\n",
    "\n",
    "pp.parse(\"Apple Computer Incorporated\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['APPLE', 'COMPUTER'],\n",
      " ['APPLE', 'COMPUTER,'],\n",
      " ['APPLE'],\n",
      " ['Apple', 'Computer'],\n",
      " ['Apple', 'Computer'],\n",
      " ['Apple', 'Computer'],\n",
      " ['Apple', 'Computer'],\n",
      " ['Apple', 'Computer'],\n",
      " ['Apple', 'Computer,'],\n",
      " ['Apple'],\n",
      " ['Apple'],\n",
      " ['Apple,']]\n"
     ]
    }
   ],
   "source": [
    "company_names_alternative_1 = [\n",
    "    [\n",
    "        parsed_value\n",
    "        for parsed_value, parsed_type\n",
    "        in pp.parse(c)\n",
    "        if parsed_type == 'CorporationName'\n",
    "    ]\n",
    "    for c in company_names_dirty\n",
    "]\n",
    "pprint.pprint(company_names_alternative_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "probablepeople, as the name suggests, can parse **people** names too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Mr.', 'PrefixMarital'),\n",
       " ('Guido', 'GivenName'),\n",
       " ('van', 'Surname'),\n",
       " ('Rossum', 'Surname')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pp.parse('Mr. Guido van Rossum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But performance isn't great from non-English names. So you can try [**nameparser**](https://github.com/derek73/python-nameparser/), which is **not probabilistic**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Flávio', 'CorporationName'),\n",
       " ('Juvenal', 'CorporationName'),\n",
       " ('da', 'CorporationName'),\n",
       " ('Silva', 'CorporationName'),\n",
       " ('Jr', 'CorporationName')]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<HumanName : [\n",
       "\ttitle: '' \n",
       "\tfirst: 'Flávio' \n",
       "\tmiddle: 'Juvenal' \n",
       "\tlast: 'da Silva' \n",
       "\tsuffix: 'Jr'\n",
       "\tnickname: ''\n",
       "]>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from nameparser import HumanName\n",
    "from IPython.display import display\n",
    "\n",
    "display(pp.parse('Flávio Juvenal da Silva Jr'))\n",
    "display(HumanName('Flávio Juvenal da Silva Jr'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If it's useful to **ignore accents**, try [**unidecode**](https://github.com/avian2/unidecode):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ASCII transliteration:\n",
      "Flávio ->\n",
      "Flavio\n"
     ]
    }
   ],
   "source": [
    "import unidecode\n",
    "\n",
    "print(\"ASCII transliteration:\")\n",
    "brazilian_name = \"Flávio\"\n",
    "print(brazilian_name, \"->\")\n",
    "print(unidecode.unidecode(brazilian_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It's important to consider how the data values were inputted. If the value was inputted by someone while hearing from someone else, try **phonetic encoding** with [**doublemetaphone**](https://github.com/dedupeio/doublemetaphone) to normalize words that sound similar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('ANTR', 'ANTRF')\n",
      "('ANTR', 'ANTR')\n"
     ]
    }
   ],
   "source": [
    "from doublemetaphone import doublemetaphone\n",
    "\n",
    "print(doublemetaphone('Andrew'))\n",
    "print(doublemetaphone('André'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "If data was obtained via Optical Character Recognition (OCR), techniques exist to correct common misspellings that OCRs make (like C to L, S to 5, q to g, etc). See Christen, page 47 [2]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cleaning addresses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Geocoding street addresses, i.e., converting them to **latitude/longitude** is very useful for matching, because geocoders usually clean irrelevant address variations. Also, having lat/lng enables the calculation of geometric distances between addresses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import geocoder\n",
    "\n",
    "full_addresses = [\n",
    "    \"2066 Crist Drive, 94024, Los Altos, CALIFORNIA, US\",\n",
    "    \"2066 Crist Dr, 94024, LOS ALTOS, CALIFORNIA, US\",\n",
    "    \"20863 Stevens Creek Blvd., Suite 300, 95015, CUPERTINO, CALIFORNIA, US\",\n",
    "    \"20863 STEVENS CREEK BLVD STE 330, 95014, Lupertino, CALIFORNIA, US\",\n",
    "    \"10260 Bandley Drive, 95014, Cupertino, CALIFORNIA, US\",\n",
    "    \"10260 Bandley Dr., 95014, Cupertino, CALIFORNIA, US\",\n",
    "    \"20525 MARIANI AVENUE, 95014, CUPERTINO, CALIFORNIA, US\",\n",
    "    \"20525 Mariani Ave, 95014, CUPERTINO, CALIFORNIA, US\",\n",
    "    \"1 Infinite Loop, 95014, Cupertino, CALIFORNIA, US\",\n",
    "    \"One Infinite Loop,, 95014, Cupertino, CALIFORNIA, US\",\n",
    "    \"One Apple Park Way, 95014, Cupertino, CALIFORNIA, US\",\n",
    "    \"1 Apple Park Way, 95014, Cupertino, CALIFORNIA, US\",\n",
    "]\n",
    "\n",
    "full_addresses_latlng = []\n",
    "with requests.Session() as session:\n",
    "    for a in full_addresses:\n",
    "        a_geocoded = geocoder.google(a, session=session)\n",
    "        full_addresses_latlng.append(a_geocoded.latlng)\n",
    "\n",
    "address_latlng = list(zip(full_addresses, full_addresses_latlng))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(('20863 Stevens Creek Blvd., Suite 300, 95015, CUPERTINO, CALIFORNIA, US',\n",
      "  [37.3241563, -122.0387297]),\n",
      " ('20863 STEVENS CREEK BLVD STE 330, 95014, Lupertino, CALIFORNIA, US',\n",
      "  [37.3241563, -122.0387297]))\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pprint.pprint((address_latlng[2], address_latlng[3]))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Google geocoder is able to:\n",
    "- Ignore lower/upper case difference\n",
    "- Ignore the difference between Suite 300 vs 330 (building entrance may be the same)\n",
    "- Consider 'Lupertino' as 'Cupertino'\n",
    "- Expand abbreviations, like 'STE' vs 'Suite'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(('1 Infinite Loop, 95014, Cupertino, CALIFORNIA, US',\n",
      "  [37.3324975, -122.0289785]),\n",
      " ('One Infinite Loop,, 95014, Cupertino, CALIFORNIA, US',\n",
      "  [37.33182, -122.03118]))\n"
     ]
    }
   ],
   "source": [
    "pprint.pprint((address_latlng[8], address_latlng[9]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "However, it gets slighly different addresses on the case of '1' vs 'One'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note geocoding from web APIs is slow and has quota limits! Alternatively, you can build **your own geocoder** with:\n",
    "- https://github.com/openstreetmap/Nominatim\n",
    "- https://github.com/pelias/pelias\n",
    "- https://github.com/komoot/photon\n",
    "- https://postgis.net/docs/Extras.html\n",
    "- https://github.com/openvenues/lieu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If geocoding is inviable, we can use the library [**pypostal**](https://github.com/openvenues/pypostal) to **normalize** addresses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 Infinite Loop, 95014, Cupertino, CALIFORNIA, US\n",
      "['1 infinite loop 95014 cupertino california us']\n",
      "\n",
      "One Infinite Loop,, 95014, Cupertino, CALIFORNIA, US\n",
      "['1 infinite loop 95014 cupertino california us']\n"
     ]
    }
   ],
   "source": [
    "from postal.expand import expand_address\n",
    "\n",
    "print(full_addresses[8])\n",
    "pprint.pprint(expand_address(full_addresses[8]))\n",
    "print()\n",
    "print(full_addresses[9])\n",
    "pprint.pprint(expand_address(full_addresses[9]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20863 Stevens Creek Blvd., Suite 300, 95015, CUPERTINO, CALIFORNIA, US\n",
      "['20863 stevens creek boulevard suite 300 95015 cupertino california us']\n",
      "\n",
      "20863 STEVENS CREEK BLVD STE 330, 95014, Lupertino, CALIFORNIA, US\n",
      "['20863 stevens creek boulevard suite 330 95014 lupertino california us',\n",
      " '20863 stevens creek boulevard sainte 330 95014 lupertino california us']\n"
     ]
    }
   ],
   "source": [
    "print(full_addresses[2])\n",
    "pprint.pprint(expand_address(full_addresses[2]))\n",
    "print()\n",
    "print(full_addresses[3])\n",
    "pprint.pprint(expand_address(full_addresses[3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "From [DataMade](https://datamade.us/), the creators of probablepeople, there's a [**usaddress**](https://github.com/datamade/usaddress) **parser**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2066 Crist Drive, 94024, Los Altos, CALIFORNIA, US\n",
      "[('2066', 'AddressNumber'),\n",
      " ('Crist', 'StreetName'),\n",
      " ('Drive,', 'StreetNamePostType'),\n",
      " ('94024,', 'SubaddressIdentifier'),\n",
      " ('Los', 'PlaceName'),\n",
      " ('Altos,', 'PlaceName'),\n",
      " ('CALIFORNIA,', 'PlaceName'),\n",
      " ('US', 'StateName')]\n",
      "\n",
      "2066 Crist Dr, 94024, LOS ALTOS, CALIFORNIA, US\n",
      "[('2066', 'AddressNumber'),\n",
      " ('Crist', 'StreetName'),\n",
      " ('Dr,', 'StreetNamePostType'),\n",
      " ('94024,', 'ZipCode'),\n",
      " ('LOS', 'PlaceName'),\n",
      " ('ALTOS,', 'PlaceName'),\n",
      " ('CALIFORNIA,', 'PlaceName'),\n",
      " ('US', 'StateName')]\n"
     ]
    }
   ],
   "source": [
    "import usaddress\n",
    "\n",
    "print(full_addresses[0])\n",
    "pprint.pprint(usaddress.parse(full_addresses[0]))\n",
    "print()\n",
    "print(full_addresses[1])\n",
    "pprint.pprint(usaddress.parse(full_addresses[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cleaning phone numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[**phonenumbers**](https://github.com/daviddrysdale/python-phonenumbers) library can **normalize** phone numbers from many countries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phone number normalization:\n",
      "(541) 555-3010 ->\n",
      "+15415553010\n"
     ]
    }
   ],
   "source": [
    "import phonenumbers\n",
    "\n",
    "print(\"Phone number normalization:\")\n",
    "phone = \"(541) 555-3010\"\n",
    "print(phone, '->')\n",
    "print(\n",
    "    phonenumbers.format_number(\n",
    "        phonenumbers.parse(phone, 'US'),\n",
    "        phonenumbers.PhoneNumberFormat.E164)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cleaning dates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[**dateparser**](https://github.com/scrapinghub/dateparser) library can **guess** date formats and parse them as datetime objects. It can even guess DD/MM or MM/DD by the language:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 10:00:00\n",
      "2018-01-10 10:00:00\n",
      "{'date_obj': datetime.datetime(2018, 1, 10, 10, 0),\n",
      " 'locale': 'pt',\n",
      " 'period': 'day'}\n"
     ]
    }
   ],
   "source": [
    "import dateparser\n",
    "\n",
    "print(dateparser.parse(\"at 10/1/2018 10am\"))\n",
    "print(dateparser.parse(\"às 10/1/2018 10:00\"))\n",
    "pprint.pprint(dateparser.DateDataParser().get_date_data(\"às 10/1/2018 10:00\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Check also [**dateutil parser**](https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.parse) with `fuzzy=True`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Cleaning a real dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now we know the tools, let's **grab a dataset to preprocess and go on the other deduplication steps**. Our dataset is based* on the [Restaurant](https://www.cs.utexas.edu/users/ml/riddle/data.html) dataset, a well-known dataset used by researchers. It contains 881 restaurant records and contains 150 duplicates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\\* We've introduced some changes that you can check by doing a `diff restaurant.original.csv restaurant.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>phone</th>\n",
       "      <th>type</th>\n",
       "      <th>cluster</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>310/246-1501</td>\n",
       "      <td>american</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>310-246-1501</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>310-246-1501</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art's delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>818/762-1221</td>\n",
       "      <td>american</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>818-762-1221</td>\n",
       "      <td>delis</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>818-762-1221</td>\n",
       "      <td>delis</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel-air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>310/472-1211</td>\n",
       "      <td>californian</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel-air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>310-472-1211</td>\n",
       "      <td>californian</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel-air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "      <td>(310) 472-1211</td>\n",
       "      <td>american</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city  \\\n",
       "0  arnie morton's of chicago       435 s. la cienega blv.  los angeles   \n",
       "1  arnie morton's of chicago      435 s. la cienega blvd.  los angeles   \n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles   \n",
       "3         art's delicatessen          12224 ventura blvd.  studio city   \n",
       "4                 art's deli          12224 ventura blvd.  studio city   \n",
       "5                 art's deli          12224 ventura blvd.  los angeles   \n",
       "6              hotel bel-air         701 stone canyon rd.      bel air   \n",
       "7              bel-air hotel         701 stone canyon rd.      bel air   \n",
       "8                    bel-air        701 stone canyon road      bel air   \n",
       "\n",
       "            phone         type  cluster  \n",
       "0    310/246-1501     american        0  \n",
       "1    310-246-1501  steakhouses        0  \n",
       "2    310-246-1501  steakhouses        0  \n",
       "3    818/762-1221     american        1  \n",
       "4    818-762-1221        delis        1  \n",
       "5    818-762-1221        delis        1  \n",
       "6    310/472-1211  californian        2  \n",
       "7    310-472-1211  californian        2  \n",
       "8  (310) 472-1211     american        2  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_with_truth = pd.read_csv('restaurant.csv', skip_blank_lines=True)\n",
    "df_with_truth.head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The dataset comes with the true matches indicated by the 'cluster' column. We'll remove it. We'll also remove the 'phone' to makes things more difficult:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton's of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art's delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>delis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art's deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>delis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel-air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel-air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel-air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city  \\\n",
       "0  arnie morton's of chicago       435 s. la cienega blv.  los angeles   \n",
       "1  arnie morton's of chicago      435 s. la cienega blvd.  los angeles   \n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles   \n",
       "3         art's delicatessen          12224 ventura blvd.  studio city   \n",
       "4                 art's deli          12224 ventura blvd.  studio city   \n",
       "5                 art's deli          12224 ventura blvd.  los angeles   \n",
       "6              hotel bel-air         701 stone canyon rd.      bel air   \n",
       "7              bel-air hotel         701 stone canyon rd.      bel air   \n",
       "8                    bel-air        701 stone canyon road      bel air   \n",
       "\n",
       "          type  \n",
       "0     american  \n",
       "1  steakhouses  \n",
       "2  steakhouses  \n",
       "3     american  \n",
       "4        delis  \n",
       "5        delis  \n",
       "6  californian  \n",
       "7  californian  \n",
       "8     american  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_with_truth.drop(columns=['cluster', 'phone'])\n",
    "df.head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "By running `info` method, we assert our dataset is quite clean, missing only 1 value of 'type'. We'll just replace it with an empty string. Note that on other datasets you might need to do more **profiling** to know what to fix. Check Mull's talk [1]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 881 entries, 0 to 880\n",
      "Data columns (total 4 columns):\n",
      "name    881 non-null object\n",
      "addr    881 non-null object\n",
      "city    881 non-null object\n",
      "type    880 non-null object\n",
      "dtypes: object(4)\n",
      "memory usage: 27.6+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info(verbose=True, null_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df = df.fillna('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we'll clean the column values! Cleaning **name**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton s of chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art s delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art s deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>delis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art s deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>delis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name                         addr         city  \\\n",
       "0  arnie morton s of chicago       435 s. la cienega blv.  los angeles   \n",
       "1  arnie morton s of chicago      435 s. la cienega blvd.  los angeles   \n",
       "2               arnie morton  435 s. la cienega boulevard  los angeles   \n",
       "3         art s delicatessen          12224 ventura blvd.  studio city   \n",
       "4                 art s deli          12224 ventura blvd.  studio city   \n",
       "5                 art s deli          12224 ventura blvd.  los angeles   \n",
       "6              hotel bel air         701 stone canyon rd.      bel air   \n",
       "7              bel air hotel         701 stone canyon rd.      bel air   \n",
       "8                    bel air        701 stone canyon road      bel air   \n",
       "\n",
       "          type  \n",
       "0     american  \n",
       "1  steakhouses  \n",
       "2  steakhouses  \n",
       "3     american  \n",
       "4        delis  \n",
       "5        delis  \n",
       "6  californian  \n",
       "7  californian  \n",
       "8     american  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def assign_no_symbols_name(df):\n",
    "    return df.assign(\n",
    "        name=df['name']\n",
    "             .str.replace(irrelevant_regex, ' ')\n",
    "             .str.replace(multispace_regex, ' '))\n",
    "\n",
    "df = assign_no_symbols_name(df)\n",
    "df.head(9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('s', 184),\n",
      " ('cafe', 81),\n",
      " ('the', 39),\n",
      " ('grill', 32),\n",
      " ('restaurant', 26),\n",
      " ('la', 24),\n",
      " ('le', 20),\n",
      " ('house', 20),\n",
      " ('bar', 19),\n",
      " ('of', 15),\n",
      " ('bistro', 13),\n",
      " ('room', 11),\n",
      " ('kitchen', 10),\n",
      " ('deli', 9),\n",
      " ('club', 9),\n",
      " ('and', 9),\n",
      " ('ritz', 9),\n",
      " ('carlton', 9),\n",
      " ('on', 8),\n",
      " ('buckhead', 8)]\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "possible_stopwords = Counter(\" \".join(df[\"name\"]).split()).most_common(20)\n",
    "pprint.pprint(possible_stopwords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>delis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>delis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "      <td>american</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name                         addr         city         type\n",
       "0  arnie morton chicago       435 s. la cienega blv.  los angeles     american\n",
       "1  arnie morton chicago      435 s. la cienega blvd.  los angeles  steakhouses\n",
       "2          arnie morton  435 s. la cienega boulevard  los angeles  steakhouses\n",
       "3      art delicatessen          12224 ventura blvd.  studio city     american\n",
       "4              art deli          12224 ventura blvd.  studio city        delis\n",
       "5              art deli          12224 ventura blvd.  los angeles        delis\n",
       "6         hotel bel air         701 stone canyon rd.      bel air  californian\n",
       "7         bel air hotel         701 stone canyon rd.      bel air  californian\n",
       "8               bel air        701 stone canyon road      bel air     american"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def assign_cleaned_name(df):\n",
    "    restaurant_stopwords = {\n",
    "        's', 'the', 'la', 'le', 'of', 'and', 'on', 'l'}\n",
    "    restaurant_stopwords_regex = r'\\b(?:{})\\b'.format(\n",
    "        '|'.join(restaurant_stopwords))\n",
    "    return df.assign(\n",
    "        name=df['name']\n",
    "             .str.replace(restaurant_stopwords_regex, '')\n",
    "             .str.replace(multispace_regex, ' ')\n",
    "             .str.strip())\n",
    "\n",
    "df = assign_cleaned_name(df)\n",
    "df.head(9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Geocoding **addr**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "881 819\n"
     ]
    }
   ],
   "source": [
    "all_addresses = df['addr'].str.cat(df['city'], sep=', ').values\n",
    "unique_addresses = np.unique(all_addresses)\n",
    "print(len(all_addresses), len(unique_addresses))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Failed to get postal from 8\n",
      "Failed to get latlng from 0\n"
     ]
    }
   ],
   "source": [
    "import os.path\n",
    "import json\n",
    "\n",
    "geocoding_filename = 'address_to_geocoding.json'\n",
    "\n",
    "def geocode_addresses(address_to_geocoding):\n",
    "    remaining_addresses = (\n",
    "        set(unique_addresses) -\n",
    "        set(k for k, v in address_to_geocoding.items() if v is not None))\n",
    "    \n",
    "    with requests.Session() as session:\n",
    "        for i, address in enumerate(remaining_addresses):\n",
    "            print(f\"Geocoding {i + 1}/{len(remaining_addresses)}\")\n",
    "            geocode_result = geocoder.google(address, session=session)\n",
    "            address_to_geocoding[address] = geocode_result.json\n",
    "\n",
    "        with open(geocoding_filename, 'w') as f:\n",
    "            json.dump(address_to_geocoding, f, indent=4)\n",
    "\n",
    "if not os.path.exists(geocoding_filename):\n",
    "    address_to_geocoding = {}\n",
    "    geocode_addresses(address_to_geocoding)\n",
    "else:\n",
    "    with open(geocoding_filename) as f:\n",
    "        address_to_geocoding = json.load(f)\n",
    "    geocode_addresses(address_to_geocoding)\n",
    " \n",
    "address_to_postal = {\n",
    "    k: v['postal']\n",
    "    for k, v in address_to_geocoding.items()\n",
    "    if v is not None and 'postal' in v\n",
    "}\n",
    "address_to_latlng = {\n",
    "    k: (v['lat'], v['lng'])\n",
    "    for k, v in address_to_geocoding.items()\n",
    "    if v is not None\n",
    "}\n",
    "print(f\"Failed to get postal from {len(address_to_geocoding) - len(address_to_postal)}\")\n",
    "print(f\"Failed to get latlng from {len(address_to_geocoding) - len(address_to_latlng)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>american</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>american</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>delis</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>delis</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name                         addr         city  \\\n",
       "0  arnie morton chicago       435 s. la cienega blv.  los angeles   \n",
       "1  arnie morton chicago      435 s. la cienega blvd.  los angeles   \n",
       "2          arnie morton  435 s. la cienega boulevard  los angeles   \n",
       "3      art delicatessen          12224 ventura blvd.  studio city   \n",
       "4              art deli          12224 ventura blvd.  studio city   \n",
       "5              art deli          12224 ventura blvd.  los angeles   \n",
       "\n",
       "          type postal        lat         lng  \n",
       "0     american  90048  34.070609 -118.376722  \n",
       "1  steakhouses  90048  34.070609 -118.376722  \n",
       "2  steakhouses  90048  34.070609 -118.376722  \n",
       "3     american  91604  34.142966 -118.399469  \n",
       "4        delis  91604  34.142966 -118.399469  \n",
       "5        delis  91604  34.142966 -118.399469  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def assign_postal_lat_lng(df):\n",
    "    addresses = df['addr'].str.cat(df['city'], sep=', ')\n",
    "    addresses_to_postal = [address_to_postal.get(a) for a in addresses]\n",
    "    addresses_to_lat = [address_to_latlng[a][0] if a in address_to_latlng else None for a in addresses]\n",
    "    addresses_to_lng = [address_to_latlng[a][1] if a in address_to_latlng else None for a in addresses]\n",
    "\n",
    "    return df.assign(postal=addresses_to_postal, lat=addresses_to_lat, lng=addresses_to_lng)\n",
    "\n",
    "df = assign_postal_lat_lng(df)\n",
    "df.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Adding **addr_variations**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>(435 san la cienega boulevard, 435 sur la cien...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>(435 san la cienega boulevard, 435 san louisia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>(435 san la cienega boulevard, 435 san louisia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name                         addr  \\\n",
       "0  arnie morton chicago       435 s. la cienega blv.   \n",
       "1  arnie morton chicago      435 s. la cienega blvd.   \n",
       "2          arnie morton  435 s. la cienega boulevard   \n",
       "3      art delicatessen          12224 ventura blvd.   \n",
       "4              art deli          12224 ventura blvd.   \n",
       "5              art deli          12224 ventura blvd.   \n",
       "\n",
       "                                     addr_variations  \n",
       "0  (435 san la cienega boulevard, 435 sur la cien...  \n",
       "1  (435 san la cienega boulevard, 435 san louisia...  \n",
       "2  (435 san la cienega boulevard, 435 san louisia...  \n",
       "3                          (12224 ventura boulevard)  \n",
       "4                          (12224 ventura boulevard)  \n",
       "5                          (12224 ventura boulevard)  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def assign_addr_variations(df):\n",
    "    return df.assign(\n",
    "        addr_variations=df['addr'].apply(\n",
    "            lambda addr: frozenset(expand_address(addr))))\n",
    "\n",
    "df = assign_addr_variations(df)\n",
    "df[['name', 'addr', 'addr_variations']].head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now, with a clean dataset, we can move to the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1/4 - Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To explain the following deduplication steps, we'll use the Python libary **recordlinkage**, aka **[Python Record Linkage Toolkit](http://recordlinkage.readthedocs.io/en/stable/)**. We choose it because it doesn't abstract too much the details of the process, even though it has a simple API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have the cleaned records, we now need the pairs we want to compare to find matches.\n",
    "To **produce the pairs**, we could do a \"full\" index, i.e., all records against all records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:recordlinkage:indexing - performance warning - A full index can result in large number of record pairs.\n",
      "Full index: 881 records, 387640 pairs\n"
     ]
    }
   ],
   "source": [
    "import recordlinkage as rl\n",
    "from recordlinkage.index import Full\n",
    "\n",
    "full_indexer = Full()\n",
    "pairs = full_indexer.index(df)\n",
    "\n",
    "print(f\"Full index: {len(df)} records, {len(pairs)} pairs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The formula for the total number of pairs is:  \n",
    "`len(df) * (len(df) - 1) / 2 == 387640`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div><img style=\"display: block; margin: 0 auto;\" src=\"full-indexing.png?\" width=800></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The number of pairs grows too fast as the number of records grows: it **grows quadratically**.\n",
    "That's why we need **indexing**. We need to produce only pairs that are good candidates of being duplicates to avoid wasting too much time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Indexing is also called **blocking** because the trivial way to index is to produce pairs that have **some column value in common**. By doing this, we produce blocks of record pairs and compare only those in the same block:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div><img style=\"display: block; margin: 0 auto;\" src=\"standard-blocking.png?\" width=800></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Below we produce pairs that have **equal values for postal** code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Postal index: 6536 pairs\n"
     ]
    }
   ],
   "source": [
    "from recordlinkage.index import Block\n",
    "\n",
    "postal_indexer = Block('postal')\n",
    "postal_index_pairs = postal_indexer.index(df)\n",
    "\n",
    "print(f\"Postal index: {len(postal_index_pairs)} pairs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We could also produce pairs by sorting the unique values of some column and, for each value, getting the records with **neighboring values**. The idea is to produce pairs with close values on some column, like `johnny` and `john` or  `2018-01-02` and `2018-01-05`. It looks like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div><img style=\"display: block; margin: 0 auto;\" src=\"sorted-neighbourhood.png?\" width=800></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Below we produce pairs that have **neighboring values for name**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name index: 3132 pairs\n"
     ]
    }
   ],
   "source": [
    "from recordlinkage.index import SortedNeighbourhood\n",
    "\n",
    "name_indexer = SortedNeighbourhood('name', window=7)\n",
    "name_index_pairs = name_indexer.index(df)\n",
    "\n",
    "print(f\"Name index: {len(name_index_pairs)} pairs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that simply sorting values wouldn't be able to get `kamila` and `camila` as neighbors, as sorting is sensitive to the leading characters of strings. There are other ways to index that could handle that, check Christen, chapter 4 [2]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To produce more pairs without introducing redundant pairs, we should **union** different indexes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Postal or name index: 9465 pairs\n"
     ]
    }
   ],
   "source": [
    "pairs = postal_index_pairs.union(name_index_pairs)\n",
    "\n",
    "print(f\"Postal or name index: {len(pairs)} pairs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# We'll swap the levels of pairs MultiIndex returned by recordlinkage\n",
    "# to make it simpler to interpret. Here we get the upper triangle of the pairs matrix.\n",
    "pairs = pairs.swaplevel().sortlevel()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We now have **which pairs** we want to run comparisons on!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2/4 - Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we want to run comparisons on the indexed pairs to produce **a comparison vector for each pair**. A comparison vector represents the similarity between 2 records by holding similarity values between 0 to 1 for each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>type</th>\n",
       "      <th>latlng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <th>200</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name  addr  type  latlng  addr_variations\n",
       "100 200   1.0  0.75   0.0    0.25              0.0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[1.0, 0.75, 0.0, 0.25, 0.0]],\n",
    "             columns=['name', 'addr', 'type', 'latlng', 'addr_variations'],\n",
    "             index=pd.MultiIndex.from_arrays([[100], [200]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For example, the comparison vector above means the pair of records `(100, 200)` has:\n",
    "- Equal `name`s\n",
    "- Similar `addr`s\n",
    "- Completely different `type`s\n",
    "- Some distance on `latlng`s\n",
    "- No agreement on `addr_variations`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To compute the comparison vectors for all indexed pairs, we define a **similarity function** for each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "vectorized_intersection = np.vectorize(\n",
    "    lambda x, y: float(bool(x.intersection(y))))\n",
    "\n",
    "def compare_addr_variations(a1, a2):\n",
    "    return vectorized_intersection(a1, a2)\n",
    "\n",
    "comp = rl.Compare()\n",
    "comp.string('name', 'name', method='jarowinkler', label='name')\n",
    "comp.string('addr', 'addr', method='jarowinkler', label='addr')\n",
    "comp.string('city', 'city', method='jarowinkler', label='city')\n",
    "comp.string('type', 'type', method='jarowinkler', label='type')\n",
    "comp.geo('lat', 'lng', 'lat', 'lng', method='exp', scale=0.5, label='latlng')\n",
    "comp.compare_vectorized(compare_addr_variations,\n",
    "                        'addr_variations', 'addr_variations',\n",
    "                        label='addr_variations');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are many similarity functions we could use for **strings**. For example:\n",
    "- `jarowinkler` gives priority to the begining of the string\n",
    "- `levenshtein` cares more about the order\n",
    "- `lcs` cares less about the order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jarowinkler(\"arnie morton\", \"arnie morton s of chicago\") == 0.896\n",
      "levenshtein(\"arnie morton\", \"arnie morton s of chicago\") == 0.48\n",
      "lcs(\"arnie morton\", \"arnie morton s of chicago\") == 0.6486486486486487\n",
      "\n",
      "jarowinkler(\"the palm\", \"palm the\") == 0.4166666666666667\n",
      "levenshtein(\"the palm\", \"palm the\") == 0.0\n",
      "lcs(\"the palm\", \"palm the\") == 0.875\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from recordlinkage.algorithms.string import (\n",
    "    jarowinkler_similarity as jarowinkler,\n",
    "    levenshtein_similarity as levenshtein,\n",
    "    longest_common_substring_similarity as lcs)\n",
    "\n",
    "for s1, s2 in [[\"arnie morton\", \"arnie morton s of chicago\"],\n",
    "               [\"the palm\", \"palm the\"]]:\n",
    "    print(f'jarowinkler(\"{s1}\", \"{s2}\") ==',\n",
    "          jarowinkler([s1], [s2])[0])\n",
    "    print(f'levenshtein(\"{s1}\", \"{s2}\") ==',\n",
    "          levenshtein([s1], [s2])[0])\n",
    "    print(f'lcs(\"{s1}\", \"{s2}\") ==',\n",
    "          lcs([s1], [s2])[0])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For more details on the different similarity functions you can use, check Christen, chapter 5 [2]. Also, check the similarity functions [implemented on recordlikage library](http://recordlinkage.readthedocs.io/en/stable/ref-compare.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we'll compute the **comparison vectors**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>latlng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.985507</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.310606</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.910774</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.310606</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.559722</td>\n",
       "      <td>0.593620</td>\n",
       "      <td>0.40404</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.572222</td>\n",
       "      <td>0.593620</td>\n",
       "      <td>0.40404</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.572222</td>\n",
       "      <td>0.593620</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name      addr     city      type   latlng  addr_variations\n",
       "0 1  1.000000  0.985507  1.00000  0.310606  1.00000              1.0\n",
       "  2  0.920000  0.910774  1.00000  0.310606  1.00000              1.0\n",
       "  3  0.559722  0.593620  0.40404  1.000000  0.00001              0.0\n",
       "  4  0.572222  0.593620  0.40404  0.550000  0.00001              0.0\n",
       "  5  0.572222  0.593620  1.00000  0.550000  0.00001              0.0"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comparison_vectors = comp.compute(pairs, df)\n",
    "comparison_vectors.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "It's good to check the **statistics** of the vectors to see if they look right:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>latlng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>9465.000000</td>\n",
       "      <td>9465.000000</td>\n",
       "      <td>9465.000000</td>\n",
       "      <td>9465.000000</td>\n",
       "      <td>9465.000000</td>\n",
       "      <td>9465.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.515619</td>\n",
       "      <td>0.630213</td>\n",
       "      <td>0.827979</td>\n",
       "      <td>0.558777</td>\n",
       "      <td>0.326727</td>\n",
       "      <td>0.015320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.167495</td>\n",
       "      <td>0.142923</td>\n",
       "      <td>0.263938</td>\n",
       "      <td>0.223105</td>\n",
       "      <td>0.298554</td>\n",
       "      <td>0.122827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.447712</td>\n",
       "      <td>0.526462</td>\n",
       "      <td>0.589744</td>\n",
       "      <td>0.436508</td>\n",
       "      <td>0.000093</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.517949</td>\n",
       "      <td>0.593957</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.527381</td>\n",
       "      <td>0.295513</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.588462</td>\n",
       "      <td>0.706536</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.690476</td>\n",
       "      <td>0.561194</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name         addr         city         type       latlng  \\\n",
       "count  9465.000000  9465.000000  9465.000000  9465.000000  9465.000000   \n",
       "mean      0.515619     0.630213     0.827979     0.558777     0.326727   \n",
       "std       0.167495     0.142923     0.263938     0.223105     0.298554   \n",
       "min       0.000000     0.000000     0.000000     0.000000     0.000000   \n",
       "25%       0.447712     0.526462     0.589744     0.436508     0.000093   \n",
       "50%       0.517949     0.593957     1.000000     0.527381     0.295513   \n",
       "75%       0.588462     0.706536     1.000000     0.690476     0.561194   \n",
       "max       1.000000     1.000000     1.000000     1.000000     1.000000   \n",
       "\n",
       "       addr_variations  \n",
       "count      9465.000000  \n",
       "mean          0.015320  \n",
       "std           0.122827  \n",
       "min           0.000000  \n",
       "25%           0.000000  \n",
       "50%           0.000000  \n",
       "75%           0.000000  \n",
       "max           1.000000  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comparison_vectors.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "By looking at the summary we can see it's easier to get higher similarities on 'city', 'addr' and 'type', not so easy on 'name' and 'latlng', and very difficult on 'addr_variations'. Therefore, it seems that:\n",
    "- we have few unique values of 'city' in our dataset and that also causes...\n",
    "- ...lots of 'addr' to be similar\n",
    "- there only a limited number of 'type's, so it repeats a lot\n",
    "- and getting a 1 on 'addr_variations' is rare since it only considers an exact agreement on address variation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now, with our comparison vectors, we'll explore **different ways** to classify them as matches and nonmatches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3/4 - Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Before, let's remember what's on records `0` to `5`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blv.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>american</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "      <td>(435 san la cienega boulevard, 435 sur la cien...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arnie morton chicago</td>\n",
       "      <td>435 s. la cienega blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "      <td>(435 san la cienega boulevard, 435 san louisia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arnie morton</td>\n",
       "      <td>435 s. la cienega boulevard</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.070609</td>\n",
       "      <td>-118.376722</td>\n",
       "      <td>(435 san la cienega boulevard, 435 san louisia...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>american</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>delis</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>delis</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name                         addr         city  \\\n",
       "0  arnie morton chicago       435 s. la cienega blv.  los angeles   \n",
       "1  arnie morton chicago      435 s. la cienega blvd.  los angeles   \n",
       "2          arnie morton  435 s. la cienega boulevard  los angeles   \n",
       "3      art delicatessen          12224 ventura blvd.  studio city   \n",
       "4              art deli          12224 ventura blvd.  studio city   \n",
       "5              art deli          12224 ventura blvd.  los angeles   \n",
       "\n",
       "          type postal        lat         lng  \\\n",
       "0     american  90048  34.070609 -118.376722   \n",
       "1  steakhouses  90048  34.070609 -118.376722   \n",
       "2  steakhouses  90048  34.070609 -118.376722   \n",
       "3     american  91604  34.142966 -118.399469   \n",
       "4        delis  91604  34.142966 -118.399469   \n",
       "5        delis  91604  34.142966 -118.399469   \n",
       "\n",
       "                                     addr_variations  \n",
       "0  (435 san la cienega boulevard, 435 sur la cien...  \n",
       "1  (435 san la cienega boulevard, 435 san louisia...  \n",
       "2  (435 san la cienega boulevard, 435 san louisia...  \n",
       "3                          (12224 ventura boulevard)  \n",
       "4                          (12224 ventura boulevard)  \n",
       "5                          (12224 ventura boulevard)  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Threshold-Based Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A simple way to classify comparison vectors as matches or nonmatches is to compute a **weighted average** over the vectors to get a **score**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>latlng</th>\n",
       "      <th>addr_variations</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>1</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.985507</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.310606</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.929611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.910774</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.310606</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.898138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.559722</td>\n",
       "      <td>0.593620</td>\n",
       "      <td>0.40404</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.347484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.572222</td>\n",
       "      <td>0.593620</td>\n",
       "      <td>0.40404</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.306234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.572222</td>\n",
       "      <td>0.593620</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.336032</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name      addr     city      type   latlng  addr_variations     score\n",
       "0 1  1.000000  0.985507  1.00000  0.310606  1.00000              1.0  0.929611\n",
       "  2  0.920000  0.910774  1.00000  0.310606  1.00000              1.0  0.898138\n",
       "  3  0.559722  0.593620  0.40404  1.000000  0.00001              0.0  0.347484\n",
       "  4  0.572222  0.593620  0.40404  0.550000  0.00001              0.0  0.306234\n",
       "  5  0.572222  0.593620  1.00000  0.550000  0.00001              0.0  0.336032"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = np.average(\n",
    "    comparison_vectors.values,\n",
    "    axis=1,\n",
    "    weights=[30, 10, 5, 10, 30, 15])\n",
    "scored_comparison_vectors = comparison_vectors.assign(score=scores)\n",
    "scored_comparison_vectors.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "By looking at the data, we know record `0` truly matches `1` and `2`, so we'll **classify as a match** any pair with score `>= 0.85`. That's our **threshold**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>latlng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
       "      <th>1</th>\n",
       "      <td>1.00</td>\n",
       "      <td>0.985507</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.310606</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.92</td>\n",
       "      <td>0.910774</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.310606</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <td>0.92</td>\n",
       "      <td>0.923779</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>4</th>\n",
       "      <td>0.90</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.90</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.40404</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     name      addr     city      type  latlng  addr_variations\n",
       "0 1  1.00  0.985507  1.00000  0.310606     1.0              1.0\n",
       "  2  0.92  0.910774  1.00000  0.310606     1.0              1.0\n",
       "1 2  0.92  0.923779  1.00000  1.000000     1.0              1.0\n",
       "3 4  0.90  1.000000  1.00000  0.550000     1.0              1.0\n",
       "  5  0.90  1.000000  0.40404  0.550000     1.0              1.0"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matches = comparison_vectors[\n",
    "    scored_comparison_vectors['score'] >= 0.85]\n",
    "matches.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's check another match we've found with this threshold: what are the matches of record `3`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>art delicatessen</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>american</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               name                 addr         city      type postal  \\\n",
       "3  art delicatessen  12224 ventura blvd.  studio city  american  91604   \n",
       "\n",
       "         lat         lng            addr_variations  \n",
       "3  34.142966 -118.399469  (12224 ventura boulevard)  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matched:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>studio city</td>\n",
       "      <td>delis</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>art deli</td>\n",
       "      <td>12224 ventura blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>delis</td>\n",
       "      <td>91604</td>\n",
       "      <td>34.142966</td>\n",
       "      <td>-118.399469</td>\n",
       "      <td>(12224 ventura boulevard)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name                 addr         city   type postal        lat  \\\n",
       "4  art deli  12224 ventura blvd.  studio city  delis  91604  34.142966   \n",
       "5  art deli  12224 ventura blvd.  los angeles  delis  91604  34.142966   \n",
       "\n",
       "          lng            addr_variations  \n",
       "4 -118.399469  (12224 ventura boulevard)  \n",
       "5 -118.399469  (12224 ventura boulevard)  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(df.loc[[3]])\n",
    "print(\"matched:\")\n",
    "display(df.loc[matches.loc[(3,)].index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Seems OK! Since we have the true match status on `cluster` column, we can **evaluate** how well our threshold classification did:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Golden pairs: 150\n"
     ]
    }
   ],
   "source": [
    "golden_pairs = Block('cluster').index(df_with_truth)\n",
    "golden_pairs = golden_pairs.swaplevel().sortlevel()[0]\n",
    "print(\"Golden pairs:\", len(golden_pairs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true_positives total: 99\n",
      "false_positives total: 7\n",
      "false_negatives total: 51\n"
     ]
    }
   ],
   "source": [
    "found_pairs_set = set(matches.index)\n",
    "\n",
    "golden_pairs_set = set(golden_pairs)\n",
    "\n",
    "true_positives = golden_pairs_set & found_pairs_set\n",
    "false_positives = found_pairs_set - golden_pairs_set\n",
    "false_negatives = golden_pairs_set - found_pairs_set\n",
    "\n",
    "print('true_positives total:', len(true_positives))\n",
    "print('false_positives total:', len(false_positives))\n",
    "print('false_negatives total:', len(false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We've got a small number of **false positives**. Some are really tricky cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False positives:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>palace court</td>\n",
       "      <td>3570 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>continental</td>\n",
       "      <td>36.116156</td>\n",
       "      <td>-115.175058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>562</th>\n",
       "      <td>cafe roma</td>\n",
       "      <td>3570 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>coffee shops/diners</td>\n",
       "      <td>36.116156</td>\n",
       "      <td>-115.175058</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name                    addr       city                 type  \\\n",
       "156  palace court  3570 las vegas blvd. s  las vegas          continental   \n",
       "562     cafe roma  3570 las vegas blvd. s  las vegas  coffee shops/diners   \n",
       "\n",
       "           lat         lng  \n",
       "156  36.116156 -115.175058  \n",
       "562  36.116156 -115.175058  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>palace court</td>\n",
       "      <td>3570 las vegas blvd. s.</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>french (new)</td>\n",
       "      <td>36.116156</td>\n",
       "      <td>-115.175058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>562</th>\n",
       "      <td>cafe roma</td>\n",
       "      <td>3570 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>coffee shops/diners</td>\n",
       "      <td>36.116156</td>\n",
       "      <td>-115.175058</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name                     addr       city                 type  \\\n",
       "157  palace court  3570 las vegas blvd. s.  las vegas         french (new)   \n",
       "562     cafe roma   3570 las vegas blvd. s  las vegas  coffee shops/diners   \n",
       "\n",
       "           lat         lng  \n",
       "157  36.116156 -115.175058  \n",
       "562  36.116156 -115.175058  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>ritz carlton cafe buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>ritz carlton dining room buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                   addr     city  \\\n",
       "196         ritz carlton cafe buckhead  3434 peachtree rd. ne  atlanta   \n",
       "198  ritz carlton dining room buckhead  3434 peachtree rd. ne  atlanta   \n",
       "\n",
       "               type        lat        lng  \n",
       "196  american (new)  33.850807 -84.364227  \n",
       "198  american (new)  33.850807 -84.364227  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>restaurant ritz carlton atlanta</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>continental</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>839</th>\n",
       "      <td>ritz carlton cafe atlanta</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                name               addr     city  \\\n",
       "199  restaurant ritz carlton atlanta  181 peachtree st.  atlanta   \n",
       "839        ritz carlton cafe atlanta  181 peachtree st.  atlanta   \n",
       "\n",
       "               type        lat        lng  \n",
       "199     continental  33.758579 -84.387066  \n",
       "839  american (new)  33.758579 -84.387066  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>les celebrites</td>\n",
       "      <td>160 central park s</td>\n",
       "      <td>new york</td>\n",
       "      <td>french</td>\n",
       "      <td>40.766016</td>\n",
       "      <td>-73.978641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329</th>\n",
       "      <td>cafe botanica</td>\n",
       "      <td>160 central park s</td>\n",
       "      <td>new york</td>\n",
       "      <td>french</td>\n",
       "      <td>40.766016</td>\n",
       "      <td>-73.978641</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               name                addr      city    type        lat  \\\n",
       "99   les celebrites  160 central park s  new york  french  40.766016   \n",
       "329   cafe botanica  160 central park s  new york  french  40.766016   \n",
       "\n",
       "           lng  \n",
       "99  -73.978641  \n",
       "329 -73.978641  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>ritz carlton restaurant</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>french (classic)</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>839</th>\n",
       "      <td>ritz carlton cafe atlanta</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          name               addr     city              type  \\\n",
       "200    ritz carlton restaurant  181 peachtree st.  atlanta  french (classic)   \n",
       "839  ritz carlton cafe atlanta  181 peachtree st.  atlanta    american (new)   \n",
       "\n",
       "           lat        lng  \n",
       "200  33.758579 -84.387066  \n",
       "839  33.758579 -84.387066  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>cafe ritz carlton buckhead</td>\n",
       "      <td>3434 peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>ext 6108 international</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>dining room ritz carlton buckhead</td>\n",
       "      <td>3434 peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>international</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                addr     city  \\\n",
       "195         cafe ritz carlton buckhead  3434 peachtree rd.  atlanta   \n",
       "197  dining room ritz carlton buckhead  3434 peachtree rd.  atlanta   \n",
       "\n",
       "                       type        lat        lng  \n",
       "195  ext 6108 international  33.850807 -84.364227  \n",
       "197           international  33.850807 -84.364227  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f\"False positives:\")\n",
    "for false_positive_pair in false_positives:\n",
    "    display(df.loc[list(false_positive_pair)][['name', 'addr', 'city', 'type', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand, we got a lot of **false negatives**. We've **missed** a lot of matches!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False negatives (sample 10 of 51):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>brasserie coze</td>\n",
       "      <td>3393 peachtree rd.  lenox square mall  near ne...</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>french</td>\n",
       "      <td>33.846181</td>\n",
       "      <td>-84.364109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>171</th>\n",
       "      <td>brasserie coze</td>\n",
       "      <td>3393 peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>french bistro</td>\n",
       "      <td>33.846869</td>\n",
       "      <td>-84.362512</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               name                                               addr  \\\n",
       "170  brasserie coze  3393 peachtree rd.  lenox square mall  near ne...   \n",
       "171  brasserie coze                                 3393 peachtree rd.   \n",
       "\n",
       "        city           type        lat        lng  \n",
       "170  atlanta         french  33.846181 -84.364109  \n",
       "171  atlanta  french bistro  33.846869 -84.362512  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>fenix at argyle</td>\n",
       "      <td>8358 sunset blvd.</td>\n",
       "      <td>w. hollywood</td>\n",
       "      <td>french (new)</td>\n",
       "      <td>34.095097</td>\n",
       "      <td>-118.371967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>fenix at argyle</td>\n",
       "      <td>8358 sunset blvd. west</td>\n",
       "      <td>hollywood</td>\n",
       "      <td>american</td>\n",
       "      <td>34.095097</td>\n",
       "      <td>-118.371967</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               name                    addr          city          type  \\\n",
       "22  fenix at argyle       8358 sunset blvd.  w. hollywood  french (new)   \n",
       "23  fenix at argyle  8358 sunset blvd. west     hollywood      american   \n",
       "\n",
       "          lat         lng  \n",
       "22  34.095097 -118.371967  \n",
       "23  34.095097 -118.371967  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>heera india</td>\n",
       "      <td>595 piedmont ave.  rio shopping mall</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>asian</td>\n",
       "      <td>33.795817</td>\n",
       "      <td>-84.370602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>heera india</td>\n",
       "      <td>595 piedmont ave.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>indian</td>\n",
       "      <td>33.770495</td>\n",
       "      <td>-84.381425</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name                                  addr     city    type  \\\n",
       "182  heera india  595 piedmont ave.  rio shopping mall  atlanta   asian   \n",
       "183  heera india                     595 piedmont ave.  atlanta  indian   \n",
       "\n",
       "           lat        lng  \n",
       "182  33.795817 -84.370602  \n",
       "183  33.770495 -84.381425  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>sign dove</td>\n",
       "      <td>1110 3rd ave. at 65th st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>american</td>\n",
       "      <td>40.76564</td>\n",
       "      <td>-73.963968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>sign dove</td>\n",
       "      <td>1110 third ave.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>40.76564</td>\n",
       "      <td>-73.963968</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          name                       addr           city            type  \\\n",
       "138  sign dove  1110 3rd ave. at 65th st.       new york        american   \n",
       "139  sign dove            1110 third ave.  new york city  american (new)   \n",
       "\n",
       "          lat        lng  \n",
       "138  40.76564 -73.963968  \n",
       "139  40.76564 -73.963968  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>abruzzi</td>\n",
       "      <td>2355 peachtree rd.  peachtree battle shopping ...</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>italian</td>\n",
       "      <td>33.820137</td>\n",
       "      <td>-84.387280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>abruzzi</td>\n",
       "      <td>2355 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>italian</td>\n",
       "      <td>33.824647</td>\n",
       "      <td>-84.387453</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        name                                               addr     city  \\\n",
       "164  abruzzi  2355 peachtree rd.  peachtree battle shopping ...  atlanta   \n",
       "165  abruzzi                              2355 peachtree rd. ne  atlanta   \n",
       "\n",
       "        type        lat        lng  \n",
       "164  italian  33.820137 -84.387280  \n",
       "165  italian  33.824647 -84.387453  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>carmine</td>\n",
       "      <td>2450 broadway  between 90th and 91st sts.</td>\n",
       "      <td>new york</td>\n",
       "      <td>italian</td>\n",
       "      <td>40.791096</td>\n",
       "      <td>-73.973991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>carmine</td>\n",
       "      <td>2450 broadway</td>\n",
       "      <td>new york city</td>\n",
       "      <td>italian</td>\n",
       "      <td>40.791096</td>\n",
       "      <td>-73.973991</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name                                       addr           city  \\\n",
       "69  carmine  2450 broadway  between 90th and 91st sts.       new york   \n",
       "70  carmine                              2450 broadway  new york city   \n",
       "\n",
       "       type        lat        lng  \n",
       "69  italian  40.791096 -73.973991  \n",
       "70  italian  40.791096 -73.973991  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>dining room ritz carlton buckhead</td>\n",
       "      <td>3434 peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>international</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>ritz carlton dining room buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                   addr     city  \\\n",
       "197  dining room ritz carlton buckhead     3434 peachtree rd.  atlanta   \n",
       "198  ritz carlton dining room buckhead  3434 peachtree rd. ne  atlanta   \n",
       "\n",
       "               type        lat        lng  \n",
       "197   international  33.850807 -84.364227  \n",
       "198  american (new)  33.850807 -84.364227  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>restaurant katsu</td>\n",
       "      <td>1972 n. hillhurst ave.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>asian</td>\n",
       "      <td>34.107405</td>\n",
       "      <td>-118.28719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>katsu</td>\n",
       "      <td>1972 hillhurst ave.</td>\n",
       "      <td>los feliz</td>\n",
       "      <td>japanese</td>\n",
       "      <td>34.107405</td>\n",
       "      <td>-118.28719</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name                    addr         city      type  \\\n",
       "28  restaurant katsu  1972 n. hillhurst ave.  los angeles     asian   \n",
       "29             katsu     1972 hillhurst ave.    los feliz  japanese   \n",
       "\n",
       "          lat        lng  \n",
       "28  34.107405 -118.28719  \n",
       "29  34.107405 -118.28719  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w 3rd</td>\n",
       "      <td>st los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>locanda</td>\n",
       "      <td>w. third st.</td>\n",
       "      <td>st los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>34.068958</td>\n",
       "      <td>-118.320928</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name          addr            city     type        lat  \\\n",
       "36  locanda veneta    8638 w 3rd  st los angeles  italian  34.073417   \n",
       "37         locanda  w. third st.  st los angeles  italian  34.068958   \n",
       "\n",
       "           lng  \n",
       "36 -118.381096  \n",
       "37 -118.320928  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>bone</td>\n",
       "      <td>3130 piedmont road</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american</td>\n",
       "      <td>33.842103</td>\n",
       "      <td>-84.371103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>bone restaurant</td>\n",
       "      <td>3130 piedmont rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>33.842103</td>\n",
       "      <td>-84.371103</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name                  addr     city         type        lat  \\\n",
       "168             bone    3130 piedmont road  atlanta     american  33.842103   \n",
       "169  bone restaurant  3130 piedmont rd. ne  atlanta  steakhouses  33.842103   \n",
       "\n",
       "           lng  \n",
       "168 -84.371103  \n",
       "169 -84.371103  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f\"False negatives (sample 10 of {len(false_negatives)}):\")\n",
    "for false_negative_pair in list(false_negatives)[:10]:\n",
    "    display(df.loc[list(false_negative_pair)][['name', 'addr', 'city', 'type', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We've set the weights and the threshold **by guessing**, could we do any better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Supervised Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of trying to guess weights and thresholds, we can train a **classifier** to learn how to classify matches and nonmatches based on some training data we provide:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>cluster</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>3rd st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w. third st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w 3rd st</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>coffee bar</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>coffeehouses</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>les celebrites</td>\n",
       "      <td>160 central park s</td>\n",
       "      <td>new york</td>\n",
       "      <td>french</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>les celebrites</td>\n",
       "      <td>155 w. 58th st.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>french (classic)</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>second avenue deli</td>\n",
       "      <td>156 2nd ave. at 10th st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>delicatessen</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>second avenue deli</td>\n",
       "      <td>156 second ave.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>delis</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>smith &amp; wollensky</td>\n",
       "      <td>201 e. 49th st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>american</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>smith &amp; wollensky</td>\n",
       "      <td>797 third ave.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>chin's</td>\n",
       "      <td>3200 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>asian</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>chin's</td>\n",
       "      <td>3200 las vegas blvd. s.</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>chinese</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>toulouse</td>\n",
       "      <td>b peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>french</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>toulouse</td>\n",
       "      <td>293-b peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>french (new)</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>rose pistola</td>\n",
       "      <td>532 columbus ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>italian</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>rose pistola</td>\n",
       "      <td>532 columbus ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>italian</td>\n",
       "      <td>111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>bistro garden</td>\n",
       "      <td>176 n. canon dr.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>californian</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>remi</td>\n",
       "      <td>3rd st. promenade</td>\n",
       "      <td>santa monica</td>\n",
       "      <td>italian</td>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>remi</td>\n",
       "      <td>145 w. 53rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>italian</td>\n",
       "      <td>334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>west</td>\n",
       "      <td>63rd street steakhouse 44 w. 63rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>american</td>\n",
       "      <td>375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>bistro</td>\n",
       "      <td>3400 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>continental</td>\n",
       "      <td>429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>mikado</td>\n",
       "      <td>3400 las vegas blvd. s</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>asian</td>\n",
       "      <td>446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>l'osteria del forno</td>\n",
       "      <td>519 columbus ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>italian</td>\n",
       "      <td>490</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>stars</td>\n",
       "      <td>150 redwood alley</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>american</td>\n",
       "      <td>514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>stars cafe</td>\n",
       "      <td>500 van ness ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>american</td>\n",
       "      <td>515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>belvedere  the</td>\n",
       "      <td>9882 little santa monica blvd.</td>\n",
       "      <td>beverly hills</td>\n",
       "      <td>pacific new wave</td>\n",
       "      <td>537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>bernard's</td>\n",
       "      <td>515 s. olive st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>continental</td>\n",
       "      <td>539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>bistro 45</td>\n",
       "      <td>45 s. mentor ave.</td>\n",
       "      <td>pasadena</td>\n",
       "      <td>californian</td>\n",
       "      <td>540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>cafe '50s</td>\n",
       "      <td>838 lincoln blvd.</td>\n",
       "      <td>venice</td>\n",
       "      <td>american</td>\n",
       "      <td>545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>cafe blanc</td>\n",
       "      <td>9777 little santa monica blvd.</td>\n",
       "      <td>beverly hills</td>\n",
       "      <td>pacific new wave</td>\n",
       "      <td>546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>la cachette</td>\n",
       "      <td>10506 little santa monica blvd.</td>\n",
       "      <td>century city</td>\n",
       "      <td>french (new)</td>\n",
       "      <td>568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>moongate</td>\n",
       "      <td>3400 las vegas blvd. s.</td>\n",
       "      <td>las vegas</td>\n",
       "      <td>chinese</td>\n",
       "      <td>666</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name                                   addr           city  \\\n",
       "0        locanda veneta                                3rd st.    los angeles   \n",
       "1        locanda veneta                      8638 w. third st.    los angeles   \n",
       "2        locanda veneta                          8638 w 3rd st    los angeles   \n",
       "3             cafe lalo                        201 w. 83rd st.       new york   \n",
       "4             cafe lalo                        201 w. 83rd st.  new york city   \n",
       "5        les celebrites                     160 central park s       new york   \n",
       "6        les celebrites                        155 w. 58th st.  new york city   \n",
       "7    second avenue deli               156 2nd ave. at 10th st.       new york   \n",
       "8    second avenue deli                        156 second ave.  new york city   \n",
       "9     smith & wollensky                        201 e. 49th st.       new york   \n",
       "10    smith & wollensky                         797 third ave.  new york city   \n",
       "11               chin's                 3200 las vegas blvd. s      las vegas   \n",
       "12               chin's                3200 las vegas blvd. s.      las vegas   \n",
       "13             toulouse                        b peachtree rd.        atlanta   \n",
       "14             toulouse                    293-b peachtree rd.        atlanta   \n",
       "15         rose pistola                      532 columbus ave.  san francisco   \n",
       "16         rose pistola                      532 columbus ave.  san francisco   \n",
       "17        bistro garden                       176 n. canon dr.    los angeles   \n",
       "18                 remi                      3rd st. promenade   santa monica   \n",
       "19                 remi                        145 w. 53rd st.       new york   \n",
       "20                 west  63rd street steakhouse 44 w. 63rd st.       new york   \n",
       "21               bistro                 3400 las vegas blvd. s      las vegas   \n",
       "22               mikado                 3400 las vegas blvd. s      las vegas   \n",
       "23  l'osteria del forno                      519 columbus ave.  san francisco   \n",
       "24                stars                      150 redwood alley  san francisco   \n",
       "25           stars cafe                      500 van ness ave.  san francisco   \n",
       "26       belvedere  the         9882 little santa monica blvd.  beverly hills   \n",
       "27            bernard's                       515 s. olive st.    los angeles   \n",
       "28            bistro 45                      45 s. mentor ave.       pasadena   \n",
       "29            cafe '50s                      838 lincoln blvd.         venice   \n",
       "30           cafe blanc         9777 little santa monica blvd.  beverly hills   \n",
       "31          la cachette        10506 little santa monica blvd.   century city   \n",
       "32             moongate                3400 las vegas blvd. s.      las vegas   \n",
       "\n",
       "                type  cluster  \n",
       "0            italian       13  \n",
       "1            italian       13  \n",
       "2            italian       13  \n",
       "3         coffee bar       26  \n",
       "4       coffeehouses       26  \n",
       "5             french       42  \n",
       "6   french (classic)       42  \n",
       "7       delicatessen       58  \n",
       "8              delis       58  \n",
       "9           american       62  \n",
       "10       steakhouses       62  \n",
       "11             asian       67  \n",
       "12           chinese       67  \n",
       "13            french       92  \n",
       "14      french (new)       92  \n",
       "15           italian      111  \n",
       "16           italian      111  \n",
       "17       californian      115  \n",
       "18           italian      159  \n",
       "19           italian      334  \n",
       "20          american      375  \n",
       "21       continental      429  \n",
       "22             asian      446  \n",
       "23           italian      490  \n",
       "24          american      514  \n",
       "25          american      515  \n",
       "26  pacific new wave      537  \n",
       "27       continental      539  \n",
       "28       californian      540  \n",
       "29          american      545  \n",
       "30  pacific new wave      546  \n",
       "31      french (new)      568  \n",
       "32           chinese      666  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_training = pd.read_csv('restaurant-training.csv', skip_blank_lines=True)\n",
    "df_training = df_training.drop(columns=['phone'])\n",
    "df_training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We need to **preprocess** our training data too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>cluster</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>3rd st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>13</td>\n",
       "      <td>None</td>\n",
       "      <td>34.068958</td>\n",
       "      <td>-118.320928</td>\n",
       "      <td>(3 street, 3rd saint, 3rd street, 3 saint)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w. third st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>13</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "      <td>(8638 west 3rd street, 8638 west 3 saint, 8638...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w 3rd st</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>13</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(8638 west 3rd street, 8638 west 3 saint, 8638...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>coffee bar</td>\n",
       "      <td>26</td>\n",
       "      <td>10024</td>\n",
       "      <td>40.785981</td>\n",
       "      <td>-73.976727</td>\n",
       "      <td>(201 w 83 street, 201 west 83rd street, 201 w ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cafe lalo</td>\n",
       "      <td>201 w. 83rd st.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>coffeehouses</td>\n",
       "      <td>26</td>\n",
       "      <td>10024</td>\n",
       "      <td>40.785981</td>\n",
       "      <td>-73.976727</td>\n",
       "      <td>(201 w 83 street, 201 west 83rd street, 201 w ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name               addr           city          type  cluster  \\\n",
       "0  locanda veneta            3rd st.    los angeles       italian       13   \n",
       "1  locanda veneta  8638 w. third st.    los angeles       italian       13   \n",
       "2  locanda veneta      8638 w 3rd st    los angeles       italian       13   \n",
       "3       cafe lalo    201 w. 83rd st.       new york    coffee bar       26   \n",
       "4       cafe lalo    201 w. 83rd st.  new york city  coffeehouses       26   \n",
       "\n",
       "  postal        lat         lng  \\\n",
       "0   None  34.068958 -118.320928   \n",
       "1  90048  34.073417 -118.381096   \n",
       "2   None        NaN         NaN   \n",
       "3  10024  40.785981  -73.976727   \n",
       "4  10024  40.785981  -73.976727   \n",
       "\n",
       "                                     addr_variations  \n",
       "0         (3 street, 3rd saint, 3rd street, 3 saint)  \n",
       "1  (8638 west 3rd street, 8638 west 3 saint, 8638...  \n",
       "2  (8638 west 3rd street, 8638 west 3 saint, 8638...  \n",
       "3  (201 w 83 street, 201 west 83rd street, 201 w ...  \n",
       "4  (201 w 83 street, 201 west 83rd street, 201 w ...  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_training = assign_no_symbols_name(df_training)\n",
    "df_training = assign_cleaned_name(df_training)\n",
    "df_training = assign_postal_lat_lng(df_training)\n",
    "df_training = assign_addr_variations(df_training)\n",
    "df_training.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We'll feed a **Support Vector Machine** classifier with our training data. SVMs are resilient to noise, can handle correlated features (like 'addr' and 'latlng') and are robust to imbalanced training sets. That last attribute is relevant for deduplication, because we'll usually find more negative pairs than positive pairs to add to a training set. (Bilenko [3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:recordlinkage:indexing - performance warning - A full index can result in large number of record pairs.\n"
     ]
    }
   ],
   "source": [
    "all_training_pairs = Full().index(df_training)\n",
    "matches_training_pairs = Block('cluster').index(df_training)\n",
    "\n",
    "training_vectors = comp.compute(all_training_pairs, df_training)\n",
    "\n",
    "svm = rl.SVMClassifier()\n",
    "svm.fit(training_vectors, matches_training_pairs);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true_positives total: 99\n",
      "false_positives total: 7\n",
      "false_negatives total: 51\n",
      "\n",
      "svm_true_positives total: 124\n",
      "svm_false_positives total: 5\n",
      "svm_false_negatives total: 26\n"
     ]
    }
   ],
   "source": [
    "svm_pairs = svm.predict(comparison_vectors)\n",
    "svm_found_pairs_set = set(svm_pairs)\n",
    "\n",
    "svm_true_positives = golden_pairs_set & svm_found_pairs_set\n",
    "svm_false_positives = svm_found_pairs_set - golden_pairs_set\n",
    "svm_false_negatives = golden_pairs_set - svm_found_pairs_set\n",
    "\n",
    "print('true_positives total:', len(true_positives))\n",
    "print('false_positives total:', len(false_positives))\n",
    "print('false_negatives total:', len(false_negatives))\n",
    "print()\n",
    "print('svm_true_positives total:', len(svm_true_positives))\n",
    "print('svm_false_positives total:', len(svm_false_positives))\n",
    "print('svm_false_negatives total:', len(svm_false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Much better results**! The only false positive we got on the SVM classifier and not on the threshold method is a really difficult case where most columns are very similar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(SVM false positives) - (Threshold false positives):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>643</th>\n",
       "      <td>stars</td>\n",
       "      <td>150 redwood alley</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>american</td>\n",
       "      <td>37.780894</td>\n",
       "      <td>-122.419481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>644</th>\n",
       "      <td>stars cafe</td>\n",
       "      <td>500 van ness ave.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>american</td>\n",
       "      <td>37.780298</td>\n",
       "      <td>-122.420002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name               addr           city      type        lat  \\\n",
       "643       stars  150 redwood alley  san francisco  american  37.780894   \n",
       "644  stars cafe  500 van ness ave.  san francisco  american  37.780298   \n",
       "\n",
       "            lng  \n",
       "643 -122.419481  \n",
       "644 -122.420002  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"(SVM false positives) - (Threshold false positives):\")\n",
    "for svm_false_positive in (svm_false_positives - false_positives):\n",
    "    display(df.loc[list(svm_false_positive)][['name', 'addr', 'city', 'type', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But the SVM was tricked by some apparently simple cases, so **we can't be very confident** it really learned well to classify matches:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(SVM false negatives) - (Threshold false negatives):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>palm</td>\n",
       "      <td>9001 santa monica blvd.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>american</td>\n",
       "      <td>34.083064</td>\n",
       "      <td>-118.387282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>palm los angeles</td>\n",
       "      <td>9001 santa monica blvd.</td>\n",
       "      <td>w. hollywood</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>34.083064</td>\n",
       "      <td>-118.387282</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name                     addr          city         type  \\\n",
       "40              palm  9001 santa monica blvd.   los angeles     american   \n",
       "41  palm los angeles  9001 santa monica blvd.  w. hollywood  steakhouses   \n",
       "\n",
       "          lat         lng  \n",
       "40  34.083064 -118.387282  \n",
       "41  34.083064 -118.387282  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name                  addr     city         type        lat  \\\n",
       "6  hotel bel air  701 stone canyon rd.  bel air  californian  34.086594   \n",
       "7  bel air hotel  701 stone canyon rd.  bel air  californian  34.086594   \n",
       "\n",
       "          lng  \n",
       "6 -118.446351  \n",
       "7 -118.446351  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"(SVM false negatives) - (Threshold false negatives):\")\n",
    "for svm_false_negative in (svm_false_negatives - false_negatives):\n",
    "    display(df.loc[list(svm_false_negative)][['name', 'addr', 'city', 'type', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are [other classifiers](https://recordlinkage.readthedocs.io/en/stable/notebooks/classifiers.html) from recordlinkage library we could try, but the truth is:\n",
    "- It's **very difficult to build a good training set** that takes in account all important cases of matches/nonmatches\n",
    "- It's possible to tune classifier parameters to get better results, but it's very difficult to **decide the right parameters that will generalize well** for future predictions\n",
    "- And we're not even sure **if the indexing rules we used are really sane**: we can be dropping true positives that are not being blocked together, or even introducing false negatives that are being blocked together but our classifier isn't being able classify them as nonmatching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The **alternative** to all that uncertainty is...\n",
    "\n",
    "### Active Learning Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Active Learning methods identify **training examples** that \"lead to maximal accuracy improvements\" (Bilenko [3]), both to train **optimal classifier weights**, as well as to find **optimal indexing/blocking rules**! A Python library called [Dedupe](https://docs.dedupe.io/en/latest/index.html) implements that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We'll see it in practice. First we define the fields/columns of our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import logging; logging.disable(level=logging.NOTSET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import dedupe\n",
    "\n",
    "fields = [\n",
    "    {\n",
    "        'field': 'name',\n",
    "        'variable name': 'name',\n",
    "        'type': 'ShortString',\n",
    "        'has missing': True\n",
    "    },\n",
    "    {\n",
    "        'field': 'addr',\n",
    "        'variable name': 'addr',\n",
    "        'type': 'ShortString',\n",
    "    },\n",
    "    {\n",
    "        'field': 'city',\n",
    "        'variable name': 'city',\n",
    "        'type': 'ShortString',\n",
    "        'has missing': True\n",
    "    },\n",
    "] # ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dedupe comes with [**built-in variables**](https://docs.dedupe.io/en/latest/Variable-definition.html) for common data types like names, addresses, latlong, datetimes, thereby reducing the preprocessing work. It also supports custom variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def addr_variations_comparator(x, y):\n",
    "    return 1.0 - float(bool(x.intersection(y)))\n",
    "\n",
    "fields.extend([\n",
    "    {\n",
    "        'field': 'postal',\n",
    "        'variable name': 'postal',\n",
    "        'type': 'ShortString',\n",
    "        'has missing': True\n",
    "    },\n",
    "    {\n",
    "        'field': 'latlng',\n",
    "        'variable name': 'latlng',\n",
    "        'type': 'LatLong',\n",
    "        'has missing': True\n",
    "    },\n",
    "    {\n",
    "        'field': 'addr_variations',\n",
    "        'variable name': 'addr_variations',\n",
    "        'type': 'Custom',\n",
    "        'comparator': addr_variations_comparator\n",
    "    },\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Other nice feature of Dedupe is the built-in ability to model [**interactions**](https://en.wikipedia.org/wiki/Interaction_(statistics)) between fields. What's that for? Imagine we have an exact address match: it makes no sense to have all 'addr', 'city', 'postal', 'latlng', 'addr_variations' contributing additively to the matching odds. It's expected all of them to match if 'addr' and 'city' match exactly. So to prevent a redundant increase of odds, we should add a interaction on those fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fields.extend([\n",
    "    {\n",
    "        'type': 'Interaction',\n",
    "        'interaction variables': [\n",
    "            'addr',\n",
    "            'city',\n",
    "            'postal',\n",
    "            'latlng',\n",
    "            'addr_variations'\n",
    "        ]\n",
    "    }\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we **initialize** the Dedupe instance with the fields. We can also use a pre-saved pickled Dedupe instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:dedupe.api:((SimplePredicate: (firstTokenPredicate, name), TfidfNGramCanopyPredicate: (0.6, addr)), (SimplePredicate: (commonTwoTokens, name), SimplePredicate: (fingerprint, name)))\n"
     ]
    }
   ],
   "source": [
    "settings_filename = 'dedupe-settings.pickle'\n",
    "if os.path.exists(settings_filename):\n",
    "    with open(settings_filename, 'rb') as sf:\n",
    "        deduper = dedupe.StaticDedupe(sf, num_cores=4)\n",
    "else:\n",
    "    deduper = dedupe.Dedupe(fields, num_cores=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "We need to **adapt** the data a bit to the format Dedupe wants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "data_for_dedupe = df.to_dict('index')\n",
    "for record in data_for_dedupe.values():\n",
    "    # Change nans to None\n",
    "    for k, v in record.items():\n",
    "        if isinstance(v, float) and np.isnan(v):\n",
    "            record[k] = None\n",
    "    \n",
    "    # Move lat and lng to a single field latlng\n",
    "    lat = record.pop('lat')\n",
    "    lng = record.pop('lng')\n",
    "    if lat is not None and lng is not None:\n",
    "        record['latlng'] = (lat, lng)\n",
    "    else:\n",
    "        record['latlng'] = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we're using a Dedupe instance that we trained before. Let's check how was the **training input/output**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name : cite\n",
      "addr : 120 w. 51st st.\n",
      "city : new york\n",
      "postal : 10019\n",
      "latlng : (40.7607952, -73.9812268)\n",
      "addr_variations : frozenset({'120 west 51st saint', '120 west 51 saint', '120 w 51st street', '120 w 51st saint', '120 west 51st street', '120 w 51 saint', '120 w 51 street', '120 west 51 street'})\n",
      "\n",
      "name : new york noodletown\n",
      "addr : 28 1/2 bowery at bayard st.\n",
      "city : new york\n",
      "postal : 10013\n",
      "latlng : (40.7150317, -73.9970383)\n",
      "addr_variations : frozenset({'28 1 2 bowery at bayard saint', '28 1 2 bowery at bayard street'})\n",
      "\n",
      "0/10 positive, 0/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished\n",
      "n\n",
      "name : bernardin\n",
      "addr : 155 w. 51st st.\n",
      "city : new york city\n",
      "postal : 10019\n",
      "latlng : (40.7615691, -73.98180479999999)\n",
      "addr_variations : frozenset({'155 w 51st saint', '155 west 51st saint', '155 w 51st street', '155 west 51 saint', '155 w 51 street', '155 w 51 saint', '155 west 51 street', '155 west 51st street'})\n",
      "\n",
      "name : republic\n",
      "addr : 37a union sq. w  between 16th and 17th sts.\n",
      "city : new york\n",
      "postal : 10003\n",
      "latlng : (40.7369985, -73.9907851)\n",
      "addr_variations : frozenset({'37a union square w between 16th and 17 streets', '37 a union square west between 16th and 17 streets', '37a union square w between 16th and 17th streets', '37 a union square west between 16th and 17th streets', '37 a union square west between 16 and 17th streets', '37 a union square w between 16th and 17th streets', '37 a union square w between 16 and 17 streets', '37 a union square w between 16th and 17 streets', '37 a union square west between 16 and 17 streets', '37 a union square w between 16 and 17th streets', '37a union square w between 16 and 17 streets', '37a union square w between 16 and 17th streets', '37a union square west between 16 and 17th streets', '37a union square west between 16 and 17 streets', '37a union square west between 16th and 17 streets', '37a union square west between 16th and 17th streets'})\n",
      "\n",
      "0/10 positive, 1/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n",
      "n\n",
      "name : dawat\n",
      "addr : 210 e. 58th st.\n",
      "city : new york\n",
      "postal : None\n",
      "latlng : None\n",
      "addr_variations : frozenset({'210 e 58 street', '210 e 58th saint', '210 east 58 saint', '210 east 58th street', '210 east 58 street', '210 e 58 saint', '210 e 58th street', '210 east 58th saint'})\n",
      "\n",
      "name : dawat\n",
      "addr : 210 e. 58th st.\n",
      "city : new york city\n",
      "postal : 10022\n",
      "latlng : (40.7604227, -73.9664276)\n",
      "addr_variations : frozenset({'210 e 58 street', '210 e 58th saint', '210 east 58 saint', '210 east 58th street', '210 east 58 street', '210 e 58 saint', '210 e 58th street', '210 east 58th saint'})\n",
      "\n",
      "0/10 positive, 2/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n",
      "y\n",
      "name : art delicatessen\n",
      "addr : 12224 ventura blvd.\n",
      "city : studio city\n",
      "postal : 91604\n",
      "latlng : (34.1429661, -118.3994688)\n",
      "addr_variations : frozenset({'12224 ventura boulevard'})\n",
      "\n",
      "name : art deli\n",
      "addr : 12224 ventura blvd.\n",
      "city : los angeles\n",
      "postal : 91604\n",
      "latlng : (34.1429661, -118.3994688)\n",
      "addr_variations : frozenset({'12224 ventura boulevard'})\n",
      "\n",
      "1/10 positive, 2/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n",
      "y\n",
      "INFO:dedupe.training:Final predicate set:\n",
      "INFO:dedupe.training:(SimplePredicate: (fingerprint, addr), SimplePredicate: (wholeFieldPredicate, name))\n",
      "name : gotham bar grill\n",
      "addr : 12 e 12th st\n",
      "city : new york city\n",
      "postal : 10003\n",
      "latlng : (40.734207, -73.99369899999999)\n",
      "addr_variations : frozenset({'12 e 12th street', '12 east 12 saint', '12 e 12 street', '12 e 12 saint', '12 east 12th saint', '12 east 12th street', '12 east 12 street', '12 e 12th saint'})\n",
      "\n",
      "name : gotham\n",
      "addr : 12 e 12th st\n",
      "city : new york\n",
      "postal : 10003\n",
      "latlng : (40.734207, -73.99369899999999)\n",
      "addr_variations : frozenset({'12 e 12th street', '12 east 12 saint', '12 e 12 street', '12 e 12 saint', '12 east 12th saint', '12 east 12th street', '12 east 12 street', '12 e 12th saint'})\n",
      "\n",
      "2/10 positive, 2/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n",
      "y\n",
      "INFO:dedupe.training:Final predicate set:\n",
      "INFO:dedupe.training:(SimplePredicate: (fingerprint, addr), SimplePredicate: (sortedAcronym, name))\n",
      "name : dawat\n",
      "addr : 210 e. 58th st.\n",
      "city : new york\n",
      "postal : None\n",
      "latlng : None\n",
      "addr_variations : frozenset({'210 e 58 street', '210 e 58th saint', '210 east 58 saint', '210 east 58th street', '210 east 58 street', '210 e 58 saint', '210 e 58th street', '210 east 58th saint'})\n",
      "\n",
      "name : rainbow restaurant\n",
      "addr : 2118 n. decatur rd.\n",
      "city : decatur\n",
      "postal : 30033\n",
      "latlng : (33.7908588, -84.3052307)\n",
      "addr_variations : frozenset({'2118 north decatur road', '2118 n decatur road'})\n",
      "\n",
      "3/10 positive, 2/10 negative\n",
      "Do these records refer to the same thing?\n",
      "(y)es / (n)o / (u)nsure / (f)inished / (p)revious\n",
      "n\n",
      "INFO:dedupe.training:Final predicate set:\n",
      "INFO:dedupe.training:(SimplePredicate: (fingerprint, addr), TfidfNGramCanopyPredicate: (0.6, name))\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "training_input_output = 'training-input-output.txt'\n",
    "if os.path.exists(training_input_output):\n",
    "    with open(training_input_output) as t:\n",
    "        txt = t.read()\n",
    "        print('\\n'.join(txt.split('\\n')[:114]))\n",
    "print('...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "You can check the full training at `training-input-output.txt`.\n",
    "\n",
    "If you want to train it yourself, do a `rm dedupe-settings.pickle dedupe-slides-training.json` and run this whole Active Learning session again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "if not isinstance(deduper, dedupe.StaticDedupe):\n",
    "    deduper.sample(data_for_dedupe)\n",
    "    \n",
    "    training_filename = 'dedupe-slides-training.json'\n",
    "    if os.path.exists(training_filename):\n",
    "        with open(training_filename) as tf:\n",
    "            deduper.readTraining(tf)\n",
    "\n",
    "    dedupe.consoleLabel(deduper)\n",
    "    \n",
    "    with open(training_filename, 'w') as tf:\n",
    "        deduper.writeTraining(tf)\n",
    "    \n",
    "    deduper.train()\n",
    "    \n",
    "    with open(settings_filename, 'wb') as sf:\n",
    "        deduper.writeSettings(sf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After training, we can see which **blocking predicates** (indexing rules) the deduper learned from our training input. It's good to do that to check if we trained enough:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((SimplePredicate: (firstTokenPredicate, name),\n",
       "  TfidfNGramCanopyPredicate: (0.6, addr)),\n",
       " (SimplePredicate: (commonTwoTokens, name),\n",
       "  SimplePredicate: (fingerprint, name)))"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deduper.predicates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Those predicates **make sense**:\n",
    "- Pairs where the first token of the name is the same AND the addr is similar\n",
    "- ... union with ...\n",
    "- Pairs where the name shares two tokens AND the fingerprint of the name is the same (fingerprint means the sorted tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The deduper selected those predicates from this extense list of **possible predicates**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{ExistsPredicate: (Exists, city),\n",
       " ExistsPredicate: (Exists, latlng),\n",
       " ExistsPredicate: (Exists, name),\n",
       " ExistsPredicate: (Exists, postal),\n",
       " LevenshteinCanopyPredicate: (1, addr),\n",
       " LevenshteinCanopyPredicate: (1, city),\n",
       " LevenshteinCanopyPredicate: (1, name),\n",
       " LevenshteinCanopyPredicate: (1, postal),\n",
       " LevenshteinCanopyPredicate: (2, addr),\n",
       " LevenshteinCanopyPredicate: (2, city),\n",
       " LevenshteinCanopyPredicate: (2, name),\n",
       " LevenshteinCanopyPredicate: (2, postal),\n",
       " LevenshteinCanopyPredicate: (3, addr),\n",
       " LevenshteinCanopyPredicate: (3, city),\n",
       " LevenshteinCanopyPredicate: (3, name),\n",
       " LevenshteinCanopyPredicate: (3, postal),\n",
       " LevenshteinCanopyPredicate: (4, addr),\n",
       " LevenshteinCanopyPredicate: (4, city),\n",
       " LevenshteinCanopyPredicate: (4, name),\n",
       " LevenshteinCanopyPredicate: (4, postal),\n",
       " SimplePredicate: (alphaNumericPredicate, addr),\n",
       " SimplePredicate: (alphaNumericPredicate, city),\n",
       " SimplePredicate: (alphaNumericPredicate, name),\n",
       " SimplePredicate: (alphaNumericPredicate, postal),\n",
       " SimplePredicate: (commonFourGram, addr),\n",
       " SimplePredicate: (commonFourGram, city),\n",
       " SimplePredicate: (commonFourGram, name),\n",
       " SimplePredicate: (commonFourGram, postal),\n",
       " SimplePredicate: (commonIntegerPredicate, addr),\n",
       " SimplePredicate: (commonIntegerPredicate, city),\n",
       " SimplePredicate: (commonIntegerPredicate, name),\n",
       " SimplePredicate: (commonIntegerPredicate, postal),\n",
       " SimplePredicate: (commonSixGram, addr),\n",
       " SimplePredicate: (commonSixGram, city),\n",
       " SimplePredicate: (commonSixGram, name),\n",
       " SimplePredicate: (commonSixGram, postal),\n",
       " SimplePredicate: (commonThreeTokens, addr),\n",
       " SimplePredicate: (commonThreeTokens, city),\n",
       " SimplePredicate: (commonThreeTokens, name),\n",
       " SimplePredicate: (commonThreeTokens, postal),\n",
       " SimplePredicate: (commonTwoTokens, addr),\n",
       " SimplePredicate: (commonTwoTokens, city),\n",
       " SimplePredicate: (commonTwoTokens, name),\n",
       " SimplePredicate: (commonTwoTokens, postal),\n",
       " SimplePredicate: (doubleMetaphone, addr),\n",
       " SimplePredicate: (doubleMetaphone, city),\n",
       " SimplePredicate: (doubleMetaphone, name),\n",
       " SimplePredicate: (doubleMetaphone, postal),\n",
       " SimplePredicate: (fingerprint, addr),\n",
       " SimplePredicate: (fingerprint, city),\n",
       " SimplePredicate: (fingerprint, name),\n",
       " SimplePredicate: (fingerprint, postal),\n",
       " SimplePredicate: (firstIntegerPredicate, addr),\n",
       " SimplePredicate: (firstIntegerPredicate, city),\n",
       " SimplePredicate: (firstIntegerPredicate, name),\n",
       " SimplePredicate: (firstIntegerPredicate, postal),\n",
       " SimplePredicate: (firstTokenPredicate, addr),\n",
       " SimplePredicate: (firstTokenPredicate, city),\n",
       " SimplePredicate: (firstTokenPredicate, name),\n",
       " SimplePredicate: (firstTokenPredicate, postal),\n",
       " SimplePredicate: (hundredIntegerPredicate, addr),\n",
       " SimplePredicate: (hundredIntegerPredicate, city),\n",
       " SimplePredicate: (hundredIntegerPredicate, name),\n",
       " SimplePredicate: (hundredIntegerPredicate, postal),\n",
       " SimplePredicate: (hundredIntegersOddPredicate, addr),\n",
       " SimplePredicate: (hundredIntegersOddPredicate, city),\n",
       " SimplePredicate: (hundredIntegersOddPredicate, name),\n",
       " SimplePredicate: (hundredIntegersOddPredicate, postal),\n",
       " SimplePredicate: (latLongGridPredicate, latlng),\n",
       " SimplePredicate: (metaphoneToken, addr),\n",
       " SimplePredicate: (metaphoneToken, city),\n",
       " SimplePredicate: (metaphoneToken, name),\n",
       " SimplePredicate: (metaphoneToken, postal),\n",
       " SimplePredicate: (nearIntegersPredicate, addr),\n",
       " SimplePredicate: (nearIntegersPredicate, city),\n",
       " SimplePredicate: (nearIntegersPredicate, name),\n",
       " SimplePredicate: (nearIntegersPredicate, postal),\n",
       " SimplePredicate: (oneGramFingerprint, addr),\n",
       " SimplePredicate: (oneGramFingerprint, city),\n",
       " SimplePredicate: (oneGramFingerprint, name),\n",
       " SimplePredicate: (oneGramFingerprint, postal),\n",
       " SimplePredicate: (sameFiveCharStartPredicate, addr),\n",
       " SimplePredicate: (sameFiveCharStartPredicate, city),\n",
       " SimplePredicate: (sameFiveCharStartPredicate, name),\n",
       " SimplePredicate: (sameFiveCharStartPredicate, postal),\n",
       " SimplePredicate: (sameSevenCharStartPredicate, addr),\n",
       " SimplePredicate: (sameSevenCharStartPredicate, city),\n",
       " SimplePredicate: (sameSevenCharStartPredicate, name),\n",
       " SimplePredicate: (sameSevenCharStartPredicate, postal),\n",
       " SimplePredicate: (sameThreeCharStartPredicate, addr),\n",
       " SimplePredicate: (sameThreeCharStartPredicate, city),\n",
       " SimplePredicate: (sameThreeCharStartPredicate, name),\n",
       " SimplePredicate: (sameThreeCharStartPredicate, postal),\n",
       " SimplePredicate: (sortedAcronym, addr),\n",
       " SimplePredicate: (sortedAcronym, city),\n",
       " SimplePredicate: (sortedAcronym, name),\n",
       " SimplePredicate: (sortedAcronym, postal),\n",
       " SimplePredicate: (suffixArray, addr),\n",
       " SimplePredicate: (suffixArray, city),\n",
       " SimplePredicate: (suffixArray, name),\n",
       " SimplePredicate: (suffixArray, postal),\n",
       " SimplePredicate: (tokenFieldPredicate, addr),\n",
       " SimplePredicate: (tokenFieldPredicate, city),\n",
       " SimplePredicate: (tokenFieldPredicate, name),\n",
       " SimplePredicate: (tokenFieldPredicate, postal),\n",
       " SimplePredicate: (twoGramFingerprint, addr),\n",
       " SimplePredicate: (twoGramFingerprint, city),\n",
       " SimplePredicate: (twoGramFingerprint, name),\n",
       " SimplePredicate: (twoGramFingerprint, postal),\n",
       " SimplePredicate: (wholeFieldPredicate, addr),\n",
       " SimplePredicate: (wholeFieldPredicate, city),\n",
       " SimplePredicate: (wholeFieldPredicate, name),\n",
       " SimplePredicate: (wholeFieldPredicate, postal),\n",
       " TfidfNGramCanopyPredicate: (0.2, addr),\n",
       " TfidfNGramCanopyPredicate: (0.2, city),\n",
       " TfidfNGramCanopyPredicate: (0.2, name),\n",
       " TfidfNGramCanopyPredicate: (0.2, postal),\n",
       " TfidfNGramCanopyPredicate: (0.4, addr),\n",
       " TfidfNGramCanopyPredicate: (0.4, city),\n",
       " TfidfNGramCanopyPredicate: (0.4, name),\n",
       " TfidfNGramCanopyPredicate: (0.4, postal),\n",
       " TfidfNGramCanopyPredicate: (0.6, addr),\n",
       " TfidfNGramCanopyPredicate: (0.6, city),\n",
       " TfidfNGramCanopyPredicate: (0.6, name),\n",
       " TfidfNGramCanopyPredicate: (0.6, postal),\n",
       " TfidfNGramCanopyPredicate: (0.8, addr),\n",
       " TfidfNGramCanopyPredicate: (0.8, city),\n",
       " TfidfNGramCanopyPredicate: (0.8, name),\n",
       " TfidfNGramCanopyPredicate: (0.8, postal)}"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deduper.data_model.predicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To proceed with the deduplication, we compute the **clustering threshold** and call the actual **match**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:dedupe.api:Maximum expected recall and precision\n",
      "INFO:dedupe.api:recall: 0.998\n",
      "INFO:dedupe.api:precision: 0.859\n",
      "INFO:dedupe.api:With threshold: 0.195\n"
     ]
    }
   ],
   "source": [
    "import itertools\n",
    "\n",
    "threshold = deduper.threshold(data_for_dedupe, recall_weight=2)\n",
    "clustered_dupes = deduper.match(data_for_dedupe, threshold)\n",
    "\n",
    "dedupe_found_pairs_set = set()\n",
    "for cluster, __ in clustered_dupes:  # we'll explain that later\n",
    "    for pair in itertools.combinations(cluster, 2):\n",
    "        dedupe_found_pairs_set.add(tuple(pair))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we'll **evaluate** how it performed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true_positives total: 99\n",
      "false_positives total: 7\n",
      "false_negatives total: 51\n",
      "\n",
      "svm_true_positives total: 124\n",
      "svm_false_positives total: 5\n",
      "svm_false_negatives total: 26\n",
      "\n",
      "dedupe_true_positives total: 141\n",
      "dedupe_false_positives total: 7\n",
      "dedupe_false_negatives total: 9\n"
     ]
    }
   ],
   "source": [
    "dedupe_true_positives = golden_pairs_set & dedupe_found_pairs_set\n",
    "dedupe_false_positives = dedupe_found_pairs_set - golden_pairs_set\n",
    "dedupe_false_negatives = golden_pairs_set - dedupe_found_pairs_set\n",
    "\n",
    "print('true_positives total:', len(true_positives))\n",
    "print('false_positives total:', len(false_positives))\n",
    "print('false_negatives total:', len(false_negatives))\n",
    "print()\n",
    "print('svm_true_positives total:', len(svm_true_positives))\n",
    "print('svm_false_positives total:', len(svm_false_positives))\n",
    "print('svm_false_negatives total:', len(svm_false_negatives))\n",
    "print()\n",
    "print('dedupe_true_positives total:', len(dedupe_true_positives))\n",
    "print('dedupe_false_positives total:', len(dedupe_false_positives))\n",
    "print('dedupe_false_negatives total:', len(dedupe_false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Great! It found **more true positives** than the other methods. But it also found more false positives than SVM. It's possible to control that by lowering `recall_weight`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:dedupe.api:Maximum expected recall and precision\n",
      "INFO:dedupe.api:recall: 0.972\n",
      "INFO:dedupe.api:precision: 0.885\n",
      "INFO:dedupe.api:With threshold: 0.727\n"
     ]
    }
   ],
   "source": [
    "threshold_low_recall = deduper.threshold(data_for_dedupe, recall_weight=0.5)\n",
    "clustered_dupes_low_recall = deduper.match(data_for_dedupe, threshold_low_recall)\n",
    "\n",
    "dedupe_low_recall_found_pairs_set = set()\n",
    "for cluster, __ in clustered_dupes_low_recall:\n",
    "    for pair in itertools.combinations(cluster, 2):\n",
    "        dedupe_low_recall_found_pairs_set.add(tuple(pair))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dedupe_true_positives total: 141\n",
      "dedupe_false_positives total: 7\n",
      "dedupe_false_negatives total: 9\n",
      "\n",
      "dedupe_low_recall_true_positives total: 137\n",
      "dedupe_low_recall_false_positives total: 1\n",
      "dedupe_low_recall_false_negatives total: 13\n"
     ]
    }
   ],
   "source": [
    "dedupe_low_recall_true_positives = golden_pairs_set & dedupe_low_recall_found_pairs_set\n",
    "dedupe_low_recall_false_positives = dedupe_low_recall_found_pairs_set - golden_pairs_set\n",
    "dedupe_low_recall_false_negatives = golden_pairs_set - dedupe_low_recall_found_pairs_set\n",
    "\n",
    "print('dedupe_true_positives total:', len(dedupe_true_positives))\n",
    "print('dedupe_false_positives total:', len(dedupe_false_positives))\n",
    "print('dedupe_false_negatives total:', len(dedupe_false_negatives))\n",
    "print()\n",
    "print('dedupe_low_recall_true_positives total:', len(dedupe_low_recall_true_positives))\n",
    "print('dedupe_low_recall_false_positives total:', len(dedupe_low_recall_false_positives))\n",
    "print('dedupe_low_recall_false_negatives total:', len(dedupe_low_recall_false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "As we can see, there's always a **tradeoff** between true positives and false positives, or, more precisely, between [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall). You have to decide what's more important for **your business**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But let's suppose we want to find more true positives and use the previous `dedupe_found_pairs_set`. What **false positives** it found?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dedupe false positives\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>337</th>\n",
       "      <td>caffe lure</td>\n",
       "      <td>169 sullivan st.  between houston and bleecker...</td>\n",
       "      <td>new york</td>\n",
       "      <td>french</td>\n",
       "      <td>40.727928</td>\n",
       "      <td>-74.000985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>338</th>\n",
       "      <td>caffe reggio</td>\n",
       "      <td>119 macdougal st.  between 3rd and bleecker sts.</td>\n",
       "      <td>new york</td>\n",
       "      <td>coffee bar</td>\n",
       "      <td>40.730308</td>\n",
       "      <td>-74.000371</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name                                               addr  \\\n",
       "337    caffe lure  169 sullivan st.  between houston and bleecker...   \n",
       "338  caffe reggio   119 macdougal st.  between 3rd and bleecker sts.   \n",
       "\n",
       "         city        type        lat        lng  \n",
       "337  new york      french  40.727928 -74.000985  \n",
       "338  new york  coffee bar  40.730308 -74.000371  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>762</th>\n",
       "      <td>palm</td>\n",
       "      <td>837 second ave.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>40.751701</td>\n",
       "      <td>-73.971180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>763</th>\n",
       "      <td>palm too</td>\n",
       "      <td>840 second ave.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>40.751467</td>\n",
       "      <td>-73.970686</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name             addr           city         type        lat  \\\n",
       "762      palm  837 second ave.  new york city  steakhouses  40.751701   \n",
       "763  palm too  840 second ave.  new york city  steakhouses  40.751467   \n",
       "\n",
       "           lng  \n",
       "762 -73.971180  \n",
       "763 -73.970686  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>ritz carlton cafe buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>ritz carlton dining room buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                   addr     city  \\\n",
       "196         ritz carlton cafe buckhead  3434 peachtree rd. ne  atlanta   \n",
       "198  ritz carlton dining room buckhead  3434 peachtree rd. ne  atlanta   \n",
       "\n",
       "               type        lat        lng  \n",
       "196  american (new)  33.850807 -84.364227  \n",
       "198  american (new)  33.850807 -84.364227  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>335</th>\n",
       "      <td>caffe dante</td>\n",
       "      <td>81 macdougal st.  between houston and bleeker ...</td>\n",
       "      <td>new york</td>\n",
       "      <td>coffee bar</td>\n",
       "      <td>40.728878</td>\n",
       "      <td>-74.001662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>338</th>\n",
       "      <td>caffe reggio</td>\n",
       "      <td>119 macdougal st.  between 3rd and bleecker sts.</td>\n",
       "      <td>new york</td>\n",
       "      <td>coffee bar</td>\n",
       "      <td>40.730308</td>\n",
       "      <td>-74.000371</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             name                                               addr  \\\n",
       "335   caffe dante  81 macdougal st.  between houston and bleeker ...   \n",
       "338  caffe reggio   119 macdougal st.  between 3rd and bleecker sts.   \n",
       "\n",
       "         city        type        lat        lng  \n",
       "335  new york  coffee bar  40.728878 -74.001662  \n",
       "338  new york  coffee bar  40.730308 -74.000371  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>cafe ritz carlton buckhead</td>\n",
       "      <td>3434 peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>ext 6108 international</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>ritz carlton dining room buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                   addr     city  \\\n",
       "195         cafe ritz carlton buckhead     3434 peachtree rd.  atlanta   \n",
       "198  ritz carlton dining room buckhead  3434 peachtree rd. ne  atlanta   \n",
       "\n",
       "                       type        lat        lng  \n",
       "195  ext 6108 international  33.850807 -84.364227  \n",
       "198          american (new)  33.850807 -84.364227  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>ritz carlton restaurant</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>french (classic)</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>839</th>\n",
       "      <td>ritz carlton cafe atlanta</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          name               addr     city              type  \\\n",
       "200    ritz carlton restaurant  181 peachtree st.  atlanta  french (classic)   \n",
       "839  ritz carlton cafe atlanta  181 peachtree st.  atlanta    american (new)   \n",
       "\n",
       "           lat        lng  \n",
       "200  33.758579 -84.387066  \n",
       "839  33.758579 -84.387066  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>335</th>\n",
       "      <td>caffe dante</td>\n",
       "      <td>81 macdougal st.  between houston and bleeker ...</td>\n",
       "      <td>new york</td>\n",
       "      <td>coffee bar</td>\n",
       "      <td>40.728878</td>\n",
       "      <td>-74.001662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>337</th>\n",
       "      <td>caffe lure</td>\n",
       "      <td>169 sullivan st.  between houston and bleecker...</td>\n",
       "      <td>new york</td>\n",
       "      <td>french</td>\n",
       "      <td>40.727928</td>\n",
       "      <td>-74.000985</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name                                               addr      city  \\\n",
       "335  caffe dante  81 macdougal st.  between houston and bleeker ...  new york   \n",
       "337   caffe lure  169 sullivan st.  between houston and bleecker...  new york   \n",
       "\n",
       "           type        lat        lng  \n",
       "335  coffee bar  40.728878 -74.001662  \n",
       "337      french  40.727928 -74.000985  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Dedupe false positives\")\n",
    "for false_positive_pair in list(dedupe_false_positives):\n",
    "    display(df.loc[list(false_positive_pair)][['name', 'addr', 'city', 'type', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The deduper is confused by \"caffe\"s that are located near each other. \"caffe\" could be a stopword? Or even better, we could incorporate some logic for **name rarity**: if the token isn't rare, like \"caffe\", it should contribute less with the name similarity than a rarer token like \"reggio\". It's possible to do that with [an interaction with name frequency](https://github.com/dedupeio/dedupe/issues/392).\n",
    "\n",
    "Or maybe the deduper could have learned a more restrictive name blocking? Or the classifier should have put less weight on name? It's difficult to guess because those things could have introduced more **false negatives** too. What about them?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dedupe false negatives\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>219</th>\n",
       "      <td>fringale</td>\n",
       "      <td>570 4th st.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>french</td>\n",
       "      <td>37.778542</td>\n",
       "      <td>-122.397193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>fringale</td>\n",
       "      <td>570 fourth st.</td>\n",
       "      <td>san francisco</td>\n",
       "      <td>french bistro</td>\n",
       "      <td>37.778542</td>\n",
       "      <td>-122.397193</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name            addr           city           type        lat  \\\n",
       "219  fringale     570 4th st.  san francisco         french  37.778542   \n",
       "220  fringale  570 fourth st.  san francisco  french bistro  37.778542   \n",
       "\n",
       "            lng  \n",
       "219 -122.397193  \n",
       "220 -122.397193  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name                  addr     city         type        lat  \\\n",
       "6  hotel bel air  701 stone canyon rd.  bel air  californian  34.086594   \n",
       "7  bel air hotel  701 stone canyon rd.  bel air  californian  34.086594   \n",
       "\n",
       "          lng  \n",
       "6 -118.446351  \n",
       "7 -118.446351  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "      <td>american</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name                   addr     city         type        lat  \\\n",
       "6  hotel bel air   701 stone canyon rd.  bel air  californian  34.086594   \n",
       "8        bel air  701 stone canyon road  bel air     american  34.086594   \n",
       "\n",
       "          lng  \n",
       "6 -118.446351  \n",
       "8 -118.446351  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>dining room ritz carlton buckhead</td>\n",
       "      <td>3434 peachtree rd.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>international</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>ritz carlton dining room buckhead</td>\n",
       "      <td>3434 peachtree rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american (new)</td>\n",
       "      <td>33.850807</td>\n",
       "      <td>-84.364227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name                   addr     city  \\\n",
       "197  dining room ritz carlton buckhead     3434 peachtree rd.  atlanta   \n",
       "198  ritz carlton dining room buckhead  3434 peachtree rd. ne  atlanta   \n",
       "\n",
       "               type        lat        lng  \n",
       "197   international  33.850807 -84.364227  \n",
       "198  american (new)  33.850807 -84.364227  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>restaurant katsu</td>\n",
       "      <td>1972 n. hillhurst ave.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>asian</td>\n",
       "      <td>34.107405</td>\n",
       "      <td>-118.28719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>katsu</td>\n",
       "      <td>1972 hillhurst ave.</td>\n",
       "      <td>los feliz</td>\n",
       "      <td>japanese</td>\n",
       "      <td>34.107405</td>\n",
       "      <td>-118.28719</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name                    addr         city      type  \\\n",
       "28  restaurant katsu  1972 n. hillhurst ave.  los angeles     asian   \n",
       "29             katsu     1972 hillhurst ave.    los feliz  japanese   \n",
       "\n",
       "          lat        lng  \n",
       "28  34.107405 -118.28719  \n",
       "29  34.107405 -118.28719  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>spago</td>\n",
       "      <td>1114 horn ave.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>californian</td>\n",
       "      <td>34.091172</td>\n",
       "      <td>-118.383161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>spago los angeles</td>\n",
       "      <td>8795 sunset blvd.</td>\n",
       "      <td>w. hollywood</td>\n",
       "      <td>californian</td>\n",
       "      <td>34.091132</td>\n",
       "      <td>-118.383290</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 name               addr          city         type  \\\n",
       "53              spago     1114 horn ave.   los angeles  californian   \n",
       "54  spago los angeles  8795 sunset blvd.  w. hollywood  californian   \n",
       "\n",
       "          lat         lng  \n",
       "53  34.091172 -118.383161  \n",
       "54  34.091132 -118.383290  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>bone</td>\n",
       "      <td>3130 piedmont road</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>american</td>\n",
       "      <td>33.842103</td>\n",
       "      <td>-84.371103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>bone restaurant</td>\n",
       "      <td>3130 piedmont rd. ne</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>steakhouses</td>\n",
       "      <td>33.842103</td>\n",
       "      <td>-84.371103</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name                  addr     city         type        lat  \\\n",
       "168             bone    3130 piedmont road  atlanta     american  33.842103   \n",
       "169  bone restaurant  3130 piedmont rd. ne  atlanta  steakhouses  33.842103   \n",
       "\n",
       "           lng  \n",
       "168 -84.371103  \n",
       "169 -84.371103  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>shun lee west</td>\n",
       "      <td>43 w. 65th st.</td>\n",
       "      <td>new york</td>\n",
       "      <td>asian</td>\n",
       "      <td>40.772900</td>\n",
       "      <td>-73.981348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>shun lee palace</td>\n",
       "      <td>155 e. 55th st.</td>\n",
       "      <td>new york city</td>\n",
       "      <td>chinese</td>\n",
       "      <td>40.759428</td>\n",
       "      <td>-73.969068</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name             addr           city     type        lat  \\\n",
       "136    shun lee west   43 w. 65th st.       new york    asian  40.772900   \n",
       "137  shun lee palace  155 e. 55th st.  new york city  chinese  40.759428   \n",
       "\n",
       "           lng  \n",
       "136 -73.981348  \n",
       "137 -73.969068  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>restaurant ritz carlton atlanta</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>continental</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>ritz carlton restaurant</td>\n",
       "      <td>181 peachtree st.</td>\n",
       "      <td>atlanta</td>\n",
       "      <td>french (classic)</td>\n",
       "      <td>33.758579</td>\n",
       "      <td>-84.387066</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                name               addr     city  \\\n",
       "199  restaurant ritz carlton atlanta  181 peachtree st.  atlanta   \n",
       "200          ritz carlton restaurant  181 peachtree st.  atlanta   \n",
       "\n",
       "                 type        lat        lng  \n",
       "199       continental  33.758579 -84.387066  \n",
       "200  french (classic)  33.758579 -84.387066  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Dedupe false negatives\")\n",
    "for false_negative_pair in list(dedupe_false_negatives):\n",
    "    display(df.loc[list(false_negative_pair)][['name', 'addr', 'city', 'type', 'lat', 'lng']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Some of these false negatives could be prevented with better address normalization, but on others, the data is simply bad: different addresses for the same place, maybe a corner, maybe two entrances? We could try to **train more** the deduper to fix this. But we'll leave as it is and move to the last step of the deduplication process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4/4 - Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Using the Threshold or the SVM, we got the **matching pairs**. But what Dedupe returned to us were **clusters of matches**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((0, 1, 2), array([0.89782533, 0.8964192 , 0.88561887])),\n",
       " ((3, 4, 5), array([0.90827334, 0.91548833, 0.93204328])),\n",
       " ((7, 8), array([0.883838, 0.883838])),\n",
       " ((9, 10, 11, 12), array([0.91171577, 0.91171577, 0.91171577, 0.91134983])),\n",
       " ((13, 14), (0.91189873, 0.91189873))]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clustered_dupes[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dedupe went one step further on the process and **merged the matching pairs into clusters**! Why is that important? Because the following can happen:\n",
    "- We have the records A, B, and C\n",
    "- By deduplicating, we found the matching pairs (A, B) and (B, C). However, (A, C) was found to be a nonmatch\n",
    "\n",
    "It doesn't make sense to consider (A, B) and (B, C) as a match, but (A, C) as a nonmatch.\n",
    "The solution for that ambiguity is computing the **Transitive Closure** with Clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Using some private methods, it's possible to get the **unclustered pairs** from Dedupe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from dedupe.core import scoreDuplicates\n",
    "\n",
    "candidate_records = itertools.chain.from_iterable(deduper._blockedPairs(deduper._blockData(data_for_dedupe)))\n",
    "dedupe_matches = scoreDuplicates(candidate_records,\n",
    "                                 deduper.data_model,\n",
    "                                 deduper.classifier,\n",
    "                                 deduper.num_cores)\n",
    "dedupe_unclustered_found_pairs_set = {tuple(pair) for ([*pair], __) in dedupe_matches}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's **evaluate** those unclustered pairs against the clustered pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dedupe_true_positives total: 141\n",
      "dedupe_false_positives total: 7\n",
      "dedupe_false_negatives total: 9\n",
      "\n",
      "dedupe_unclustered_true_positives total: 142\n",
      "dedupe_unclustered_false_positives total: 10\n",
      "dedupe_unclustered_false_negatives total: 8\n"
     ]
    }
   ],
   "source": [
    "dedupe_unclustered_true_positives = golden_pairs_set & dedupe_unclustered_found_pairs_set\n",
    "dedupe_unclustered_false_positives = dedupe_unclustered_found_pairs_set - golden_pairs_set\n",
    "dedupe_unclustered_false_negatives = golden_pairs_set - dedupe_unclustered_found_pairs_set\n",
    "\n",
    "print('dedupe_true_positives total:', len(dedupe_true_positives))\n",
    "print('dedupe_false_positives total:', len(dedupe_false_positives))\n",
    "print('dedupe_false_negatives total:', len(dedupe_false_negatives))\n",
    "print()\n",
    "print('dedupe_unclustered_true_positives total:', len(dedupe_unclustered_true_positives))\n",
    "print('dedupe_unclustered_false_positives total:', len(dedupe_unclustered_false_positives))\n",
    "print('dedupe_unclustered_false_negatives total:', len(dedupe_unclustered_false_negatives))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We've found the unclustered pairs are different from the clustered pairs! That means the clustering process can both **create new matches** and **drop found matches**. Therefore, even though clustering is necessary to disambiguate the deduplication result, it can either improve or worsen the quality of found pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{(6, 7),\n",
       " (34, 37),\n",
       " (195, 198),\n",
       " (196, 200),\n",
       " (196, 839),\n",
       " (197, 198),\n",
       " (198, 200),\n",
       " (198, 839)}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "diff_set = dedupe_found_pairs_set ^ dedupe_unclustered_found_pairs_set\n",
    "display(diff_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here's a case where the clustering process got a **new correct match**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>3rd st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>None</td>\n",
       "      <td>34.068958</td>\n",
       "      <td>-118.320928</td>\n",
       "      <td>(3 street, 3rd saint, 3rd street, 3 saint)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w. third st.</td>\n",
       "      <td>los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "      <td>(8638 west 3rd street, 8638 west 3 saint, 8638...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>locanda veneta</td>\n",
       "      <td>8638 w 3rd</td>\n",
       "      <td>st los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>90048</td>\n",
       "      <td>34.073417</td>\n",
       "      <td>-118.381096</td>\n",
       "      <td>(8638 wohnung 3rd, 8638 wohnung 3, 8638 west 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>locanda</td>\n",
       "      <td>w. third st.</td>\n",
       "      <td>st los angeles</td>\n",
       "      <td>italian</td>\n",
       "      <td>None</td>\n",
       "      <td>34.068958</td>\n",
       "      <td>-118.320928</td>\n",
       "      <td>(west 3 saint, west 3rd street, w 3 saint, w 3...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name               addr            city     type postal  \\\n",
       "34  locanda veneta            3rd st.     los angeles  italian   None   \n",
       "35  locanda veneta  8638 w. third st.     los angeles  italian  90048   \n",
       "36  locanda veneta         8638 w 3rd  st los angeles  italian  90048   \n",
       "37         locanda       w. third st.  st los angeles  italian   None   \n",
       "\n",
       "          lat         lng                                    addr_variations  \n",
       "34  34.068958 -118.320928         (3 street, 3rd saint, 3rd street, 3 saint)  \n",
       "35  34.073417 -118.381096  (8638 west 3rd street, 8638 west 3 saint, 8638...  \n",
       "36  34.073417 -118.381096  (8638 wohnung 3rd, 8638 wohnung 3, 8638 west 3...  \n",
       "37  34.068958 -118.320928  (west 3 saint, west 3rd street, w 3 saint, w 3...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAFBCAYAAABJi8prAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XeYXWW59/HvbwJJCJEiJIQeahICoSMoYBSPqEhTRMpRelH0iMLreT3qEV8L6PGIFUIPvYi00Hsv0jQE0ggkQCCQkE4aSe73j+cZszNOMiUza83e+/e5rn3N7L3KvtfOZM0993rWcysiMDMzM7NiNJQdgJmZmVk9cfJlZmZmViAnX2ZmZmYFcvJlZmZmViAnX2ZmZmYFcvJlZmZmViAnX2bWpUiaKCla8ZjYwe/bIOms/Ph6M8t/XvHee3fke5tZfVmt7ADMzLqIBuAn+fsHgCtKjMXMapiTLzPrUiKif+VzSVGxTK3Zh6SeEbGgg0MzM+sQvuxoZlVL0lUVlwI/IekmSbOAl/Lyx/OyxU22W+51SScCH1assl/Ffu9v5q17S/qDpHclzZJ0j6StO+s4zay2uPJlZrXiVmC9/P3UTn6v4cAGFc8/C9wqaUhELOnk9zazKufKl5nVipnAnkAv4KC2bBgRFwOrV7z0QEQoPz7TzCbzgB2BjYGx+bXtgF3aHLWZ1R0nX2ZWK/4rIp6JiPkR8Uonv9evI2JkRLwN3F3x+uad/L5mVgOcfJlZrXixDeuu6pCLsRXff1Dxfc9V3K+Z1QEnX2ZWK+Y389rC/LWbpNUgzecF9G9m3WjmtRWpHJzflu3MzJx8mVlNm1Tx/YH56+ksP1gegDxQflZ+2l/SOp0cm5nVKSdfZlbLrq74/iZJc4D/pfkqGcDT+etWwIw81cSPOjNAM6s/Tr7MrGZFxAPAKcB40iXI8cCXgBdWsMlppAH0MwoJ0MzqkiI8XMHMzMysKK58mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgZx8mZmZmRXIyZeZmZlZgVYrOwAzs04n9QA2AdYFugOLgBnAm0QsKjM0M6s/Tr7MrHZJ6wCDgMGk890iYAnQjZSELUYaBYwhYmZpcZpZXfFlRzPrEJL6SwpJu5UdCwDS5sDhwHbANGAyMBWYPgS+1A9Oy68PBg7P65uZdTonX2Z1RtJpkkZKmp0fT0k6oMk6w3MiVfl4uqyY2ywlUgcA04H3SNWu5izJy6cDB1RjAiZpaP73Wb/sWMysdZx8mdWft4D/BHYBdgMeBG6RNKTJevcDG1Y8vlBkkO2WLjXuT0qqFrZyq4V5/f3z9mZmncbJl1mdiYhbI+KuiHg1IsZFxA+BOcBeTVZdGBFTKh7T2/pekvaV9IykBZLelXSupO4VyyXpDEnjJS2U9JaksyuWnyNprKT5kiZK+rWknhXLz5I0StIRkiZImtMPbh0Ja5ATr3nQsCsc1x2u7Q7X7gwnRpNz33dgl77w0x5w7uowSdI9kgat5Lg+K2mRpPWavP5LSSMrnn9c0iOS5kmaLOl8SWtVLH9Y0nl5u2mS3pP0G0kNFet0l/Sr/NnMk/SspP3zsv7AQ3nVqbkCNjwv+5ykxyTNkDS9pWMys+I4+TKrY5K6SToC6A082WTx3jkZGCfpIkl9m2w7XNLElex7Y+Au4EVgZ+AE4Ejg7IrVfgn8OL82GPgK8GbF8g+A40mD5r8JHAH8sMlb9Qe+ChzaFw6YB9sfXVGlOxAOGQn7HwV/HgZnLoWGMTC0cgdzoeeBcNtlcOZlcG5DSkZHVCaKTTxAGi/2lYrjFXAUcFV+vgNwL3AbsCPwJWAn4NIm+zoaWAx8HPgWcHo+nkaXAZ/M+94euDzHtmP+rL6c1xtMqlB+Jz9fE/gdsAfpeGe1cExmVhBFRNkxmFnBcmLwFNATmAscHRF3VCw/ApgHvE5Kbn5OukNw14hYmNc5G9gjIvbLz/vn9XePiOck/YI04H1ARCzN6xwLXECa8qGBlMCcHhHDWhn3qcCZEbF1fn4W8H+BDSJiFtJWB8FvH4Td58IpAGvA8D3gzkfgBoCFoPXh/DVh+hT4r2beZuOH4fFPwT+AT0bE4yuI5bf5WPfJz/cGHgE2j4i3JF0BfBgRJ1RssxMpGd0gIt6T9DDQIyL2qljnPmBSRJwoaStgPNA/It6oWOcW4O2I+KakoaTqV5+ImLaSz25NYPbKjsnMiuGpJszq01hSFWZt4DDgcklDI2IUQERcV7HuS5KeByaRBrHflNf5QQvvMQh4ujHxyh4nTfGwNSnx60GqIjVL0mGkStDWpOpct/yoNCkiZuXv1+0D0xbCOgCjoNcC+OieMAZgNKz1EmzYDybMSQkgANdBvx/Bv0+BbRfBOksgSMnhZis5vquA0yVtHhGTSBWsRyLirbx8V2BrSZVVLOWvW5HGmAGMZHlvA41Vxl3yNq+kwto/9SCN1VuhnLj9DPgY0CcfT0vHZGYFcPJlVociTSz6an76vKTdge+SLg02t/7bkt4CtumoEFpaQdKewHXAT3NsM4GDgN80WfXDiu+7ByyNZUkOAEvTjgaOgu0WQs/FsFFANE74dTL8d2+Ydir8eQ9Y8i5M+A+4mpQoNn8AES9IGgMcJek3pEuQ369YpQG4GDi3mc0nryB+WJb4Ne4jgN2bWW/+imLLbifdXHFKfr/FwCus5JjMrBhOvswM0i/5HitamKcx2Bh4pw37HA0cLqmhovq1N2mi0wmkvGchsB/p0lpTnwAmR8TPKuJoaSqIRQ0VY1m3h3k9YcZD8IX14Nnj4J5hcNA0WHMNWPp72HcwvDIHNvkmnH8OvAT0PQNWp3Xnx6tIFa9RpDFWN1YsewEYHBGvNrdhK71ISiT7RcRDK1incYb+f1YE840AA4FvNm4naRd8zjfrEjzg3qzO5DsI91GaFHWHPHZrKKnSg6Te+Y67vfI6Q4ERpMtkN1fs52xJK7xkCJwHbAScJ2mQ0lxi5wB/ioh5ETEH+D1wtqTjJG0laQ9J38jbjwM2lnS0pC3z60e2cHgzuuUEY0kKdqvB8No/YJexsOAxWOc22GUBrN0A09eFmY/CPt1h7o2w/zWw4W9hu/NSBWtxKz7Oq0mTuP4MGBERsyuW/QrYQ9IwSTtL2lrSFyVd0Ir9AhAR4/J7DJd0WP4cdpN0pqQv5dUmkapjB0jqI6k3qXXSNOCk/L6fBIa18pjMrJP5ryCz+tOPVLHpR7oDbiTw+Yi4Jy9fAuwAfJ00duod0oDuw3PC1GhD0tilZkXEZEmfB/4H+DvpsuE1LD/I/QekROHHpN6L7wJX5O1HSPof0h17a5DuHPxvUlK3Im8tTfHrXPjkAlhjGPzhJPjS1fCtq4H+MHkgPDwVNj0ORv4N3n4F+jwIe34N/rQuTO0OJy+A61f2IeYYJ0l6HNgnx1a5bKSkfUk3KzxCqky9RkUC20rHke7w/DXpM5oO/I08xUT+nH8C/IJ0mfOKiDg2jzX7A6kq9ypwBvDXNr63mXUC3+1oZjVluHT0B3BIb3jxCHi5Rxry9U/fhyN/DddWvjYbVrsWdlwKg/rA9YdF3FRs1GZWT1z5MrOaIGkN4IAtYaN74dmtUpVpaUvbAawFi0+BUS/DhwfCgK+kS6T35RsTzMw6lMd8mVnVk7QN8A1g7mvwu63SJcO+rOQmgiZ6AH0Hw1Wvp7sTuwOnStq0cyI2s3rmypeZVS1JPYDPksae3RwRr+dFk5DuIPV4XAK8T/PNtbsB6+WvdxAxKQ/EuDm34vmqpL8DD0eEB6ubWYdw8mVmVSlPO3EIMBE4v3Hm/X+KmIR0AzCAdAPBasCHfdLg/b6k6SQWk6aXGEvEzOU3j9GS3gAOJN01eHNETOncozKzeuDky8yqiqTVgE+TEqrbI2LsCldOCdUzSC8CmwLrvJx6KDbeffkmKxnXFREfSLoeGAJ8XdJTwBNNZu03M2sT3+1oZlVD0oakBtVTSYnXvHbs46yIOKsd260NHEwaD3ZzRLzf1n2YmYErX2ZWBSQ1kObS2gO4B3gpCv7LMSJmSboyx3BCbor9bNFxmFn1c/JlZl1abm10KLAAuKDJLPKFyonWM5JezTENlHRrRWNvM7MWeaoJM+uSlOwJHE8ao3VVmYlXpXzJ8VLgdeBkSTtKUgubmZkBrnyZWRckaR3SnYzdgIsjYnrJIf2LPOj+MUnjSePQBkkaEREflByamXVxrnyZWZeRq107AycD44HLumLiVSlPP3EhqZH1N/L8YGZmK+TKl5l1CZJ6k+bUWhu4PCLeLTmkVssTsN4vaRxwiKSBwF0RsaDk0MysC3Lly8xKJ2k74FTgPeCiakq8KkXEG8AwYBGpCrZlySGZWRfkypeZlSY3w/48sDFwXUS8VXJIqyw3475D0lhSFWwMqUn3hyWHZmZdhCtfZlYKSVuTmmHPB4bVQuJVKSJeBc4HepKadG9Sckhm1kW48mVmhZLUndQMexvgloh4reSQOk1EzAduypdVj1Bqc/RwRDTX5NvM6oQrX2ZWGEmbkcZ2rUZqhl2ziVeliHiFNBZsA1KT7g1KDsnMSuTKl5l1utwMeyiwE6kn45hyIypeRMyVdC3pMzhG0pPAk27SbVZ/nHyZWaeS1I/Uimc6qdpVt5OQ5vZEL0p6nTSJ7LaSbunqc5mZWcdy8mVmnSI3w94b+BhwLzDSTaiTiJgp6XLSZ3OipIeA5/z5mNUHJ19m1uEkrUeqdi0CLnTj6X+VE62nm2nS3SX6V5pZ5/GAezPrMLk90B7ACcBI4EonXisXEdOAS4A3gFMkDXGTbrPa5sqXmXUISWsDBwPdgUsi4v2SQ6oaedD9I7lJd2MV7PaImFdyaGbWCVz5MrNVkqtdOwGnAK8Dlzrxap+IeBu4AJhJak80oOSQzKwTuPJlZu0maU1SM+x1gSsiYkrJIVW93KT73or2RAOBuyNiYcmhmVkHceXLzNpF0iBSe6BppGbYTrw6UERMIk3MupRUBdui5JDMrIO48mVmbSKpJ6kZ9qbA9RHxZskh1axc7RohaRvgS5JeBh5wk26z6ubKl5m1mqQtSdWuRaRm2E68ChAR40lNunuT7ojcuOSQzGwVuPJlZi2StDrwb8BA4NaImFBySHUn3/l4o6TtgaMkPQc86ibdZtXHlS8zWylJm5CaYfcktQdy4lWiiBhFGgu2EWl2/L4lh2RmbeTKl5k1S1I3UjPsnYE7I+KVciOyRhExR9I1wC7AsZIeB552k26z6uDky8z+haQNSJN9ziKN7ZpbckjWRG5P9Lyk10hNugfkJt0zSg7NzFrgy45m9k+SGiTtDRwDPA1c58Sra8vJ1uXAWOAkSbu6PZFZ1+bKl5kBIOmjpGrXYlIz7Jklh2StlC83PtmkSfdtETGn5NDMrBmufJnVudweaHfgROBl0kz1TryqUES8B1wMvA2cmu+MNLMuxpUvszomaS1SM+yepJ6M00oOyVZRnnriIUnjgENzJ4I73KTbrOtw5cusDuVq1xBSM+xJwCVOvGpLREwmNemeTWpPtE3JIZlZ5sqXWZ3JzbAPANYHroqId0oOyTpJbkN0T0WT7gnAvW7SbVYuV77M6oikAaQJU2eQBtU78aoDETGR1J6ogTQWbPNyIzKrb658mdUBST2AzwH9gRsjYlK5EVnRcrXr1pyAHyZpFKlJ9+KSQzOrO658mdU4SVuQmmEvJU2Y6sSrjkXEWFIVbG1Sk+6NSg7JrO648mVWo3Iz7M8Ag4ARETG+5JCsi4iIeZL+AmwPHC3pWeAxN+k2K4YrX2Y1SNLGpDsZe5GaYTvxsuVE8hLpjshNgBMk9Sk5LLO64MqXWQ3JzbD3BXYjNcN+ueSQrIuLiNmSrgZ2BY6T9CjwTO4daWadwMmXWY2Q1JfUWmYOaWyXW8tYq+RE67mKJt0Dc5Nudzow6wS+7GhW5XIz7E8AxwLPAtc68bL2iIjpwHBgPHCypF3cpNus47nyZVbFJK1LqnYFcFFEzCg5JKtyuUn3E8006Z5bcmhmNcOVL7MqlNsD7QacBIwGhjvxso4UEe8CFwFTSBOzDi45JLOa4cqXWZWR9BFSM+xewGURMbXkkKxG5aknHqxo0j2QdCPH/JJDM6tqrnyZVRFJ25PaA71FaobtxMs6XUS8BQwD5pGadG9dckhmVc2VL7MqIKkXqRl2X+DqiHi75JCszuQm3XflJt0HSxpPatK9qOTQzKqOK19mXZykbUntgWaTmmE78bLSRMRrpPZEq5HGgm1WckhmVceVL7MuKjfD3h/YEvhrREwsNyKzJCIWALfkMWCHS/oH8JCbdJu1jitfZl2QpP6kahek9kATSwvGbAUiYgypCvZR0rxg/UoOyawquPJl1oVIWg3Yj9TweEREjCs5JLOViogPJN0ADAG+JukZ4PE8X5iZNcPJl1kXIWkj0qSW75GqXfNKDsmsVXJ7on9ImkiaBmXb3J5oWrmRmXVNTr7MSpabYe8D7A7cFRGjSg7JrF0iYpakK0k/y8dLegT4m5t0my3PyZdZiST1IVW75uFm2FYDcqL1N0kTWL5J96ySQzPrMjzg3qwEuT3QXsBxwAukubuceFnNiIj3gcuACcApknZyk26zxJUvs4JJWodUEWgALo6I6SWHZNYp8qD7x5s06R4RER+UHJpZqVz5MitIrnbtApwMjCM1w3biZTUvIqaQmnRPI7UnGlRySGalcuXLrAC5GfaBwEdISdd7JYdkVqg8Aev9uT1RY5Puu/KErWZ1xZUvs04maTCpGfYU0mVGJ15WtyLiTVKT7oWkKthWJYdkVjhXvsw6iaQ1SM2w+wHXRMTkkkMy6xJyM+47K5p0jwXuc5NuqxeufJl1Aklbk9oDzQUucOJl9q8iYgKpPVEPUpPuTUsOyawQrnyZdSBJ3YHPAlsDN0fE6yWHZNalRcR84KY8CP+rkl4EHnGTbqtlrnyZdRBJm5GqXd1IE6Y68TJrpYgYTRoL1gc4SdIGJYdk1mlc+TJbRbkZ9qdIjYVvj4ixJYdkVpUiYq6k64EdgWMkPQU84SbdVmucfJmtAkkbkiaPfJ9U7fLkkWarILcn+ruk10mTETc26X6/5NDMOoyTL7N2kNQA7A18DLgHeMnNg806Tm7SfQWwB3CCpIeA5/z/zGqBky+zNpK0PqnatYB0J+PskkMyq0k50XomN+lubE90q//PWbXzgHuzVsrtgT4GHA/8HbjKvwTMOl9ETAMuASaRmnTv6CbdVs1c+TJrhdwM+2DS/xk3wzYrWB50/6ik8Syrgt3ucZZWjVz5MluJXO3amdQMewJwmRMvs/JExDvAhcB0UnuigSWHZNZmrnyZrYCk3qRm2OsAl0fEuyWHZGb8s0n3fU2adN/tJt1WLVz5MmuGpO1IzbDfAy504mXW9UTEG6T2RItJVbAtSg7JrFVc+TKrIKkn8AVgY+D6iHiz5JDMbCVyM+7bcz/VQyWNBu6PiA9LDs1shVz5MsskbUVqD9Q4hYQTL7MqERGvkqpgvUhNujcpOSSzFXLly+pebob9b8C2wK0R8VrJIZlZO+Qm3X+VNBg4UtLzpCbdS0oOzWw5rnxZXZO0KWlsV3fgfCdeZtUvIl4mVcH6kZp09y05JLPluPJl1U/qAWwCrEtKohYBM4A3SeNBmtlEqwFDgZ2AOyJidDHBmlkRcpPua4GdgWMlPQE8tdIm3e04l5i1h5Mvq15p4tNBwGDSz/IiYAnQjXTiXIw0ChhDxMxlm6kfaZLGGaRqlydpNKtBuT3RC7lJ98HAgNyke/m5+tp5LjFrL192tFaR1F9SSNqt7FgAkDYHDge2A6YBk4GpwPQh8KV+cFp+fTBwONLmkhok7QN8DXiSdDejEy+zApVxLomIGcDlwGjgREm7/bM9UTvOJUXFbbXLyVeNkXSapJGSZufHU5IOaGa9bSXdJGmmpHmSXpA0qIyY2yyd/A4gzXD9Hukv1OYsycunT4HD94MzgS1I83b9I/9V3KVJGpp/Ua1fdixWX1p7LqlY/4L8s3pmkXG2ViRPAZcBuwBH358G5rfpXAIcUI0JmM8lXYuTr9rzFvCfpJPLbsCDwC2ShjSukCcifAJ4Hfg0sD3wI2Bu4dG2Vbo8sD/pRLiwpdWXADfCZhfDkJ/AOjPhtoiY1dlhmtWAFs8ljSQdBuwBvF1ohO0QEVOBS3aHma/CTx+AnktacS7JFpLOPfvnc5FZuzj5qjERcWtE3BURr0bEuIj4ITAH2KtitV8A90bEGRHxQkS8FhF3tnVeK0n7SnpG0gJJ70o6N0/b0Lhcks6QNF7SQklvSTq7Yvk5ksZKmi9poqRf50lOG5efJWmUpCMkTZA0px/cOhLWIJ8s50HDrnBcd7i2O1y7M5wY+ed6EvT6X/jUlbDvH2D3z8B31odJku5ZWZVP0mclLZK0XpPXfylpZMXzj0t6JFcOJ0s6X9JaFcsflnRe3m6apPck/UZSQ8U63SX9Kn828yQ9K2n/vKw/8FBedWr+q3V4XvY5SY9JmiFpekvHZNZWrTyXoFQF+j1wFNCuiU2LPpcAM9+A/7MFjHwCtvkjfGIS9FzRuaTRd2CXvvDTHnDu6j6X2Cpw8lXDJHWTdATQmzTGifyf9UDgFUl3S5qa/5N+tcm2wyVNXMm+NwbuAl4k3U10AnAkcHbFar8EfpxfGwx8BahM8D4AjicNdP0mcATwwyZv1R/4KnBoXzhgHmx/dJqBHtKBHDIS9j8K/jwMzlwKDWNgaEDPi2D/jeHd9WD0gXDzZXDmZXBuQ/oFMqLy5N7EA6QxHl+pOF6RfrlclZ/vANwL3AbsCHyJdOfkpU32dTSp9cnHgW8Bp+fjaXQZ8Mm87+1J41JGSNoxf1ZfzusNBjYEvpOfrwn8jlRtGArMauGYzNqtuXNJfn014Frg5yu6Y7grn0vOhE98D+7pDfO+DP9nJHy+6bmkcgdzoeeBcJvPJbbKIsKPGnsAO5AuIS4GZgIHVCzrBwTpZPU90n/y7+V1K9c7G3ig4nn/vN1u+fkvgPFAQ8U6x5IqUr1IJ+kFwKltiPtU4NWK52flfawdEQRsdSDcuia8HXBgwIE94f194crG55PgK71heh+Y/A/4WuPrFY9TH0onpiXA3iuJ5bfAYxXP987bbJKfXwFc0mSbnfJn1Dc/f5h0a3vlOvcBF+fvtwKWAps1WecW4Lz8/dC8z/Vb+OzWbOmY/Fj2c1V2DNXyWNm5JC//BelSfuPzicCZTdaplnPJzJ3h2T/Dz2bAoQvgoN4weQN4qZnziM8lfqzSw1NN1KaxpP+8awOHAZdLGhoRo1hW7bw1In6bv/+70p1H3wLuAIiIH7TwHoOAp2P5OXMeJ92WvTXQE+hB+suvWUrjRE7P6/cm3dbdrclqk2LZGK11+8C0hbAOwCjotQA+uieMAZgNq/0BDusDb3wA3YakXxZcB/1+BP8+BbZdBOssSSegBmCzlRzfVcDpkjaPiEmkvzofiYi38vJdga2bVAyVv25FGhcCMJLlvQ00Tvi4S97mFUmV6/Qgja9ZIaVWSD8DPgb0ycfT0jGZtdUKzyWShpKSpJ1WtoMqOpesPRRGLIXuZ8NhP4frN4Sxs9P/L8DnEus4Tr5qUKTJAF/NT5+XtDvwXVI5fxrpr9hXmmw2mlSq75AQWlpB0p7AdcBPc2wzgYOA3zRZtXIMSfeApbHsxPQvPoQ1lsDGSyG+ny5dcD4c0AvmDYVX+sPS12Hmnam0f7ikbVcS5vvAJZKeBI4B7pN0Vl7WF/gH8HQz2x2Qx1r0B3pVbAPpskLja4Pza8NJf7Uudyh5nf75+fclzatYfhowm3TjxOy8/WmtOCaDoU3+Tax1FpJ+1q6QdBupkrIR8G7FL3wBv5b036SKT3MaB6qfLOmLpORrUZN/k4/mr6cCq+fvvy1p+fm5kk1IlxwfJl26WwAMAD5bsc+hwLqNz4+BTd+DnQJW/z4c+UF+j7HwmZkp2YtrUlVrOSfDf/eGaafCn/eAJe/ChP+Aq0mJYrMi4gVJY4CjJP2GdAny+xWrNAAXA+c2s/nkiu+bjqdrTPwa9xHA7s2sN39FsWW3k26uOCW/X+PvB1927EROvupDA+kvICJikaRnSSenStsCk9qwz9GkX/QNFX+x7k2anHAC6a/OhcB+pEsKTX0CmBwRP2t8QS3fvr2ooWKc4vYwrydMfyYdy8i1YPE5cPGlsFdPWLIPPLU+vP8/cORp8Ktz4CWg7xmw4M70V/wtETF8RW8maRFp/MSzpF8qX4uI2XnZNsDGEfEfK9l+KDAqIs6qeK0/qex/Vk6SDgP+GhEPrWAfHyclfv8bEe/m19YDfgIc3ridpF2Ab7d0TJYGX1f+m1jrSdoXeDv//PZlWeWl0T2kMWAXRcTYFeyjP2m80YUR8Zyk1UnzbP2/xnOJpGNJ55IzSeeSrwGvRMSwZvZ3BvBmRHy64rU/AJ9t/HfOSdf6//x3l3Y7Eb4BbPdruHYJcCl89h1Y83i4+hAYH8C34Zu90mTM/A0+Mgc2+SacX3EuWZ3W/R69ilTxGkW6rHdjxbIXgMGRGoO314ukc1S/FZ1LSJ8nVFQE87lkIPDNJucS5wadzB9wjZF0DunS4ZvAR0jJw1DSXDaNfg3cIOkxUkn6U6Sq1yEV+zkb2CMi9lvBW51HKvOfJ+n3wJbAOcCfImJe3sfvgbMlLQQeBdYDdo2I84FxwMaSjgaeIk0fcWQLhzejW5Of2b1gxONw2PEweV+Y9Dv4wgJYsxdMehG26wWzu8PsG2H/ITBtCgw4Dz5D+uvLfCEOAAAVkUlEQVSuJVcDPyeV5Ec0Jl7Zr4CnJQ0DLiANvB0IHBgRp7Ri30TEOElXA8PzL5AXSH/xDwVei4ibSAlxkKppI0h/xc4gVTBPkvQmsDHwP608JrNWaelcEhHvseySWOM2HwJTKhOvrn4ueR3WvAH23AFGvQg73QF95sC838EX5sNHG5OvHWGuzyXWYcoedOZHxz5Il7AmsWw+mvuB/ZtZ71jSSWs+aSzBkc3sZ2LF8/5UDJLNr+0LPJPf611S2bxHxfIG4P8Cr5H+6noT+EXF8rNJM0nPBW4i/SUaFcvPIlWOUlzQ40S4tBvMbxz0OhsO3gluWR3mrg5zh8CI7eHODeCleXDwJfCjg+GutWByAyxaDyavlX55zAWObcXn+Wg+7oOaWbYbcDfpUswHpL+G/1/F8odJv0Cafq63VzxfPR9n42c0hXTpZNeKdX4MvEO6tDg8v/Zp0l/RC/LX/Vt7TPX+wAPuW/s5tepc0mSbifzrgPsueS45IZ1LFv4QLrsGvv8+HLKic0nj+eb78MO1YJLPJX6s6kP5wzerDtJepDYg77W0aqNnYP27YM/B8OEn4PqNIh7rvACtq/NlR5PU+3vwvUGwzcfg3h3yzTlt0BcYRcQznRGf1T7P82XVZjRpzEKP1m7wMZh2BjzUA2Jv2FvSlp0Xnpl1ZZK2A059FF48Bp7YoeUB6U31IJ2Dmh3TZtYarnxZ9VnW27FVLYZIJ8u+wB1KpfmDSNNT3BcR7ZqR26qXK1/1SdIapAmaNwJujoi3VuVcQpo2wqxdXPmy6pNOeneQBpT25V/n82nULS//KPlkGemOovNJt5OfKmmTAiI2sxJJ2po0DmweMCwa59hahXNJZ8dstc2VL6teqbHtANIs3KuR5rdZQjpRrk66Y+clYCwR/zKmI19++ALpzqBHImJJQZFbiVz5qh+5Rc5ngW1IE0u/toIVV+lcYtZWTr6s+qUT7KakyRt7kC4fzATeJE04u5JN1ZvU63Jt0qWIdzs5WiuZk6/6IGkz0vQ5bwB3R8SCVmzU7nOJWVt4ni+rfumkOKF9m8ZcSdeRWqQck2ezfzKWb3ViZlUiN/v+FKmbxO0RMabVG6/CucSsLZx8Wd2LVP59UdLrpL+Ut5V0S0Q018rEzLooSf1IrcPeB86PiA9KDsmsWR5wb5ZFGstxOamv2YmSdlOTLrVm1vVIasitj75O6nd6gxMv68pc+TKrkKtgT0t6FTgUGCjptli+HYiZdRG5P+GhpFndL4iIWSWHZNYiV77MmhER04BLSG1MTpE0xFUws65DyceAE0gt0q504mXVwpUvsxXIg+4fkTSONI5koKTbIzf7NbNySFqbND5zdeCSiHi/5JDM2sSVL7MWRMQ7wAWkW86/IWlAySGZ1aVc7doJOIXUQPpSJ15WjVz5MmuFiFgM3CtpLHCIpIGkuYNa05LEzFaRpDVJc/KtC1wREVNKDsms3Vz5MmuDSG1FhgFLSVWw/qUGZFYHJA0itQeaBlzkxMuqnStfZm2Uq10jJG0DfEnSK8ADbtJt1rEk9QQ+T5p1/vqIeLPkkMw6hCtfZu0UEeNJTbp7k+6I3LjkkMxqhqStSNWuRaRm2E68rGa48mW2CiJiPnCjpMHAUZKeAx51k26z9snNsD8DDCQ1w3a7H6s5rnyZdYCIeJk0Fmwj0uz4fUsOyazqSNqUdCdjT1J7ICdeVpNc+TLrIBExR9I1wC7AsZIeB552k26zlZPUDRgK7AzcGRGvlBuRWedy8mXWgXJ7ouclvUaaBHJAbtI9o+TQzLokSRuQ2gPNIo3tmltySGadzpcdzTpBTraGA2OAkyTt6vZEZsvkZth7A8cATwPXOfGyeuHKl1knyVWwp3KT7sb2RLdFxJySQzMrVW6GfQiwGLgwImaWHJJZoVz5MutkETEVuBiYDJwqafuSQzIrRW4PtDupGfYo0kz1Trys7rjyZVaAPPXEw5LGA4fmGbvvcJNuqxeS1gIOJt3JeGlETCs5JLPSuPJlVqCImExq0j2b1J5om5JDMutUudo1hDSFxCTgEideVu9c+TIrWG5DdE9Fk+4JwL1u0m21JjfD/iKwHnBVRLxTckhmXYIrX2YliYiJpPZEIo0F27zciMw6jqQBwKnAdNKgeideZpkrX2YlytWu2yRtCxwmaRSpSffikkMza5fcDPtzwObAjRExqeSQzLocV77MuoCIGEeqgq1NatK9UckhmbWZpC1I1a4lpAlTnXiZNcOVL7MuIiLmSfoLsD1wtKRngcfcpNu6Okmrk5phDwJGRMT4kkMy69Jc+TLrQiJ5iXRH5CbACZL6lByW2QpJ2ph0J2MvUjNsJ15mLXDly6wLiojZkq4GdgWOk/Qo8EyeNd+sdLkZ9idJP6N3RsTLJYdkVjWcfJl1UTnReq6iSffA3KTbM4JbqST1JTXDnkMa2+WWWWZt4MuOZl1cREwnNekeD5wsaRc36bYy5GbYnwCOBZ4FrnXiZdZ2rnyZVYGIWAo8kZt0H8qyJt1zSw7N6oSkdUk/ewFcFBEzSg7JrGq58mVWRSLiXeAiYAppYtbtSg7JalxuD7QbcBIwGhjuxMts1bjyZVZl8tQTD0oax7Im3XdGxPySQ7Mak5thH0S6k/GyiJhackhmNcGVL7MqFRFvAcOAeaQm3VuXHJLViFzt2oE0hcRbpGbYTrzMOogrX2ZVLDfpvkvSGFKT7vGkJt2LSg7NqpSkXsABQF/g6oh4u+SQzGqOK19mNSAiXie1J1qNNBZss5JDsiqUe4x+A5hNaobtxMusE7jyZVYjImIBcIukgcDhkv4BPOQm3dYSST2A/YEtgb9GxMRyIzKrba58mdWYiBhDqoJ9lDQvWL+SQ7IuTFJ/UrULUnugiaUFY1YnXPkyq0ER8YGkG4AdgK9JegZ4PM8XZtbYDPvTpEbuIyJiXMkhmdUNJ19mNSq3JxopaRJwMLBtbk80reTQrGSSNiJNmPoeqdo1r+SQzOqKky+zGhcRsyRdCewOHC/pEeBvbtJdf3Iz7H1IPwt3A6P8c2BWPCdfZnUg/4L9m6QJLN+ke1bJoVlBJPUhVbvm4WbYZqXygHuzOhIR7wOXAROAUyTt5CbdtS1PmLoXcBzwAmnuLideZiVy5cuszuRB9483adI9IiI+KDk062C5GfYhgICLI2J6ySGZGa58mdWtiJhCatI9ldSeaFDJIVkHydWuXUjNsMeSmmE78TLrIlz5MqtjeQLWByqadA8E7soTtloVkvQR4EDgI6Sk672SQzKzJlz5MjMi4k1Sk+6FpCrYViWHZO0gaTBwKjCFdJnRiZdZF+TKl5kBkJtx3ylpLHBQrobd5ybdXZ+kNUjNsPsB10TE5JJDMrOVcOXLzJYTERNI7Ym6k5p0b1pySLYSkrYhtQeaC1zgxMus63Ply8z+RR7zdXMehP9VSS8Cj7hJd9chqTupGfZWwM0R8XrJIZlZK7nyZWYrFBGjSWPB+gAnSdqg5JAMkLQZqdrVQJow1YmXWRVx5cvMVioi5kq6HtgROEbSU8ATbtJdPEmrAZ8ChgC3R8TYkkMys3Zw8mVmLcrtif4u6XXSpJ2NTbrfLzm0uiFpQ9KkuO+Tql2eFNesSjn5MrNWy026rwD2AE6Q9DDwrJszdx5JDaRm2HsA9wAv+fM2q25OvsysTfIv/mdyk+5DgQGSbo2I2SWHVnMkrU/6jBeQ7mT0Z2xWAzzg3szaJSKmAZcAk0hNund0k+6OkdsD7QkcD/wduMqJl1ntcOXLzNotD7p/VNJ4ljXpvt3jkdpP0jrAwaTz8yUeV2dWe1z5MrNVFhHvABcC00ntiQaWHFLVydWunYGTgQnAZU68zGqTK19m1iHyBKz35fZEjU2673aT7pZJ6k1qhr0OcHlEvFtySGbWiVz5MrMOFRFvkNoTLSZVwbYoOaQuTdJ2pGbY7wEXOvEyq32ufJlZh8vNuG+XtDWpCjYauD8iPiw5tC4jN8P+PLAxcH1EvFlySGZWEFe+zKzTRMSrpCpYL1KT7k1KDqlLkLQVqT1Q4xQSTrzM6ogrX2bWqSJiPvBXSYOBIyU9T2rSvaTk0AqXm2H/G7AtcEtEvFZySGZWAle+zKwQEfEyqQrWj9Sku2/JIRVK0qaksV3dgfOdeJnVL1e+zKwwuUn3tcBOwLGSngCequUm3bkZ9lDSMd8REaPLjcjMyubky8wKldsTvVjRpHtAbtI9veTQOpykfqTJZ2eQql2efNbMfNnRzMoRETOBy4FXgBMl7VYr7YkkNUjaB/g68CTpbkYnXmYGuPJlZiXKVbCnK5p0D5R0WzX3MZS0HulYFpHuZJxVckhm1sW48mVmpYuIqaQm3W+RmnTvUG1VsNweaA/gBOAl4EonXmbWHFe+zKxLyFNPPCxpHKlyNCg36Z5XcmgtkrQ2qRl2d9wM28xa4MqXmXUpEfE2cAEwk9SeaNuSQ1qhXO3akdQM+3XgUideZtYSV77MrMvJTbrvzU26D8lNuu+JiIUlh/ZPktYEvgisR7rEOKXkkMysSrjyZWZdVkRMAoblp9+Q1L+8aJbJyeA3gOmkZthOvMys1Vz5MrMuLVe7bsuXH78s6WXggTKadEvqCXwO2Ay4ISLeKDoGM6t+rnyZWVWIiHGk9kS9SXdEblTk+0vaklTt+hAY5sTLzNrLlS8zqxr5zscbJW0PHC3pOeDRzmzSLWl1UjPsgcBtEfFqZ72XmdUHJ19mVnUiYpSkScBBpNnxb8pzhTVP6gFsAqx7DGyKtBup5c+bRCxa8WbahDTtxWRSe6D5HXkcZlafnHyZWVWKiDmSrgF2AY6T9Djw9HJNuqV1gEHAYNL5btFg+CgwhDQn12KkUcAYUrujvJm6AZ/M+74zIl4p6LDMrA54zJeZVR1JIemwSJ4HLgIGAMdIWjevtDlwOLAdMI1UvZo6FeYDU/PzaaTE7PC8PpI2AE4C+pHGdrU78ZJ0pqSJ7d3ezGqTky8z6zSSTpM0UtLs/HhK0gEVy2MFjz+35X0iYgapSfdY4KTTpAOWwgGkqSDeA1Y0JmxJXj59MXzxOOnLwDHAM8C1ETG37UdtZrZyTr7MrDO9Bfwn6fLdbsCDwC2ShuTlGzZ5HJhfv6GtbxQRSyPiye/ATbvDEefBtpNbeY57FVb/I2y3Hxz4M7g+Il7MTb/NzDqcky8z6zQRcWtE3BURr0bEuIj4ITAH2Csvn1L5IPVHHBcRj7TlfXIj7vslzf8zvPA72HIRzDofPndXmpOLr8Kn14E//i98tSdcMQBOXwL8FbY5Cr77Uzj0ODjyZ/CspIuVxos17v9YSXMl7SdplKQPJD0kaYsmcXxf0pS87hWkaTEql+8u6V5J03Il8HFJe7XnszWz6uXky8wKIambpCNICcmTzSzvDRxBGr9V+fpZklZYhcptfu4B5m4Cn/glDHsN+v8RPnMoPPI32GEofPNGOG0PuP9IuPPn8NON4O3fwtAxsOVHYMwp8Oe/wLd/DsMFHwP+2OStegA/AI4nJY/rsGz2fSQdDvwc+Amp0jcW+F6TfXwEuBLYB9gD+Dtwp6T1VvrhmVlN8d2OZtapJO0APAX0BOYCh0bES82sehTpDsTLm7w+jZTIrMhRwJrA196EvsBr0+FP58Avx8LlZ8Dd58Dlg2D0D+Bvd8HQgbBkJnzYD6YdDq/8EJZW7K/hbvjtgzBM0jEVd0+uBpwWEWPzcf0GuFSS8iXK04HLI+KCvP4vJH0K2LpxxxHxYJPP5tvAl4HPA1et5BjNrIY4+TKzzjYW2AlYGzgMuFzS0IgY1WS9k4Bbm87XFRF/Av60kv0PAkZGxBykAcCiM2HMObD0Udh0U5i3ANbaCx54GHafCtu+ALO+DI/snAbk858w5Eo4bCZsuhh6L4EgJYL9gLfz+yxsTLyyt/M665L2Mwi4uElsT1GRfEnqC/wM+BSwAdANWIN8adTM6oOTLzPrVJEmMW2cFf55SbsD3wVOaFxH0k6kAfn/tYpv152KOxuVkigA1oIZ34XHhsP7p8BTa+Rq183Q5zfw3zvAvd+Aq7eHbpdCj9vhp3l/jRY3PbT8tS3DNy4nJV3fBSYCC4EHmryPmdU4j/kys6I1kMZPVToZeB24vx37Gw3sIOkjwCKg229SK6CGfeCtfWDWGvD+Y7DjOrD4dHhijYrLjLfDNkthtSfh4h/D2ENh2ttpPFd74tizyWtNn+8N/DEi7oiIl0k3H2zYjvcysyrm5MvMOo2kcyTtI6l/viPxbGAocHXFOr2Ao4FLmpveQdK3JI1ZydtcDcwDrjgR+pwL2/0ZTusPTx4F7wDsBzc8DwftDwdfARv9Hrb4NzgEYOd0+bDhYDjoL7DBd2CvkWngf1v9njTJ60mStpH0A9LA/UrjgH+XtF2uAF5HShjNrI74sqOZdaZ+pIHk/YBZwEjg8xFxT8U6XyUNmL9sBftYnzR7fbMiYp6k/YHfXQI3dYelW8Izf4HGge+MgLu+DIvvh0Pug2O6w9wt4DmAb8HEO+DCh+GwB+HfN4XXesAZi+GathxoRFwvaUvgF0Av4Dbgt8CxFasdD1wIPE9K+s4C+rTlfcys+snzCJpZTUnzZm1Hmrm+rfoCo4h4pmODMjNbxpcdzazWjCbdRdh0XFlLeuTtVjathZnZKnPyZWa1JWImadLVvrQ+AeuR178nb29m1ml82dHMapO0ObA/aeqJ92m+uXY3YL389R4iJhUXoJnVKydfZla7Un/GAcAOpBuMPiQlYd2A1Ulzd70EjHXFy8yK4uTLzGqf1B3YlDR/Vw/S5KYzgTdJk8CamRXGyZeZmZlZgTzg3szMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxATr7MzMzMCuTky8zMzKxA/x899zpUle0c8QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAFBCAYAAABJi8prAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XecXWW1//HPNyGFECGUhNAjNSH0JkgxihI1P6p09AJSBbzqBfF6bWADlSuWS28BQUCRFop0EKRI0xBKEgJECCWEJISQRpL1++N5huwM0zOz95yZ7/v1Oq855+xy1jnJ7Fln7Wc/SxGBmZmZmZWjR9UBmJmZmXUnTr7MzMzMSuTky8zMzKxETr7MzMzMSuTky8zMzKxETr7MzMzMSuTky8waJek0SZFvp7V0WTvHMLrwOkM66nXya52Wb0d05Ou0B0mv1H0uVcdiZq2zXNUBmJl1Ij/KPx8ARlcYh5l1Ya58mZmVRFLfqmMws+o5+TKzdlPvFOEnJV0paYakdyT9RdLgeuv3lnSypCckvSdprqSJkv63mdcZ0dApzyae31LS9ZKmSJqf43lK0gWSekk6ot7pu08V9nN/YT/DJP0h72eBpKmSrpO0RROfw655nXeB5wvr7CjpBklvSfpA0ut5uyH19iVJ/y3p3/nzeUjSdi345zCzTsqnHc2so9wKDCg83g9YCfgsfFgFuhvYud52GwJfAk5ujyAk9cuvs1rh6VXybeuWvo6kXYA7geULTw/MsY6StEdEPNjApjcAq+b7M/K+DgT+CPQsrLcGcDiwp6RPRsT4/PyPWHI6FNLndS+glsRtZp2PK19m1lFeBjYANgam5ud2l7RGvv+fLEm8ngN2AlYAhgPntmMcw1iSeJ0K9CUlTbsAZwALI2J0RBSTmQciQvk2Ij93ESnxmgxsC/QhJW9v532e08jrzyK9t+WBL+Zk8DxS4vUUMDTv69PAAlJS+CsASQOA7+T9zAdGkhLai4D+bfgszKwTcOXLzDrKDyPiJQBJD5IqRADrAW8AexXWPSEiHs33n8u39vIqsJB0vDuYlAS9ADwVEf/Tkh1I2oiUJEGK/8kGVttc0uCIeLPe898vvjdJnyMlWADb5Fjq+1z+uRMpsQO4LSLuzPF8H/g60Ksl8ZtZ5+LKl5k1ZV7hfr96y4qP5zaw7fjC/fcL9+uSidULz7VXsvWRL5QRMRU4HphGSnZOB64FJkp6UNKKLdjvoBa+/qoNPPd0G/bVV9IK9fb3Wt2diJgLvNPCmMysk3HyZWZNebVwf3i9ZZs1sl6dDwr3G5qL6q3C/WGtjGt+4X7xCsL1G1o5Ii4BBgObkypwv8uLdgFObMHrTS3cv7twSvLDG9AjIp5tYNv6iWlxXxc3sa/3SQljnbXr7khanoYTPTOrAU6+zKwpd5PGIQF8XtIPJH1G0g9I448gJUL3tGHfNxfunyNpB0nLS9pE0reb2XZy4f7nJPWVtDrwrforSlpN0lmkU3hTgTHATYVV1i3cr6smrSdp5bonI2IiMCE/3F3SNyUNyK+7paQfAtc0E3Odh8kD74H/kHSopP6SVpD0CUm/An6Tlz/KkurjFyXtkSt1P8WnHM1qlpMvM2tURLzFkivtegA/JiVaP2bJ8eO0vF5r/Q74e76/GfAYMIc0BqrJalREvA7cnx9uQ0qaprB0IlWnL+mKxgdJ1bYFLJ0s3lG4Xzc2awgwvd6UFceSEiEBZ5MSqLnAP0mnMounUZuK/X3S+1sM9AauAt4DZufXP4V0VSgRMRP4Rd60T471XeBrpM/KzGqQky8za1JEnEmaJuJuYDqwKP+8G9g3L2/LfucBnwG+TRrA/j6pivYi8JcW7OIwUgVrJikpuhw4ooH1ZpCSpcdJp/EWkZKdh4FDI+LGwrpfB25jSWWqGO8DpKscryCNv/qA9DmMBX4PtGjwft7X1aRTnn8hJYQLSVdNPkFKtorznJ0OfDe/5nxSkvq5vL6Z1SBFuC2YmZmZWVlc+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxI5+TIzMzMrkZMvMzMzsxItV3UAZmYdTuoDrA2sDPQGFgAzgFeJWFBlaGbW/Tj5MrOuSxoADAOGk453C4BFQE9SErYQaRzwAhEzK4vTzLoVn3Y0s3YhaYikkLRd1bEAIK0HHAhsCkwDpgBvA9O3gP0Gw4n5+eHAgXl9M7MO5+TLrJuRdKKksZJm5dsjkkbVW2d0TqSKt0erirnVUiI1CpgOTCVVuxqyKC+fDoyqxQRM0oj877Na1bGYWcs4+TLrfl4DvgNsA2wH3AvcKGmLeuvdDaxRuH2xzCDbLJ1qHElKqua3cKv5ef2ReXszsw7j5Musm4mImyLi9oh4MSImRMT3gPeAneqtOj8i3izcprf2tSTtJukxSfMkvSXpbEm9C8sl6WRJEyXNl/SapDMKy8+UNF7SXEmvSPqlpL6F5adJGifpYEmTJL03GG4aC8uTE6850GNbOLI3XN0brt4ajo56x75vwDaD4PQ+cHYvmCzpDknDmnhfe0haIGnVes//XNLYwuNPSnpA0hxJUySdJ2nFwvL7JZ2bt5smaaqksyT1KKzTW9Iv8mczR9LjkkbmZUOA+/Kqb+cK2Oi87POSHpQ0Q9L05t6TmZXHyZdZNyapp6SDgf7Aw/UW75KTgQmSLpI0qN62oyW90sS+1wJuB54GtgaOAg4Bziis9nPgB/m54cABwKuF5e8DXyUNmj8BOBj4Xr2XGgIcBOw7CEbNgc0OK1Tp9oR9xsLIQ+Gc8+GUxdDjBRhR3MFs6Lsn3HwZnHIZnN0jJaNjioliPfeQxosdUHi/Ag4FrsyPNwfuBG4GtgT2A7YCLq23r8OAhcAngZOAb+b3U+cy4FN535sBl+fYtsyf1ZfyesNJFcpv5McrAL8BdiC933ebeU9mVhJFRNUxmFnJcmLwCNAXmA0cFhG3FpYfDMwBXiYlNz8lXSG4bUTMz+ucAewQEbvnx0Py+ttHxBOSfkYa8L5JRCzO6xwBXECa8qEHKYH5ZkSc38K4jwdOiYgN8+PTgP8GVo+Id5E22At+fS9sPxuOA1geRu8Atz0AfwKYD1oNzlsBpr8J/9PAy6x1Pzz0afgX8KmIeKiRWH6d3+uu+fEuwAPAehHxmqQrgA8i4qjCNluRktHVI2KqpPuBPhGxU2Gdu4DJEXG0pA2AicCQiPh3YZ0bgdcj4gRJI0jVr4ERMa2Jz24FYFZT78nMyuGpJsy6p/GkKsxKwP7A5ZJGRMQ4gIi4prDuM5KeBCaTBrFfn9f5bjOvMQx4tC7xyh4iTfGwISnx60OqIjVI0v6kStCGpOpcz3wrmhwR7+b7Kw+EafNhAMA46DcPVtkRXgB4HlZ8BtYYDJPeSwkgANfA4O/Dl9+EjRfAgEUQpORw3Sbe35XANyWtFxGTSRWsByLitbx8W2BDScUqlvLPDUhjzADGsrTXgboq4zZ5m+dSYe1DfUhj9RqVE7efAJ8ABub309x7MrMSOPky64YiTSz6Yn74pKTtgW+RTg02tP7rkl4DNmqvEJpbQdKOwDXA6Tm2mcBewFn1Vv2gcL93wOJYkuQAsDjtaOg42HQ+9F0IawZE3YRfx8IP+8O04+GcHWDRWzDpP+EqUqLY8BuIeErSC8Chks4inYI8tbBKD+Bi4OwGNp/SSPywJPGr20cA2zew3tzGYstuIV1ccVx+vYXAczTxnsysHE6+zAzSH/k+jS3M0xisBbzRin0+DxwoqUeh+rULaaLTSaS8Zz6wO+nUWn07A1Mi4ieFOJqbCmJBj8JY1s1gTl+YcR98cVV4/Ei443zYaxqssDws/i3sNhyeew/WPgHOOxOeAQadDL1o2fHxSlLFaxxpjNV1hWVPAcMj4sWGNmyhp0mJ5OCIuK+Rdepm6P+wIpgvBBgKnFC3naRt8DHfrFPwgHuzbiZfQbir0qSom+exWyNIlR4k9c9X3O2U1xkBjCGdJruhsJ8zJDV6yhA4F1gTOFfSMKW5xM4E/i8i5kTEe8BvgTMkHSlpA0k7SPpa3n4CsJakwyStn58/pJm3N6NnTjAWpWA3GA4v/Qu2GQ/zHoQBN8M282ClHjB9ZZj5N9i1N8y+Dkb+Edb4NWx6bqpgLWzBx3kVaRLXnwBjImJWYdkvgB0knS9pa0kbSvp/ki5owX4BiIgJ+TVGS9o/fw7bSTpF0n55tcmk6tgoSQMl9Se1TpoGHJNf91PA+S18T2bWwfwtyKz7GUyq2AwmXQE3FvhCRNyRly8CNgf+gzR26g3SgO4Dc8JUZw3S2KUGRcQUSV8AfgX8k3Ta8I8sPcj9u6RE4Qek3otvAVfk7cdI+hXpir3lSVcO/pCU1DXmtcUpfp0Nn5oHy58PvzsG9rsKTroKGAJThsL9b8M6R8LYf8Drz8HAe2HHr8D/rQxv94Zj58G1TX2IOcbJkh4Cds2xFZeNlbQb6WKFB0iVqZcoJLAtdCTpCs9fkj6j6cA/yFNM5M/5R8DPSKc5r4iII/JYs9+RqnIvAicDf2nla5tZB/DVjmbWpYyWDnsf9ukPTx8Mz/ZJQ74+dCoc8ku4uvjcLFjuathyMQwbCNfuH3F9uVGbWXfiypeZdQmSlgdGrQ9r3gmPb5CqTIub2w5gRVh4HIx7Fj7YEzY5IJ0ivStfmGBm1q485svMap6kjYCvAbNfgt9skE4ZDqKJiwjq6QMMGg5XvpyuTuwNHC9pnY6J2My6M1e+zKxmSeoD7EEae3ZDRLycF01GupXU43ER8A4NN9fuCayaf95KxOQ8EOOG3IrnIEn/BO6PCA9WN7N24eTLzGpSnnZiH+AV4Ly6mfc/FDEZ6U/AJqQLCJYDPhiYBu8PIk0nsZA0vcR4ImYuvXk8L+nfwJ6kqwZviIg3O/ZdmVl34OTLzGqKpOWAz5ASqlsiYnyjK6eE6jGkp4F1gAHPph6KdVdfvkoT47oi4n1J1wJbAP8h6RHg7/Vm7TczaxVf7WhmNUPSGqQG1W+TEq85bdjHaRFxWhu2WwnYmzQe7IaIeKe1+zAzA1e+zKwGSOpBmktrB+AO4Jko+ZtjRLwr6Q85hqNyU+zHy47DzGqfky8z69Rya6N9gXnABfVmkS9VTrQek/RijmmopJsKjb3NzJrlqSbMrFNSsiPwVdIYrSurTLyK8inHS4GXgWMlbSlJzWxmZga48mVmnZCkAaQrGXsCF0fE9IpD+og86P5BSRNJ49CGSRoTEe9XHJqZdXKufJlZp5GrXVsDxwITgcs6Y+JVlKefuJDUyPpreX4wM7NGufJlZp2CpP6kObVWAi6PiLcqDqnF8gSsd0uaAOwjaShwe0TMqzg0M+uEXPkys8pJ2hQ4HpgKXFRLiVdRRPwbOB9YQKqCrV9xSGbWCbnyZWaVyc2wvwCsBVwTEa9VHNIyy824b5U0nlQFe4HUpPuDikMzs07ClS8zq4SkDUnNsOcC53eFxKsoIl4EzgP6kpp0r11xSGbWSbjyZWalktSb1Ax7I+DGiHip4pA6TETMBa7Pp1UPVmpzdH9ENNTk28y6CVe+zKw0ktYlje1ajtQMu8smXkUR8RxpLNjqpCbdq1cckplVyJUvM+twuRn2CGArUk/GF6qNqHwRMVvS1aTP4HBJDwMPu0m3Wffj5MvMOpSkwaRWPNNJ1a5uOwlpbk/0tKSXSZPIbizpxs4+l5mZtS8nX2bWIXIz7F2ATwB3AmPdhDqJiJmSLid9NkdLug94wp+PWffg5MvM2p2kVUnVrgXAhW48/VE50Xq0gSbdnaJ/pZl1HA+4N7N2k9sD7QAcBYwF/uDEq2kRMQ24BPg3cJykLdyk26xrc+XLzNqFpJWAvYHewCUR8U7FIdWMPOj+gdyku64KdktEzKk4NDPrAK58mdkyydWurYDjgJeBS514tU1EvA5cAMwktSfapOKQzKwDuPJlZm0maQVSM+yVgSsi4s2KQ6p5uUn3nYX2REOBv0bE/IpDM7N24sqXmbWJpGGk9kDTSM2wnXi1o4iYTJqYdTGpCvbxikMys3biypeZtYqkvqRm2OsA10bEqxWH1GXlatcYSRsB+0l6FrjHTbrNapsrX2bWYpLWJ1W7FpCaYTvxKkFETCQ16e5PuiJyrYpDMrNl4MqXmTVLUi/gc8BQ4KaImFRxSN1OvvLxOkmbAYdKegL4m5t0m9UeV77MrEmS1iY1w+5Lag/kxKtCETGONBZsTdLs+IMqDsnMWsmVLzNrkKSepGbYWwO3RcRz1UZkdSLiPUl/BLYBjpD0EPCom3Sb1QYnX2b2EZJWJ032+S5pbNfsikOyenJ7oiclvURq0r1JbtI9o+LQzKwZPu1oZh+S1EPSLsDhwKPANU68OrecbF0OjAeOkbSt2xOZdW6ufJkZAJJWIVW7FpKaYc+sOCRroXy68eF6Tbpvjoj3Kg7NzBrgypdZN5fbA20PHA08S5qp3olXDYqIqcDFwOvA8fnKSDPrZFz5MuvGJK1Iaobdl9STcVrFIdkyylNP3CdpArBv7kRwq5t0m3UernyZdUO52rUFqRn2ZOASJ15dS0RMITXpnkVqT7RRxSGZWebKl1k3k5thjwJWA66MiDcqDsk6SG5DdEehSfck4E436TarlitfZt2IpE1IE6bOIA2qd+LVDUTEK6T2RD1IY8HWqzYis+7NlS+zbkBSH+DzwBDguoiYXG1EVrZc7bopJ+D7SxpHatK9sOLQzLodV77MujhJHyc1w15MmjDViVc3FhHjSVWwlUhNutesOCSzbseVL7MuKjfD/iwwDBgTERMrDsk6iYiYI+nPwGbAYZIeBx50k26zcrjyZdYFSVqLdCVjP1IzbCdetpRIniFdEbk2cJSkgRWHZdYtuPJl1oXkZti7AduRmmE/W3FI1slFxCxJVwHbAkdK+hvwWO4daWYdwMmXWRchaRCptcx7pLFdbi1jLZITrScKTbqH5ibd7nRg1gF82tGsxuVm2DsDRwCPA1c78bK2iIjpwGhgInCspG3cpNus/bnyZVbDJK1MqnYFcFFEzKg4JKtxuUn33xto0j274tDMugxXvsxqUG4PtB1wDPA8MNqJl7WniHgLuAh4kzQx6/CKQzLrMlz5Mqsxkj5GaobdD7gsIt6uOCTrovLUE/cWmnQPJV3IMbfi0MxqmitfZjVE0mak9kCvkZphO/GyDhcRrwHnA3NITbo3rDgks5rmypdZDZDUj9QMexBwVUS8XnFI1s3kJt235ybde0uaSGrSvaDi0MxqjitfZp2cpI1J7YFmkZphO/GyykTES6T2RMuRxoKtW3FIZjXHlS+zTio3wx4JrA/8JSJeqTYisyQi5gE35jFgB0r6F3Cfm3SbtYwrX2adkKQhpGoXpPZAr1QWjFkjIuIFUhVsFdK8YIMrDsmsJrjyZdaJSFoO2J3U8HhMREyoOCSzJkXE+5L+BGwBfEXSY8BDeb4wM2uAky+zTkLSmqRJLaeSql1zKg7JrEVye6J/SXqFNA3Kxrk90bRqIzPrnJx8mVUsN8PeFdgeuD0ixlUcklmbRMS7kv5A+r/8VUkPAP9wk26zpTn5MquQpIGkatcc3AzbuoCcaP1D0iSWbtL9bsWhmXUaHnBvVoHcHmgn4EjgKdLcXU68rMuIiHeAy4BJwHGStnKTbrPElS+zkkkaQKoI9AAujojpFYdk1iHyoPuH6jXpHhMR71ccmlmlXPkyK0mudm0DHAtMIDXDduJlXV5EvElq0j2N1J5oWMUhmVXKlS+zEuRm2HsCHyMlXVMrDsmsVHkC1rtze6K6Jt235wlbzboVV77MOpik4aRm2G+STjM68bJuKyJeJTXpnk+qgm1QcUhmpXPly6yDSFqe1Ax7MPDHiJhScUhmnUJuxn1boUn3eOAuN+m27sKVL7MOIGlDUnug2cAFTrzMPioiJpHaE/UhNelep+KQzErhypdZO5LUG9gD2BC4ISJerjgks04tIuYC1+dB+AdJehp4wE26rStz5cusnUhal1Tt6kmaMNWJl1kLRcTzpLFgA4FjJK1ecUhmHcaVL7NllJthf5rUWPiWiBhfcUhmNSkiZku6FtgSOFzSI8Df3aTbuhonX2bLQNIapMkj3yFVuzx5pNkyyO2J/inpZdJkxHVNut+pODSzduPky6wNJPUAdgE+AdwBPOPmwWbtJzfpvgLYAThK0n3AE/49s67AyZdZK0lajVTtmke6knFWxSGZdUk50XosN+mua090k3/nrNZ5wL1ZC+X2QJ8Avgr8E7jSfwTMOl5ETAMuASaTmnRv6SbdVstc+TJrgdwMe2/S74ybYZuVLA+6/5ukiSypgt3icZZWi1z5MmtCrnZtTWqGPQm4zImXWXUi4g3gQmA6qT3R0IpDMms1V77MGiGpP6kZ9gDg8oh4q+KQzIwPm3TfVa9J91/dpNtqhStfZg2QtCmpGfZU4EInXmadT0T8m9SeaCGpCvbxikMyaxFXvswKJPUFvgisBVwbEa9WHJKZNSE3474l91PdV9LzwN0R8UHFoZk1ypUvs0zSBqT2QHVTSDjxMqsREfEiqQrWj9Ske+2KQzJrlCtf1u3lZtifAzYGboqIlyoOyczaIDfp/ouk4cAhkp4kNeleVHFoZktx5cu6NUnrkMZ29QbOc+JlVvsi4llSFWwwqUn3oIpDMluKK19W+6Q+wNrAyqQkagEwA3iVNB6kgU20HDAC2Aq4NSKeLydYMytDbtJ9NbA1cISkvwOPNNmkuw3HErO2cPJltStNfDoMGE76v7wAWAT0JB04FyKNA14gYuaSzTSYNEnjDFK1y5M0mnVBuT3RU7lJ997AJrlJ99Jz9bXxWGLWVj7taC0iaYikkLRd1bEAIK0HHAhsCkwDpgBvA9O3gP0Gw4n5+eHAgUjrSeohaVfgK8DDpKsZnXiZlaiKY0lEzAAuB54Hjpa03YftidpwLCkrbuu6nHx1MZJOlDRW0qx8e0TSqAbW21jS9ZJmSpoj6SlJw6qIudXSwW8UaYbrqaRvqA1ZlJdPfxMO3B1OAT5OmrfrX/lbcacmaUT+Q7Va1bFY99LSY0lh/Qvy/9VTyoyzpSJ5BLgM2AY47O40ML9VxxJgVC0mYD6WdC5Ovrqe14DvkA4u2wH3AjdK2qJuhTwR4d+Bl4HPAJsB3wdmlx5ta6XTAyNJB8L5za2+CLgO1r0YtvgRDJgJN0fEux0dplkX0OyxpI6k/YEdgNdLjbANIuJt4JLtYeaLcPo90HdRC44l2XzSsWdkPhaZtYmTry4mIm6KiNsj4sWImBAR3wPeA3YqrPYz4M6IODkinoqIlyLittbOayVpN0mPSZon6S1JZ+dpG+qWS9LJkiZKmi/pNUlnFJafKWm8pLmSXpH0yzzJad3y0ySNk3SwpEmS3hsMN42F5ckHyznQY1s4sjdc3Ruu3hqOjvz/ejL0+1/49B9gt9/B9p+Fb6wGkyXd0VSVT9IekhZIWrXe8z+XNLbw+JOSHsiVwymSzpO0YmH5/ZLOzdtNkzRV0lmSehTW6S3pF/mzmSPpcUkj87IhwH151bfzt9bRednnJT0oaYak6c29J7PWauGxBKUq0G+BQ4E2TWxa9rEEmPlv+PbHYezfYaPfw86ToW9jx5I634BtBsHpfeDsXj6W2DJw8tWFSeop6WCgP2mME/mXdU/gOUl/lfR2/iU9qN62oyW90sS+1wJuB54mXU10FHAIcEZhtZ8DP8jPDQcOAIoJ3vvAV0kDXU8ADga+V++lhgAHAfsOglFzYLPD0gz0kN7IPmNh5KFwzvlwymLo8QKMCOh7EYxcC95aFZ7fE264DE65DM7ukf6AjCke3Ou5hzTG44DC+xXpj8uV+fHmwJ3AzcCWwH6kKycvrbevw0itTz4JnAR8M7+fOpcBn8r73ow0LmWMpC3zZ/WlvN5wYA3gG/nxCsBvSNWGEcC7zbwnszZr6FiSn18OuBr4aWNXDHfmY8kpsPN/wR39Yc6X4Ntj4Qv1jyXFHcyGvnvCzT6W2DKLCN+62A3YnHQKcSEwExhVWDYYCNLB6r9Iv+T/ldctrncGcE/h8ZC83Xb58c+AiUCPwjpHkCpS/UgH6XnA8a2I+3jgxcLj0/I+VooIAjbYE25aAV4P2DNgz77wzm7wh7rHk+GA/jB9IEz5F3yl7vnC7fj70oFpEbBLE7H8Gniw8HiXvM3a+fEVwCX1ttkqf0aD8uP7SZe2F9e5C7g4398AWAysW2+dG4Fz8/0ReZ+rNfPZrdDce/Jtyf+rqmOolVtTx5K8/GekU/l1j18BTqm3Tq0cS2ZuDY+fAz+ZAfvOg736w5TV4ZkGjiM+lvi2TDdPNdE1jSf98q4E7A9cLmlERIxjSbXzpoj4db7/T6Urj04CbgWIiO828xrDgEdj6TlzHiJdlr0h0BfoQ/rm1yClcSLfzOv3J13W3bPeapNjyRitlQfCtPkwAGAc9JsHq+wILwDMguV+B/sPhH+/Dz23SH8suAYGfx++/CZsvAAGLEoHoB7Auk28vyuBb0paLyImk751PhARr+Xl2wIb1qsYKv/cgDQuBGAsS3sdqJvwcZu8zXOSiuv0IY2vaZRSK6SfAJ8ABub309x7MmutRo8lkkaQkqStmtpBDR1LVhoBY4Bev4M9vgu3rwHjZ6XfL8DHEms/Tr66oEiTAb6YHz4paXvgW6Ry/jTSt9jn6m32PKlU3y4hNLeCpB2Ba4DTc2wzgb2As+qtWhxD0jtgcSw5MH3EB7D8IlhrMcSp6dQF58GofjBnBDw3BBa/DDNvS6X9AyVt3ESY7wCXSHoYOBy4S9Iq+1QpAAAT0ElEQVRpedkg4F/Aow1sNyqPtRgC9CtsA+m0Qt1zw/Nzo0nfWpd6K3mdIfnxqZLmFJafCMwiXTgxK29/Ygvek8GIev8m1jLzSf/XrpB0M6mSsibwVuEPvoBfSvohqeLTkLqB6sdK+n+k5GtBvX+TVfLP44Fe+f7XJS09P1eyNumU4/2kU3fzgE2APQr7HAGsXPf4cFhnKmwV0OtUOOT9/BoT4VPLwVuLoNfVS34/P3Qs/LA/TDseztkBFr0Fk/4TriIlig2KiKckvQAcKuks0inIUwur9AAuBs5uYPMphfv1x9PVJX51+whg+wbWm9tYbNktpIsrjsuvV/f3wacdO5CTr+6hB+kbEBGxQNLjpINT0cbA5Fbs83nSH/oehW+su5AmJ5xE+tY5H9iddEqhvp2BKRHxk7on1Pzl2wt6FMYpbgZz+sL0x9J7GbsiLDwTLr4UduoLi3aFR1aDd34Fh5wIvzgTngEGnQzzbkvf4m+MiNGNvZikBaTxE4+T/qh8JSJm5WUbAWtFxH82sf0IYFxEnFZ4bgip7H9aTpL2B/4SEfc1so9PkhK//42It/JzqwI/Ag6s207SNsDXm3tPlgZfF/9NrOUk7Qa8nv//DmJJ5aXOHaQxYBdFxPhG9jGENN7owoh4QlIv0jxbP647lkg6gnQsOYV0LPkK8FxEnN/A/k4GXo2IzxSe+x2wR92/c066Vvvw313a7mj4GrDpL1O8XAqfeQ+m/BL+DCkDOQlO7JcmY+Yf8LH3YO0T4LzCsaQXLfs7eiWp4jWOdFrvusKyp4DhkRqDt9XTpGPU4MaOJaTPEwoVwXwsGQqcUO9Y4tygg/kD7mIknUk6dfgq8DFS8jCCNJdNnV8Cf5L0IKkk/WlS1Wufwn7OAHaIiN0bealzSWX+cyX9FlgfOBP4v4iYk/fxW+AMSfOBvwGrAttGxHnABGAtSYcBj5Cmjzikmbc3o2e9/7M7wZiHYP+vwpTdYPJv4IvzYIV+MPlp2LQfzOoNs66DkVvAtDdhk3Phs6Rja3OuAn5KKsmPqUu8sl8Aj0o6H7iANPB2KLBnRBzXgn0TERMkXQWMzn9AniJ94x8BvBQR15MS4iBV08aQvsXOIFUwj5H0KrAW8KsWviezFmnuWBIRU1lySqxumw+AN4uJVy0fS+bCKnXJ15Yw28cSazdVDzrzrX1vpFNYk1kyH83dwMgG1juCdNCaSxpLcEgD+3ml8HgIhUGy+bndgMfya71FKpv3KSzvAfw38BLpW9erwM8Ky88gzSQ9G7ie9E00CstPI1WOUlzQ52i4tCfMrRv0Ogv23gpu7AWze8HsLWDMZnDb6vDMHNj7Evj+3nD7ijClByxYFaasmP54zAaOaMHn+bf8vvdqYNl2wF9Jp2LeJ30b/nFh+f2kPyD1P9dbCo975fdZ9xm9STp1sm1hnR8Ab5BOLY7Oz32G9C16Xv45sqXvqbvf8ID7ln5OLTqW1NvmFT464L7mjyV165wK31sRJvtY4tuy3pQ/fLPaIO1EagMytblV6zwGq90OOw6HD3aGa9eMeLDjArTOzqcdDWjTsaRgEDCOiMfaNyjrLjzPl9Wa50ljFvq0dINPwLST4b4+ELvALpLW77jwzKxGtPpYkvXJ2zU4ps2sJZx8WW2JmEka1DuIlh80+3wMVtkLzno5DabdR9IX80BfM+uO2ngsyevfkbc3axMnX1Z70lw5t5IGlA7io/P51OmZl68C3ErE5EhXFJ1HmjvoeElrlxCxmXVGy3AsKSdA66o85stqV2psuwlpFu7lSPPbLCIdKHuRrth5Bhjf0LdUSZuSWhU9RZr0cFFJkVuFPObLPmIZjyVmreXky2pf6kG2Dmnyxj6kK6ZmAq+SJpxtYlP1J/W6XAm4IfJcWtZ1OfmyRi3DscSsNTzPl9W+dFCc1LZNY7aka0gtUg7Ps9k/HEu3OjGz7mAZjiVmreExX9btRfI0cCGpN9wRklZpZjMzM7M2cfJllkUay3E5qa/Z0ZK2U70utWZmZsvKyZdZQa6CPQpcCmwNHCZpxYrDMjOzLsTJl1kDImIacAmpjclxkrZwFczMzNqDky+zRkTE4oh4ALgS2BU4QFK/isMyM7Ma5+TLrBkR8QZwAemS869J2qTikMzMrIZ5qgmzFoiIhcCdksaT2hMNBf4aEfMrDs3MzGqMK19mrRCprcj5wGJSFWxIpQGZmVnNceXLrJVytWuMpI2A/SQ9B9wTER9UHJqZmdUAV77M2igiJpKadPcnXRG5VsUhmZlZDXDyZbYMImJuRFwH3AccKunTknpWHZeZmXVeTr7M2kFEPEsaC7YmaXb8QRWHZGZmnZSTL7N2EhHvAX8EniD1h/ykJP+OmZnZUvyHwawd5fZETwIXAZsAh0taueKwzMysE3HyZdYBImIGMBp4AThG0rZuT2RmZuDky6zD5CrYI8BlwHakAfkfqzgsMzOrmJMvsw4WEW8DFwNTgOMlbVZxSGZmViFPsmpWgohYBNwvaSKwr6RhwK0RMafi0MzMrGSufJmVKCKmkJp0zyK1J9qo4pDMzKxkrnyZlSy3Ibqj0KR7EnCnm3SbmXUPrnyZVSQiXiG1JxJpLNh61UZkZmZlcPJlVqGImB8RNwO3A/tLGinJFWkzsy7MyZdZJxARE0hVsJVITbrXrDgkMzPrIE6+zDqJfOXjn4G/AYdJGuEm3WZmXY+TL7NOJE/M+gzpisi1gaMkDaw4LDMza0dOvsw6oYiYBVwFPAUcKWlHtycyM+sanHyZdVK5CvYEaXb8TUlNugdUHJaZmS0jJ19mnVxETCc16Z4IHCtpG1fBzMxql5MvsxoQEYsj4u/A5cAOwCGS+lcclpmZtYGTL7MaEhFvARcBb5ImZt204pDMzKyVPJmjWY3JTbrvlTSBJU26b4uIuRWHZmZmLeDKl1mNiojXgPOBOaQm3RtWHJKZmbWAK19mNSw36b5d0gukJt0TSU26F1QcmpmZNcKVL7MuICJeJrUnWo40FmzdikMyM7NGOPky6yIiYl5E3AjcCRwo6XNu0m1m1vk4+TLrYiLiBVIVbBXSvGCDKw7JzMwKnHyZdUER8T7wJ+Ah4CuSdpPk33czs07ApyTMuqiICGCspMnA3sDGkm6MiGkVh2Zm1q35m7BZFxcR7wJ/AMYCX5X0CbcnMjOrjpMvs24gN+n+B3AJsBnwH5JWqjgsM7NuycmXWTcSEe8AlwGTgOMkbeUqmJlZuTzmy6ybiYjFwEOSXgT2BYZKGpMH6ZuZWQdz5cusm4qIN0lNut8mtScaVnFIZmbdgitfZt1YRCwE7ik06R4K3B4R8yoOzcysy3Lly8yIiFdJTbrnk6pgG1QckplZl+XKl5kBkJtx3yZpPLBXrobd5SbdZmbty5UvM1tKREwitSfqTWrSvU7FIZmZdSlOvszsI3KT7huAu4CDJO3uJt1mZu3DyZeZNSoinieNBRsIHCNp9YpDMjOreU6+zKxJETEbuBZ4BDhc0q5u0m1m1nY+gJpZs3J7on8CFwDrA0dKWrXisMzMapKTLzNrsdyk+wpgHHCUpB3cnsjMrHWcfJlZq+Qq2GPApcCWwJclrVhxWGZmNcPJl5m1SURMAy4BJpOadG/pKpiZWfOcfJlZm0XE4oj4G3AlsDNwoKQVKg7LzKxTc/JlZsssIt4ALgSmk9oTDa04JDOzTsuTJppZu8hNuu/K7YnqmnT/1U26zcyW5sqXmbWriPg3qT3RQlIV7OMVh2Rm1qm48mVm7S43475F0oakKtjzwN0R8UHFoZmZVc6VLzPrMBHxIqkK1o/UpHvtikMyM6ucky8z61ARMTci/gLcCxwi6TOSelYdl5lZVZx8mVkpIuJZUhVsMKlJ96CKQzIzq4STLzMrTW7SfTXwGHCEpJ3dpNvMuhsf9MysVLk90dOkecE2IiVhq1QclplZaZx8mVklImImcDnwHHC0pO3cnsjMugNPNWFmlYmIAB6VNAnYFxgq6eaImFVxaGZmHcaVLzOrXES8TWrS/RqpSffmroKZWVfl5MvMOoWIWBQR9wNXAbsBB0jqV21UZmbtz8mXmXUqEfE6cAEwk9SeaOOKQzIza1ce82VmnU5u0n1nbtK9T27SfUdEzK84NDOzZebKl5l1WhExGTg/P/yapCHVRWNm1j6cfJlZpxYR8yPiZuA24EuSPi+pV9VxmZm1lZMvM6sJETGB1J6oP+mKyDUrDsnMrE2cfJlZzYiIORFxHXA/cJikT7tJt5nVGidfZlZzImIcaSzYmqTZ8Qc2uYHUB2kDpO0Oh3WQtsuPe5cRr5lZka92NLOaFBHvSfojsA1wpKSHgEcjYvGHK0kDgGHAcNLxbsFwWAXYAugNLEQaB7xAandkZtbhXPkys5ojKSTtn5t0PwlcBGwCHC5p5bzSesCBwKbANGAK8PbbMBd4Oz+eRkrMDszrt3ecp0h6pb33a2a1zcmXmXUYSSdKGitpVr49ImlUYXk0cjunNa8TETNITbrHA8ecKI1aDKOA6cBUYFEjmy7Ky6cDozoiATMzq8/Jl5l1pNeA75BODW4H3AvcKGmLvHyNerc98/N/au0LRcTiiHj4G3D99nDwubDxlJYf4+aTkrCR+VSlmVmHcfJlZh0mIm6KiNsj4sWImBAR3wPeA3bKy98s3oC9gQkR8UBrXic34r5b0txz4KnfwPoL4N3z4PO3w7oAB8FnBsDv/xcO6gtXbALfrNt+D9h7APyqF/y+D0yQdLEKSZikIyTNlrS7pHGS3pd0n6SP14vjVElv5nWvIE2LUVy+vaQ7JU3LlcCHJO3Uyo/VzGqcky8zK4WknpIOJiUkDzewvD9wMGn8VvH50yRFE/tdAbgDmL027PxzOP8lGPJ7+Oy+8MA/YPMRcMJ1cOIOcPchcNtP4fR1YHLdPnpAHAcX/xm+/lMYLfgE8Pt6L9UH+C7wVVLyOIAls+8j6UDgp8CPSJW+8cB/1dvHx4A/ALsCOwD/BG6TtGpj78/Muh5FNHpMMzNbZpI2Bx4B+gKzgcMi4tYG1juWlPCsHRFvF54/CTgpIoYWngvggIi4TtIxwFnA2gGDgM99F1Y+E35+FRy3F0wdCJdvAK/8Hs65HUb8Eq5uIuS1dof596bEavmIWCzpCOAyYGhEjM8xHAZcCvSNiJD0MPBsRBxTiPNuYMOIGNLIZyPgdeDbEXFlMx+lmXURrnyZWUcbD2xFqiadB1wuabMG1jsGuKmYeAFExP8VE68GDAPGRsR7wMrAglPgBWDx32Cdp6H/PFhxJ7jnfth+NqxWfwffgS3WhB/3g8t6w2/uh3NJU1EMLqw2vy7xyl7P66xciOORerte6rGkQZIukDRB0rukU7CDyKdGzax78DxfZtahImIB8GJ++KSk7YFvAUfVrSNpK9KA/P9ZxpfrTeHKRsGHpf0VYca34MFrYcviBjfAwLPgh5vDnV+DqzaDnpdCn1vg9Ly/Ogvrv7X8szVfYi8HVie9/1dIA/3vqfc6ZtbFufJlZmXrQRo/VXQs8DJwdxv29zywuaSPAQuAnmfBUKDHrvDarvDu8vDOg7DlAFh4HDxZ3PgW2GgxLPcwXPwDGL8vTHs9jedqSxw71nuu/uNdgN9HxK0R8Syp8rVGG17LzGqYky8z6zCSzpS0q6Qh+YrEM4ARwFWFdfoBhwGXRAODUCWdJOmFJl7mKmAOcMXRMPBs2PQcOHEIPHwovAGwO/zpSdhrJOx9Baz5W/j452AfgK3T6cMee8Nef4bVvwE7jU0D/1vrt6RJXo+RtJGk75JOtRZNAL4sadNcAbyGlDCaWTfi045m1pEGA1fmn+8CY4EvRMQdhXUOAlYgDWhvyGqk2esbFBFzJI0EfnMJXN8bFq8Pj/0ZLqhbZwzc/iVYeDfscxcc3htmfxyeADgJXrkVLrwf9r8XvrwOvNQHTl4If2zNG42IayWtD/wM6AfcDPwaOKKw2leBC0nVt9eB04Cm+1KaWZfjqx3NrGtJ82ZtSpo0tbUGAeOIeKx9gzIzW8KnHc2sq3ke6MlHx5U1p0/ebnxzK5qZLQsnX2bWtUTMJE26OoiWJ2B98vp35O3NzDqMTzuaWdeUmmSPJE098Q4NN9fuCayaf95BxOQG1jEza1dOvsys60r9GTcBNiddYPQBKQnrCfQizd31DDDeFS8zK4uTLzPr+qTewDqk+bv6kCY3nQm8SpoE1sysNE6+zMzMzErkAfdmZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJXLyZWZmZlYiJ19mZmZmJfr//Fnu+l05WhIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAFBCAYAAABJi8prAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XecVeW1//HPF6SIxA5ixwqIvZtYMMnVJMaWGGty7T25KXrNLzdNrzGaxBtTFTv2EmPD3nsvCaI0CygoKtJEmsD6/fE8I4dxOjN7z5n5vl+v82LO2eWsfYA966z97GcpIjAzMzOzYnQpOwAzMzOzzsTJl5mZmVmBnHyZmZmZFcjJl5mZmVmBnHyZmZmZFcjJl5mZmVmBnHyZWauTtJGkv0kaJWmWpI8ljZZ0saQd8zrjJYWk8W0cS39Jp+fHkLZ8r6WVY438eKTseMysbSxTdgBm1rFIOhK4AOhRa9GA/OgD7FdgSP2BX1c8f6TA9zYz+xxXvsys1Uj6MnAJKfEK4DfA2vn5xsD/ANNKC7CNSOpZdgxmVj2cfJlZazqbxeeVv0TELyNiYkTMj4hxEXE2cGx9G+dLgzWX3YY04fVvSXpc0oeS5kmaLOkxSafk5cOAhyve4tcV+zm9Yj9fk3SvpKmS5udLon+VtGqt+D67VCppF0lPSZoDDK1Y59Acw4wc01hJZ0nqVWtfy0u6RNI0STMlXQes1sTP2cyqmC87mlmrkNQX2L7ipT/UtV5ELGil99sB+AdLfolcLT96Af/XxP2cApxb6+V1ge8De0naMSI+qLW8D3AfsETFS9Jf83aVNiJV/PaQtGtEzJEk4FZg94r1DgZ2a0rMZlbdXPkys9bSv+LnmRExqY3fb2cWn8N2AroDawF7A9cBRMQRLJngnBERyo/TJa1NqtYB3ENKunqSEiGA9YBf1PHevYDHgPWB3sBZ+UaCmsRrGNAvr/ff+bVtgRPzz3tUxPU6MBBYHRjX5KM3s6rlypeZVau3Kn7+f8ATwGjg2Yi4o4n7+BrQreLnCXWss0c92x4ZEe/mn8dJOqJi2RH5Ude+/gh8ueK1v0bEGABJ/ws80JTAzax6ufJlZq1lfMXPy0taoxX3XdcXxVuA84F5wL6ky5zDgfck/a2J++3bhHVWqeO1DyoSr5bsq3KfE+v52cw6KCdfZtYq8rio5ype+u+61pPUUMV9XsXPleOp1q/j/SIiTgZWAnYAvgvcDXQFTpa0U82qDbxf5ViuX1RckvzsQd1J1ZxG9vXdevZVMyZuSsW6a9Xzs5l1UE6+zKw1/RxYlH/+r3yX4hqSuuWJV/8HuLiB7Ssv++0Nnw2s37/2ipJ2k/RT0hQWY4F/Ak9XrLJO/vOjitcGSaqcf+xeoOYGgFPyXY+98p2Iu0kaCvy0wSNerPJS55mSviSph6SV836vBQ7LyyvvwPy+pAGS+gG/bOJ7mVkVU0RDXwrNzJpH0jGky4Hd6lnltojYL89svy4wISL6521XJI3lWjGvO4s0oH02afA6wO4R8Yik7wJX1fMenwADImJSnoPrHWDVWuvU7Oc04HcNHNIZEXF6ju9zMVeSdD6LB9XX5ciIGJbvdnwIGFJr+ZSKOB+NiNrLzawDcOXLzFpVRFwCbE5KwMaSLtF9AowBLgXOaWDb6cA3SJcv55CqVr8Ezqtj9ReBy4FRwAxgISl5uQMYUnO3ZUTMBQ7K68+u4z1/n9/z7vx+C4DJwFOkmfGvaMaxn0S6/PlojulT0jiuh4HT8nsQ6VvvfsBleb1ZwM3UUeEzs47HlS8zMzOzArnyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBXLyZWZmZlYgJ19mZmZmBVqm7ADMzNqc1ANYC1gJ6A7MB6YB7xAxv8zQzKzzcfJlZh2XtCIwCBhMOt/NBxYCXUlJ2AKkkcBoIqaXFqeZdSq+7GhmrUJSf0khaduyYwFAWhc4ENgEmAJMAj4Epm4O3+oHJ+fXBwMH5vXNzNqcky+zTkbSyZJGSJqZH09L2qvWOsNyIlX5eKasmJstJVJ7AVOBD0jVrroszMunAntVYwImaUj++1m17FjMrGmcfJl1PhOBnwJbA9sCDwG3Stq81noPAKtXPL5RZJAtli417klKquY1cat5ef098/ZmZm3GyZdZJxMRt0XE3RHxekSMjYifAx8DO9VadV5ETK54TG3ue0naVdKzkuZKel/SeZK6VyyXpFMkjZM0T9JESWdXLD9H0hhJcySNl/R7ST0rlp8uaaSkgyW9IenjfnDbCFiWnHjNhi7bwJHd4brucN1WcEzUOvf9ELbuC2f0gPO6wQRJ90oa1MBx7SFpvqRVar3+W0kjKp5/UdKjkmZLmiTpAknLVyx/RNL5ebspkj6QdK6kLhXrdJf0u/zZzJb0vKQ987L+wMN51Q9zBWxYXvY1SY9LmiZpamPHZGbFcfJl1olJ6irpYKA38FStxTvnZGCspIsl9a217TBJ4xvY95rA3cDLwFbA0cAhwNkVq/0W+GV+bTDwHeCdiuWfAEeRBs2fBBwM/LzWW/UHDgL27wt7zYZND6uo0u0N+42APQ+Fvw+FUxdBl9EwpHIHs6Dn3nD75XDq5XBel5SMDq9MFGt5kDRe7DsVxyvgUODq/Hwz4D7gdmAL4FvAlsBltfZ1GLAA+CLwfeBH+XhqXA7slve9KXBFjm2L/Fl9O683mFSh/GF+vhzwJ2B70vHOaOSYzKwgioiyYzCzguXE4GmgJzALOCwi7qxYfjAwG3iLlNz8hnSH4DYRMS+vczawfUR8JT/vn9ffLiJekHQWacD7gIhYlNc5AriQNOVDF1IC86OIGNrEuE8ATo2IDfPz04H/B6wWETOQNtgH/vgQbDcLjgdYFoZtD3c9CjcCzAOtChcsB1Mnw//U8TZrPgJP7A7/BnaLiCfqieWP+Vh3yc93Bh4F1o2IiZKuBD6NiKMrttmSlIyuFhEfSHoE6BERO1Wscz8wISKOkbQBMA7oHxFvV6xzK/BuRJwkaQip+tUnIqY08NktB8xs6JjMrBieasKscxpDqsKsABwAXCFpSESMBIiI6yvWfUXSi8AE0iD2m/M6P2vkPQYBz9QkXtkTpCkeNiQlfj1IVaQ6STqAVAnakFSd65oflSZExIz880p9YMo8WBFgJPSaCyvvCKMBRsHyr8Dq/eCNj1MCCMD10O8X8N3JsPF8WHEhBCk5XKeB47sa+JGkdSNiAqmC9WhETMzLtwE2lFRZxVL+cwPSGDOAESzpXaCmyrh13ua1VFj7TA/SWL165cTtTGAHoE8+nsaOycwK4OTLrBOKNLHo6/npi5K2A35MujRY1/rvSpoIbNRaITS2gqQdgeuBM3Js04F9gHNrrfppxc/dAxbF4iQHgEVpRwNHwibzoOcCWCMgaib8Og5+1RumnAB/3x4Wvg9v/BdcQ0oU6z6AiJckjQYOlXQu6RLkaRWrdAEuAc6rY/NJ9cQPixO/mn0EsF0d682pL7bsDtLNFcfn91sAvEYDx2RmxXDyZWaQfsn3qG9hnsZgTeC9ZuxzFHCgpC4V1a+dSROdvkHKe+YBXyFdWqvtS8CkiDizIo7GpoKY36ViLOumMLsnTHsYvrEKPH8k3DsU9pkCyy0Li/4Muw6G1z6GtU6CC86BV4C+p0A3mnZ+vJpU8RpJGmN1U8Wyl4DBEfF6XRs20cukRLJfRDxczzo1M/R/VhHMNwIMBE6q2U7S1vicb9YueMC9WSeT7yDcRWlS1M3y2K0hpEoPknrnO+52yusMAYaTLpPdUrGfsyXVe8kQOB9YAzhf0iClucTOAf4WEbMj4mPgz8DZko6UtIGk7SWdmLcfC6wp6TBJ6+fXD2nk8KZ1zQnGwhTsBoPhzX/D1mNg7uOw4u2w9VxYoQtMXQmmPwa7dIdZN8Ge18Lqf4RNzk8VrAVN+DivIU3ieiYwPCJmViz7HbC9pKGStpK0oaRvSrqwCfsFICLG5vcYJumA/DlsK+lUSd/Kq00gVcf2ktRHUm9S66QpwLH5fXcDhjbxmMysjflbkFnn049UselHugNuBPD1iLg3L18IbAb8J2ns1HukAd0H5oSpxuqksUt1iohJkr4O/AH4F+my4bUsOcj9Z6RE4Zek3ovvA1fm7YdL+gPpjr1lSXcO/oqU1NVn4qIUv86D3ebCskPhL8fCt66B718D9IdJA+GRD2HtI2HEc/Dua9DnIdjxe/C3leDD7nDcXLihoQ8xxzhB0hPALjm2ymUjJO1KulnhUVJl6k0qEtgmOpJ0h+fvSZ/RVOA58hQT+XP+NXAW6TLnlRFxRB5r9hdSVe514BTgn818bzNrA77b0cw6lGHSYZ/Afr3h5YPh1R5pyNdnToNDfg/XVb42E5a5DrZYBIP6wA0HRNxcbNRm1pm48mVmHYKkZYG91oc17oPnN0hVpkWNbQewPCw4Hka+Cp/uDQO+ky6R3p9vTDAza1Ue82VmVU/SRsCJwKw34U8bpEuGfWngJoJaegB9B8PVb6W7E7sDJ0hau20iNrPOzJUvM6taknoAe5DGnt0SEW/lRROQ7iT1eFwIfETdzbW7AqvkP+8kYkIeiHFLbsVzkKR/AY9EhAerm1mrcPJlZlUpTzuxHzAeuKBm5v3PRExAuhEYQLqBYBng0z5p8H5f0nQSC0jTS4whYvqSm8coSW8De5PuGrwlIia37VGZWWfg5MvMqoqkZYAvkxKqOyJiTL0rp4TqWaSXgbWBFV9NPRRr7r58hwbGdUXEJ5JuADYH/lPS08CTtWbtNzNrFt/taGZVQ9LqpAbVH5ISr9kt2MfpEXF6C7ZbAdiXNB7sloj4qLn7MDMDV77MrApI6kKaS2t74F7glSj4m2NEzJB0VY7h6NwU+/mi4zCz6ufky8zatdzaaH9gLnBhrVnkC5UTrWclvZ5jGijptorG3mZmjfJUE2bWLinZETiKNEbr6jITr0r5kuNlwFvAcZK2kKRGNjMzA1z5MrN2SNKKpDsZuwKXRMTUkkP6nDzo/nFJ40jj0AZJGh4Rn5Qcmpm1c658mVm7katdWwHHAeOAy9tj4lUpTz9xEamR9Yl5fjAzs3q58mVm7YKk3qQ5tVYAroiI90sOqcnyBKwPSBoL7CdpIHB3RMwtOTQza4dc+TKz0knaBDgB+AC4uJoSr0oR8TYwFJhPqoKtX3JIZtYOufJlZqXJzbC/DqwJXB8RE0sOaanlZtx3ShpDqoKNJjXp/rTk0MysnXDly8xKIWlDUjPsOcDQjpB4VYqI14ELgJ6kJt1rlRySmbUTrnyZWaEkdSc1w94IuDUi3iw5pDYTEXOAm/Nl1YOV2hw9EhF1Nfk2s07ClS8zK4ykdUhju5YhNcPusIlXpYh4jTQWbDVSk+7VSg7JzErkypeZtbncDHsIsCWpJ+PociMqXkTMknQd6TM4XNJTwFNu0m3W+Tj5MrM2JakfqRXPVFK1q9NOQprbE70s6S3SJLIbS7q1vc9lZmaty8mXmbWJ3Ax7Z2AH4D5ghJtQJxExXdIVpM/mGEkPAy/48zHrHJx8mVmrk7QKqdo1H7jIjac/Lydaz9TRpLtd9K80s7bjAfdm1mpye6DtgaOBEcBVTrwaFhFTgEuBt4HjJW3uJt1mHZsrX2bWKiStAOwLdAcujYiPSg6pauRB94/mJt01VbA7ImJ2yaGZWRtw5cvMlkqudm0JHA+8BVzmxKtlIuJd4EJgOqk90YCSQzKzNuDKl5m1mKTlSM2wVwKujIjJJYdU9XKT7vsq2hMNBO6JiHklh2ZmrcSVLzNrEUmDSO2BppCaYTvxakURMYE0MesiUhVsvZJDMrNW4sqXmTWLpJ6kZthrAzdExDslh9Rh5WrXcEkbAd+S9CrwoJt0m1U3V77MrMkkrU+qds0nNcN24lWAiBhHatLdm3RH5Jolh2RmS8GVLzNrlKRuwH8AA4HbIuKNkkPqdPKdjzdJ2hQ4VNILwGNu0m1WfVz5MrMGSVqL1Ay7J6k9kBOvEkXESNJYsDVIs+P3LTkkM2smV77MrE6SupKaYW8F3BURr5UbkdWIiI8lXQtsDRwh6QngGTfpNqsOTr7M7HMkrUaa7HMGaWzXrJJDslpye6IXJb1JatI9IDfpnlZyaGbWCF92NLPPSOoiaWfgcOAZ4HonXu1bTrauAMYAx0raxu2JzNo3V77MDABJK5OqXQtIzbCnlxySNVG+3PhUrSbdt0fExyWHZmZ1cOXLrJPL7YG2A44BXiXNVO/EqwpFxAfAJcC7wAn5zkgza2dc+TLrxCQtT2qG3ZPUk3FKySHZUspTTzwsaSywf+5EcKebdJu1H658mXVCudq1OakZ9gTgUideHUtETCI16Z5Jak+0UckhmVnmypdZJ5ObYe8FrApcHRHvlRyStZHchujeiibdbwD3uUm3Wblc+TLrRCQNIE2YOo00qN6JVycQEeNJ7Ym6kMaCrVtuRGadmytfZp2ApB7A14D+wE0RMaHciKxoudp1W07AD5A0ktSke0HJoZl1Oq58mXVwktYjNcNeRJow1YlXJxYRY0hVsBVITbrXKDkks07HlS+zDio3w/4qMAgYHhHjSg7J2omImC3pH8CmwGGSngced5Nus2K48mXWAUlak3QnYy9SM2wnXraESF4h3RG5FnC0pD4lh2XWKbjyZdaB5GbYuwLbkpphv1pySNbORcRMSdcA2wBHSnoMeDb3jjSzNuDky6yDkNSX1FrmY9LYLreWsSbJidYLFU26B+Ym3e50YNYGfNnRrMrlZthfAo4Angeuc+JlLRERU4FhwDjgOElbu0m3Wetz5cusiklaiVTtCuDiiJhWckhW5XKT7ifraNI9q+TQzDoMV77MqlBuD7QtcCwwChjmxMtaU0S8D1wMTCZNzDq45JDMOgxXvsyqjKQvkJph9wIuj4gPSw7JOqg89cRDFU26B5Ju5JhTcmhmVc2VL7MqImlTUnugiaRm2E68rM1FxERgKDCb1KR7w5JDMqtqrnyZVQFJvUjNsPsC10TEuyWHZJ1MbtJ9d27Sva+kcaQm3fNLDs2s6rjyZdbOSdqY1B5oJqkZthMvK01EvElqT7QMaSzYOiWHZFZ1XPkya6dyM+w9gfWBf0bE+HIjMksiYi5wax4DdqCkfwMPu0m3WdO48mXWDknqT6p2QWoPNL60YMzqERGjSVWwlUnzgvUrOSSzquDKl1k7ImkZ4CukhsfDI2JsySGZNSgiPpF0I7A58D1JzwJP5PnCzKwOTr7M2glJa5AmtfyAVO2aXXJIZk2S2xP9W9J40jQoG+f2RFPKjcysfXLyZVay3Ax7F2A74O6IGFlySGYtEhEzJF1F+rd8lKRHgefcpNtsSU6+zEokqQ+p2jUbN8O2DiAnWs9JeoMlm3TPKDk0s3bDA+7NSpDbA+0EHAm8RJq7y4mXdRgR8RFwOfAGcLykLd2k2yxx5cusYJJWJFUEugCXRMTUkkMyaxN50P0TtZp0D4+IT0oOzaxUrnyZFSRXu7YGjgPGkpphO/GyDi8iJpOadE8htScaVHJIZqVy5cusALkZ9t7AF0hJ1wclh2RWqDwB6wO5PVFNk+6784StZp2KK19mbUzSYFIz7Mmky4xOvKzTioh3SE2655GqYBuUHJJZ4Vz5MmsjkpYlNcPuB1wbEZNKDsmsXcjNuO+qaNI9BrjfTbqts3Dly6wNSNqQ1B5oFnChEy+zz4uIN0jtiXqQmnSvXXJIZoVw5cusFUnqDuwBbAjcEhFvlRySWbsWEXOAm/Mg/IMkvQw86ibd1pG58mXWSiStQ6p2dSVNmOrEy6yJImIUaSxYH+BYSauVHJJZm3Hly2wp5WbYu5MaC98REWNKDsmsKkXELEk3AFsAh0t6GnjSTbqto3HyZbYUJK1OmjzyI1K1y5NHmi2F3J7oX5LeIk1GXNOk+6OSQzNrNU6+zFpAUhdgZ2AH4F7gFTcPNms9uUn3lcD2wNGSHgZe8P8z6wicfJk1k6RVSdWuuaQ7GWeWHJJZh5QTrWdzk+6a9kS3+f+cVTsPuDdrotweaAfgKOBfwNX+JWDW9iJiCnApMIHUpHsLN+m2aubKl1kT5GbY+5L+z7gZtlnB8qD7xySNY3EV7A6Ps7Rq5MqXWQNytWsrUjPsN4DLnXiZlSci3gMuAqaS2hMNLDkks2Zz5cusHpJ6k5phrwhcERHvlxySmfFZk+77azXpvsdNuq1auPJlVgdJm5CaYX8AXOTEy6z9iYi3Se2JFpCqYOuVHJJZk7jyZVZBUk/gG8CawA0R8U7JIZlZA3Iz7jtyP9X9JY0CHoiIT0sOzaxernyZZZI2ILUHqplCwomXWZWIiNdJVbBepCbda5Ucklm9XPmyTi83w/4PYGPgtoh4s+SQzKwFcpPuf0oaDBwi6UVSk+6FJYdmtgRXvqxTk7Q2aWxXd+ACJ15m1S8iXiVVwfqRmnT3LTkksyW48mXVT+oBrAWsREqi5gPTgHdI40Hq2ETLAEOALYE7I2JUMcGaWRFyk+7rgK2AIyQ9CTzdYJPuFpxLzFrCyZdVrzTx6SBgMOnf8nxgIdCVdOJcgDQSGE3E9MWbqR9pksZppGqXJ2k064Bye6KXcpPufYEBuUn3knP1tfBcYtZSvuxoTSKpv6SQtG3ZsQAgrQscCGwCTAEmAR8CUzeHb/WDk/Prg4EDkdaV1EXSLsD3gKdIdzM68TIrUBnnkoiYBlwBjAKOkbTtZ+2JWnAuKSpu67icfHUwkk6WNELSzPx4WtJeday3saSbJU2XNFvSS5IGlRFzs6WT316kGa4/IH1DrcvCvHzqZDjwK3AqsB5p3q5/52/F7ZqkIfkX1aplx2KdS1PPJRXrX5j/rZ5aZJxNFcnTwOXA1sBhD6SB+c06lwB7VWMC5nNJ++Lkq+OZCPyUdHLZFngIuFXS5jUr5IkInwTeAr4MbAr8AphVeLTNlS4P7Ek6Ec5rbPWFwE2wziWw+a9hxelwe0TMaOswzTqARs8lNSQdAGwPvFtohC0QER8Cl24H01+HMx6EngubcC7J5pHOPXvmc5FZizj56mAi4raIuDsiXo+IsRHxc+BjYKeK1c4C7ouIUyLipYh4MyLuau68VpJ2lfSspLmS3pd0Xp62oWa5JJ0iaZykeZImSjq7Yvk5ksZImiNpvKTf50lOa5afLmmkpIMlvSHp435w2whYlnyynA1dtoEju8N13eG6reCYyP+uJ0Cv/4Pdr4Jd/wLbfRV+uCpMkHRvQ1U+SXtImi9plVqv/1bSiIrnX5T0aK4cTpJ0gaTlK5Y/Iun8vN0USR9IOldSl4p1ukv6Xf5sZkt6XtKeeVl/4OG86of5W+uwvOxrkh6XNE3S1MaOyay5mnguQakK9GfgUKBFE5sWfS4Bpr8N/70ejHgSNvorfGkC9KzvXFLjh7B1XzijB5zXzecSWwpOvjowSV0lHQz0Jo1xIv9n3Rt4TdI9kj7M/0kPqrXtMEnjG9j3msDdwMuku4mOBg4Bzq5Y7bfAL/Nrg4HvAJUJ3ifAUaSBricBBwM/r/VW/YGDgP37wl6zYdPD0gz0kA5kvxGw56Hw96Fw6iLoMhqGBPS8GPZcE95fBUbtDbdcDqdeDud1Sb9Ahlee3Gt5kDTG4zsVxyvSL5er8/PNgPuA24EtgG+R7py8rNa+DiO1Pvki8H3gR/l4alwO7Jb3vSlpXMpwSVvkz+rbeb3BwOrAD/Pz5YA/kaoNQ4AZjRyTWYvVdS7Jry8DXAf8pr47htvzueRU+NJP4N7eMPvb8N8j4Ou1zyWVO5gFPfeG230usaUWEX50sAewGekS4gJgOrBXxbJ+QJBOVj8h/Sf/SV63cr2zgQcrnvfP222bn58FjAO6VKxzBKki1Yt0kp4LnNCMuE8AXq94fnrexwoRQcAGe8Nty8G7AXsH7N0TPtoVrqp5PgG+0xum9oFJ/4bv1bxe8Tjh4XRiWgjs3EAsfwQer3i+c95mrfz8SuDSWttsmT+jvvn5I6Rb2yvXuR+4JP+8AbAIWKfWOrcC5+efh+R9rtrIZ7dcY8fkx+J/V2XHUC2Phs4leflZpEv5Nc/HA6fWWqdaziXTt4Ln/w5nToP958I+vWHSavBKHecRn0v8WKqHp5romMaQ/vOuABwAXCFpSESMZHG187aI+GP++V9Kdx59H7gTICJ+1sh7DAKeiSXnzHmCdFv2hkBPoAfpm1+dlMaJ/Civ35t0W3fXWqtNiMVjtFbqA1PmwYoAI6HXXFh5RxgNMBOW+Qsc0Afe/gS6bp5+WXA99PsFfHcybDwfVlyYTkBdgHUaOL6rgR9JWjciJpC+dT4aERPz8m2ADWtVDJX/3IA0LgRgBEt6F6iZ8HHrvM1rkirX6UEaX1MvpVZIZwI7AH3y8TR2TGbNVe+5RNIQUpK0ZUM7qKJzyQpDYPgi6H42HPAbuGF1GDMz/f8CfC6x1uPkqwOKNBng6/npi5K2A35MKudPIX2Lfa3WZqNIpfpWCaGxFSTtCFwPnJFjmw7sA5xba9XKMSTdAxbF4hPT53wKyy6ENRdBnJYuXXAB7NULZg+B1/rDordg+l2ptH+gpI0bCPMj4FJJTwGHA/dLOj0v6wv8G3imju32ymMt+gO9KraBdFmh5rXB+bVhpG+tSxxKXqd/fn6apNkVy08GZpJunJiZtz+5CcdkMKTW34k1zTzSv7UrJd1OqqSsAbxf8QtfwO8l/YpU8alLzUD14yR9k5R8za/1d7Jy/vMEoFv++QeSlpyfK1mLdMnxEdKlu7nAAGCPin0OAVaqeX44rP0BbBnQ7TSlPI56AAAX/ElEQVQ45JP8HmPgq9NTshfXpqrWEo6DX/WGKSfA37eHhe/DG/8F15ASxTpFxEuSRgOHSjqXdAnytIpVugCXAOfVsfmkip9rj6erSfxq9hHAdnWsN6e+2LI7SDdXHJ/fr+b3gy87tiEnX51DF9I3ICJivqTnSSenShsDE5qxz1GkX/RdKr6x7kyanPAN0rfOecBXSJcUavsSMCkizqx5QY3fvj2/S8U4xU1hdk+Y+mw6lhHLw4Jz4JLLYKeesHAXeHpV+OgPcMjJ8Ltz4BWg7ykw9670Lf7WiBhW35tJmk8aP/E86ZfK9yJiZl62EbBmRPxXA9sPAUZGxOkVr/Unlf1Pz0nSAcA/I+LhevbxRVLi938R8X5+bRXg18CBNdtJ2hr4QWPHZGnwdeXfiTWdpF2Bd/O/374srrzUuJc0BuziiBhTzz76k8YbXRQRL0jqRppn639rziWSjiCdS04lnUu+B7wWEUPr2N8pwDsR8eWK1/4C7FHz95yTrlU/+3uXtj0GTgQ2+T1ctxC4DPZ4D5Y7Cq7ZD8YF8AM4qVeajJnn4Asfw1onwQUV55JuNO336NWkitdI0mW9myqWvQQMjtQYvKVeJp2j+tV3LiF9nlBREcznkoHASbXOJc4N2pg/4A5G0jmkS4fvAF8gJQ9DSHPZ1Pg9cKOkx0kl6d1JVa/9KvZzNrB9RHylnrc6n1TmP1/Sn4H1gXOAv0XE7LyPPwNnS5oHPAasAmwTERcAY4E1JR0GPE2aPuKQRg5vWtda/2Z3guFPwAFHwaRdYcKf4BtzYbleMOFl2KQXzOwOM2+CPTeHKZNhwPnwVdK3u8ZcA/yGVJIfXpN4Zb8DnpE0FLiQNPB2ILB3RBzfhH0TEWMlXQMMy79AXiJ94x8CvBkRN5MS4iBV04aTvsVOI1Uwj5X0DrAm8IcmHpNZkzR2LomID1h8Saxmm0+ByZWJV3s/l7wFy90IO24GI1+GLe+EPh/D7D/BN+bAyjXJ1xYwy+cSazVlDzrzo3UfpEtYE1g8H80DwJ51rHcE6aQ1hzSW4JA69jO+4nl/KgbJ5td2BZ7N7/U+qWzeo2J5F+D/AW+SvnW9A5xVsfxs0kzSs4CbSd9Eo2L56aTKUYoLehwDl3WFOTWDXmfCvlvCrd1gVjeYtTkM3xTuWg1emQ37Xgq/2BfuXh4mdYH5q8Ck5dMvj1nAEU34PB/Lx71PHcu2Be4hXYr5hPRt+H8rlj9C+gVS+3O9o+J5t3ycNZ/RZNKlk20q1vkl8B7p0uKw/NqXSd+i5+Y/92zqMXX2Bx5w39TPqUnnklrbjOfzA+7b5bnk6HQumfdzuPxaOO0j2K++c0nN+eY0+PnyMMHnEj+W9qH84ZtVB2knUhuQDxpbtcazsOrdsONg+PRLcMMaEY+3XYDW3vmyo0nq/RP4ySDYaAe4b7N8c04z9AVGEvFsW8RnHZ/n+bJqM4o0ZqFHUzfYAaacAg/3gNgZdpa0ftuFZ2btmaRNgBMeg5cPhyc3a3xAem09SOegOse0mTWFK19WfRb3dmxSiyHSybIvcKdSaX4f0vQU90dEi2bkturlylfnJGlZ0gTNawC3RMTEpTmXkKaNMGsRV76s+qST3p2kAaV9+fx8PjW65uUrk0+Wke4ouoB0O/kJktYqIGIzK5GkDUnjwGYDQ6Nmjq2lOJe0dczWsbnyZdUrNbYdQJqFexnS/DYLSSfKbqQ7dl4BxhDxuTEd+fLDN0h3Bj0aEQsLitxK5MpX55Fb5OwBbESaWPrNelZcqnOJWXM5+bLql06wa5Mmb+xBunwwHXiHNOFsA5uqN6nX5QqkSxHvt3G0VjInX52DpHVI0+e8DdwTEXObsFGLzyVmzeF5vqz6pZPiGy3bNGZJup7UIuXwPJv9U7FkqxMzqxK52ffupG4Sd0TE6CZvvBTnErPmcPJlnV6k8u/Lkt4ifVPeWNKtEVFXKxMza6ck9SO1DvsIuCAiPik5JLM6ecC9WRZpLMcVpL5mx0jaVrW61JpZ+yOpS2599J+kfqc3OvGy9syVL7MKuQr2jKTXgf2BgZJujyXbgZhZO5H7E+5PmtX9woiYUXJIZo1y5cusDhExBbiU1MbkeEmbuwpm1n4o2QE4mtQi7SonXlYtXPkyq0cedP+opLGkcSQDJd0RudmvmZVD0gqk8ZndgEsj4qOSQzJrFle+zBoREe8BF5JuOT9R0oCSQzLrlHK1a0vgeFID6cuceFk1cuXLrAkiYgFwn6QxwH6SBpLmDmpKSxIzW0qSliPNybcScGVETC45JLMWc+XLrBkitRUZCiwiVcH6lxqQWScgaRCpPdAU4GInXlbtXPkya6Zc7RouaSPgW5JeAx50k26z1iWpJ/B10qzzN0TEOyWHZNYqXPkya6GIGEdq0t2bdEfkmiWHZNZhSNqAVO2aT2qG7cTLOgxXvsyWQkTMAW6SNBg4VNILwGNu0m3WMrkZ9leBgaRm2G73Yx2OK19mrSAiXiWNBVuDNDt+35JDMqs6ktYm3cnYk9QeyImXdUiufJm1koj4WNK1wNbAEZKeAJ5xk26zhknqCgwBtgLuiojXyo3IrG05+TJrRbk90YuS3iRNAjkgN+meVnJoZu2SpNVI7YFmkMZ2zSo5JLM258uOZm0gJ1vDgNHAsZK2cXsis8VyM+ydgcOBZ4DrnXhZZ+HKl1kbyVWwp3OT7pr2RLdHxMclh2ZWqtwMez9gAXBRREwvOSSzQrnyZdbGIuJD4BJgEnCCpE1LDsmsFLk90HakZtgjSTPVO/GyTseVL7MC5KknHpE0Dtg/z9h9p5t0W2chaXlgX9KdjJdFxJSSQzIrjStfZgWKiEmkJt0zSe2JNio5JLM2latdm5OmkJgAXOrEyzo7V77MCpbbEN1b0aT7DeA+N+m2jiY3w/4msApwdUS8V3JIZu2CK19mJYmI8aT2RCKNBVu33IjMWo+kAcAJwFTSoHonXmaZK19mJcrVrtslbQwcIGkkqUn3gpJDM2uR3Az7a8C6wE0RMaHkkMzaHVe+zNqBiBhLqoKtQGrSvUbJIZk1m6T1SNWuhaQJU514mdXBlS+zdiIiZkv6B7ApcJik54HH3aTb2jtJ3UjNsAcBwyNiXMkhmbVrrnyZtSORvEK6I3It4GhJfUoOy6xektYk3cnYi9QM24mXWSNc+TJrhyJipqRrgG2AIyU9BjybZ803K11uhr0b6d/oXRHxaskhmVUNJ19m7VROtF6oaNI9MDfp9ozgVipJfUnNsD8mje1yyyyzZvBlR7N2LiKmkpp0jwOOk7S1m3RbGXIz7C8BRwDPA9c58TJrPle+zKpARCwCnsxNuvdncZPuWSWHZp2EpJVI//YCuDgippUcklnVcuXLrIpExPvAxcBk0sSsm5QcknVwuT3QtsCxwChgmBMvs6XjypdZlclTTzwkaSyLm3TfFRFzSg7NOpjcDHsf0p2Ml0fEhyWHZNYhuPJlVqUiYiIwFJhNatK9YckhWQeRq12bkaaQmEhqhu3Ey6yVuPJlVsVyk+67JY0mNekeR2rSPb/k0KxKSeoF7AX0Ba6JiHdLDsmsw3Hly6wDiIi3SO2JliGNBVun5JCsCuUeoycCM0nNsJ14mbUBV77MOoiImAvcKmkgcKCkfwMPu0m3NUZSD2BPYH3gnxExvtyIzDo2V77MOpiIGE2qgq1MmhesX8khWTsmqT+p2gWpPdD40oIx6yRc+TLrgCLiE0k3ApsB35P0LPBEni/MrKYZ9pdJjdyHR8TYkkMy6zScfJl1ULk90QhJE4B9gY1ze6IpJYdmJZO0BmnC1A9I1a7ZJYdk1qk4+TLr4CJihqSrgO2AoyQ9CjznJt2dT26GvQvp38I9wEj/OzArnpMvs04g/4J9TtIbLNmke0bJoVlBJPUhVbtm42bYZqXygHuzTiQiPgIuB94Ajpe0pZt0d2x5wtSdgCOBl0hzdznxMiuRK19mnUwedP9ErSbdwyPik5JDs1aWm2HvBwi4JCKmlhySmeHKl1mnFRGTSU26PyS1JxpUckjWSnK1a2tSM+wxpGbYTrzM2glXvsw6sTwB64MVTboHAnfnCVutCkn6ArA38AVS0vVBySGZWS2ufJkZEfEOqUn3PFIVbIOSQ7IWkDQYOAGYTLrM6MTLrB1y5cvMAMjNuO+SNAbYJ1fD7neT7vZP0rKkZtj9gGsjYlLJIZlZA1z5MrMlRMQbpPZE3UlNutcuOSRrgKSNSO2BZgEXOvEya/9c+TKzz8ljvm7Jg/APkvQy8KibdLcfkrqTmmFvANwSEW+VHJKZNZErX2ZWr4gYRRoL1gc4VtJqJYdkgKR1SNWuLqQJU514mVURV77MrEERMUvSDcAWwOGSngaedJPu4klaBtgd2By4IyLGlBySmbWAky8za1RuT/QvSW+RJu2sadL9UcmhdRqSVidNivsRqdrlSXHNqpSTLzNrstyk+0pge+BoSY8Az7s5c9uR1IXUDHt74F7gFX/eZtXNyZeZNUv+xf9sbtK9PzBA0m0RMbPk0DocSauSPuO5pDsZ/RmbdQAecG9mLRIRU4BLgQmkJt1buEl368jtgXYEjgL+BVztxMus43Dly8xaLA+6f0zSOBY36b7D45FaTtKKwL6k8/OlHldn1vG48mVmSy0i3gMuAqaS2hMNLDmkqpOrXVsBxwFvAJc78TLrmFz5MrNWkSdgvT+3J6pp0n2Pm3Q3TlJvUjPsFYErIuL9kkMyszbkypeZtaqIeJvUnmgBqQq2XskhtWuSNiE1w/4AuMiJl1nH58qXmbW63Iz7Dkkbkqpgo4AHIuLTkkNrN3Iz7K8DawI3RMQ7JYdkZgVx5cvM2kxEvE6qgvUiNeleq+SQ2gVJG5DaA9VMIeHEy6wTceXLzNpURMwB/ilpMHCIpBdJTboXlhxa4XIz7P8ANgZujYg3Sw7JzErgypeZFSIiXiVVwfqRmnT3LTmkQklamzS2qztwgRMvs87LlS8zK0xu0n0dsCVwhKQngac7cpPu3Ax7COmY74yIUeVGZGZlc/JlZoXK7YlermjSPSA36Z5acmitTlI/0uSz00jVLk8+a2a+7Ghm5YiI6cAVwGvAMZK27SjtiSR1kbQL8J/AU6S7GZ14mRngypeZlShXwZ6paNI9UNLt1dzHUNIqpGOZT7qTcUbJIZlZO+PKl5mVLiI+JDXpnkhq0r1ZtVXBcnug7YGjgVeAq5x4mVldXPkys3YhTz3xiKSxpMrRoNyke3bJoTVK0gqkZtjdcTNsM2uEK19m1q5ExLvAhcB0UnuijUsOqV652rUFqRn2W8BlTrzMrDGufJlZu5ObdN+Xm3Tvl5t03xsR80oO7TOSlgO+CaxCusQ4ueSQzKxKuPJlZu1WREwAhuanJ0rqX140i+Vk8ERgKqkZthMvM2syV77MrF3L1a7b8+XHb0t6FXiwjCbdknoCXwPWAW6MiLeLjsHMqp8rX2ZWFSJiLKk9UW/SHZFrFPn+ktYnVbs+BYY68TKzlnLly8yqRr7z8SZJmwKHSXoBeKwtm3RL6kZqhj0QuD0iXm+r9zKzzsHJl5lVnYgYKWkCsA9pdvyb81xhdZN6AGsBKx0OayNtS2r58w4R8+vfTGuRpr2YRGoPNKc1j8PMOicnX2ZWlSLiY0nXAlsDR0p6AnhmiSbd0orAIGAw6Xw3fzCsDGxOmpNrAdJIYDSp3VHeTF2B3fK+74qI1wo6LDPrBDzmy8yqjqSQdEAkLwIXAwOAwyWtlFdaFzgQ2ASYQqpeffghzAE+zM+nkBKzA/P6SFoNOBboRxrb1eLES9Kpksa3dHsz65icfJlZm5F0sqQRkmbmx9OS9qpYHvU8/t6c94mIaaQm3WOAY0+W9loEe5GmgvgAqG9M2MK8fOoC+OaR0reBw4FngesiYlbzj9rMrGFOvsysLU0Efkq6fLct8BBwq6TN8/LVaz32zq/f2Nw3iohFEfHUD+Hm7eDg82HjSU08x70O3f4Km3wF9j4TboiIl3PTbzOzVufky8zaTETcFhF3R8TrETE2In4OfAzslJdPrnyQ+iOOjYhHm/M+uRH3A5Lm/B1e+hOsPx9mXABfuzvNycVB8OUV4a//Bwf1hCsHwI8WAv+EjQ6FH58B+x8Jh5wJz0u6RGm8WM3+j5A0S9JXJI2U9ImkhyWtVyuO0yRNzuteSZoWo3L5dpLukzQlVwKfkLRTSz5bM6teTr7MrBCSuko6mJSQPFXH8t7AwaTxW5Wvny6p3ipUbvNzLzBrLfjSb2Hom9D/r/DV/eHR52CzIXDSTXDy9vDAIXDXb+CMNeDdP8KQ0bD+F2D08fD3f8APfgPDBDsAf631Vj2AnwFHkZLHFVk8+z6SDgR+A/yaVOkbA/yk1j6+AFwF7AJsD/wLuEvSKg1+eGbWofhuRzNrU5I2A54GegKzgP0j4pU6Vj2UdAfiFbVen0JKZOpzKLAc8L13oC/w5lT42znw2zFwxSlwzzlwxSAY9TN47m4YMhAWTodP+8GUA+G1n8Oiiv11uQf++BAMlXR4xd2TywAnR8SYfFznApdJUr5E+SPgioi4MK9/lqTdgQ1rdhwRD9X6bH4AfBv4OnB1A8doZh2Iky8za2tjgC2BFYADgCskDYmIkbXWOxa4rfZ8XRHxN+BvDex/EDAiIj5GGgDMPxVGnwOLHoO114bZc2H5neDBR2C7D2Hjl2DGt+HRrdKAfH4Km18FB0yHtRdA74UQpESwH/Bufp95NYlX9m5eZyXSfgYBl9SK7Wkqki9JfYEzgd2B1YCuwLLkS6Nm1jk4+TKzNhVpEtOaWeFflLQd8GPg6Jp1JG1JGpD/P0v5dt2puLNRKYkCYHmY9mN4fBh8dDw8vWyudt0Cfc6FX20G950I12wKXS+DHnfAGXl/NRbUPrT8Z3OGb1xBSrp+DIwH5gEP1nofM+vgPObLzIrWhTR+qtJxwFvAAy3Y3yhgM0lfAOYDXc9NrYC67AITd4EZy8JHj8MWK8KCH8GTy1ZcZrwDNloEyzwFl/wSxuwPU95N47laEseOtV6r/Xxn4K8RcWdEvEq6+WD1FryXmVUxJ19m1mYknSNpF0n98x2JZwNDgGsq1ukFHAZcWtf0DpK+L2l0A29zDTAbuPIY6HMebPJ3OLk/PHUovAfwFbjxRdhnT9j3Sljjz7Def8B+AFuly4dd9oV9/gGr/RB2GpEG/jfXn0mTvB4raSNJPyMN3K80FviupE1yBfB6UsJoZp2ILzuaWVvqRxpI3g+YAYwAvh4R91ascxBpwPzl9exjVdLs9XWKiNmS9gT+dCnc3B0WrQ/P/gNqBr4zHO7+Nix4APa7Hw7vDrPWgxcAvg/j74SLHoEDHoLvrg1v9oBTFsC1zTnQiLhB0vrAWUAv4Hbgj8ARFasdBVwEvEhK+k4H+jTnfcys+snzCJpZh5LmzdqENHN9c/UFRhLxbOsGZWa2mC87mllHM4p0F2HtcWWN6ZG3a2haCzOzpebky8w6lojppElX+9L0BKxHXv/evL2ZWZvxZUcz65ikdYE9SVNPfETdzbW7AqvkP+8lYkJxAZpZZ+Xky8w6rtSfcQCwGekGo09JSVhXoBtp7q5XgDGueJlZUZx8mVnHJ3UH1ibN39WDNLnpdOAd0iSwZmaFcfJlZmZmViAPuDczMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMrkJMvMzMzswI5+TIzMzMr0P8HzJ4RnUQzy2MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from graph_utils import show_cluster_graphs\n",
    "\n",
    "good_diff_all_ids = {34, 35, 36, 37}\n",
    "show_cluster_graphs(\n",
    "    df,\n",
    "    golden_pairs_set, dedupe_found_pairs_set, dedupe_unclustered_found_pairs_set,\n",
    "    good_diff_all_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And here's a case where the clustering process **dropped a true match**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>addr</th>\n",
       "      <th>city</th>\n",
       "      <th>type</th>\n",
       "      <th>postal</th>\n",
       "      <th>lat</th>\n",
       "      <th>lng</th>\n",
       "      <th>addr_variations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bel air</td>\n",
       "      <td>701 stone canyon road</td>\n",
       "      <td>bel air</td>\n",
       "      <td>american</td>\n",
       "      <td>90077</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "      <td>(701 stone canyon road)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>hotel bel air</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "      <td>90077</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "      <td>(701 stone canyon road)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bel air hotel</td>\n",
       "      <td>701 stone canyon rd.</td>\n",
       "      <td>bel air</td>\n",
       "      <td>californian</td>\n",
       "      <td>90077</td>\n",
       "      <td>34.086594</td>\n",
       "      <td>-118.446351</td>\n",
       "      <td>(701 stone canyon road)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name                   addr     city         type postal  \\\n",
       "8        bel air  701 stone canyon road  bel air     american  90077   \n",
       "6  hotel bel air   701 stone canyon rd.  bel air  californian  90077   \n",
       "7  bel air hotel   701 stone canyon rd.  bel air  californian  90077   \n",
       "\n",
       "         lat         lng          addr_variations  \n",
       "8  34.086594 -118.446351  (701 stone canyon road)  \n",
       "6  34.086594 -118.446351  (701 stone canyon road)  \n",
       "7  34.086594 -118.446351  (701 stone canyon road)  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAFBCAYAAABJi8prAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm0nXV97/H3JwmEmSAYGWQGBYLVVmuxerFWKlVUrFVqW8drW69i77W9Xltb9eK1drRX60Lv1dap2svQ1okGFQEVQUGUMWRgSkIgkDAFQoCM3/vH7wn7EE7ISXKy9xner7XO2ufs/Tx7fzcLFp/1fX7P75uqQpIkSf0xZdAFSJIkTSaGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqI8OXpDElyaIkNYKfRaP8uVOSnNH9vHmY1/9iyGe/aDQ/W9LkMm3QBUjSGDEF+J/d7xcB/zzAWiRNYIYvSWNKVR029O8kNeS1jOQ9kuxSVY+OcmmSNCq87Chp3ErylSGXAl+Y5KtJHgCu716/tHtt3SbnPe75JL8HrB1yyEuHvO+Fw3z0Hkk+mWRZkgeSfCfJUTvqe0qaWOx8SZoovgHs2/1+9w7+rC8CTxvy98uAbyT5uapav4M/W9I4Z+dL0kSxAjgB2A149dacWFX/BOw05KmLqirdz0nDnPIw8GzgIGBB99xxwC9sddWSJh3Dl6SJ4s+q6oqqeqSq5u7gz/rbqrquqpYC3x7y/KE7+HMlTQCGL0kTxdVbcez2LrlYMOT3VUN+32U731fSJGD4kjRRPDLMc6u7x6lJpkHbzws4bJhja5jnNmfo4vytOU+SDF+SJrTFQ35/Vff4Hh6/WB6AbqH8A92fhyWZsYNrkzRJGb4kTWT/MuT3ryZZCfw9w3fJAC7vHo8E7u+2mvjAjixQ0uRj+JI0YVXVRcA7gJtolyBvAl4LXLWZU06nLaC/vy8FSpqUUuVyBUmSpH6x8yVJktRHhi9JkqQ+MnxJkiT1keFLkiSpjwxfkiRJfWT4kiRJ6iPDlyRJUh8ZviRJkvrI8CVJktRHhi9JkqQ+MnxJkiT1keFLkiSpjwxfkiRJfWT4kiRJ6iPDlyRJUh8ZviRJkvrI8CVJktRHhi9JkqQ+MnxJkiT1keFLkiSpjwxfkiRJfWT4kiRJ6iPDlyRJUh8ZviRJkvrI8CVJktRHhi9JkqQ+MnxJkiT1keFLkiSpjwxfkiRJfWT4kiRJ6iPDlyRJUh8ZviRJkvrI8CVJktRHhi9JkqQ+MnxJkiT1keFLkiSpjwxfkiRJfWT4kiRJ6qNpgy5AGhXJdODpwD7AzsAa4H5gCVVrBlmaJElDGb40viUzgGOBWbR/n9cA64GptBC2jmQOMJ+qFQOrU5KkjpcdNSqSHJakkjyvjx96KHAacNx/gpP2hvcDdwP3dY93APcAsy6Ed45GfUnOSAtz2/Meb03y0Pa8hyRp/DJ8aVhJDkjypSR3J3k0ydwkL+7D534/yZkjOPBQ4BRa0Fq+ATZs5sj1wPKHYQXAa2H/0ap1O5wDHDHoIiRJg2H40hOkXcq7DAgt4BwL/CGwfJB1PabVdzKtntUjOWVtuxzJy+EF3fkDU1WPVNVm/1kmmZYk/axJktQ/hi8N533AnVX15qr6SVUtrKqLqmreCM49NMl3kzzcdct+beiLSU5MckXXTVuW5ONJdu5e+yLwYuD07hJhJTmse+24JLOTrNwZbj0e3n4x7Lq1X+xqeOoM+H73+fOTvGyT+h77nCTLk5yVZKu6ZUn+OsmCJI8kWZTkb5PsMuT1x1123Hgps3v+Flqg3H1rv5skaXwwfGk4rwGuSHJOF0CuSfLuod2YLjDUMOd+FPgk8GzgSuDsJHt05xwEfAu4Gvh54O3AbwN/1Z3734AfA18ADuh+liQ5ALgEmHMovPDz8MmVMOU0+MDq1p0bsS/Aa98Klz8dfhH4LvCNri6Gfg7wfOAkYI/umK35b2UV8J9pHcN3AW8A/nwL5xwO/A7weto/u0e34vMkSeOI4UvDOYIWGm6lXd77B+CvgdOHHHMPsGCYcz9eVedV1U3AnwFPAZ7TvfYuYCnwrqqaV1X/Afwp8O4ku1XVA7TLgw9X1V3dz3rgncC1VfUni2DVG+Gur8Pf3wvP+DgctTVf7BfhW5+Aa5bAw7Swt6R7f4Z+TlffdcCbaUFsxAv1q+ojVXVZVS2qqvOBv6SFzCezM/CmqrqqquZU1bqt+V6SpPHDrSY0nCnAT6vq/d3fVyc5mha+zgSoqjM3/r6J64b8vrR7nNk9HgtcXlVDF8dfSgseR21y7lDPBU5M8tAUmDIFphQUwJzWHbtp44GrYcpqmLIXDBteToD5wFpgRlVtSHIFcNymnzPMqUcCP9lMfY+T5HXAe7rvtAdt24upWzjt9qpaNpL3lySNb4YvDedOYO4mz82jdYq2ZO3GX6qquiuVI+mwDncJc6MpwGzgvZ+En386HLYKHgB4bncX40bXwz5fgDccDdceC4tfCEv3aHc8DrUemP5knzPMayMKRklOAM4GPgz8Ea2+VwMf28Kpq0by/pKk8c/wpeFcBjxzk+eeASzezvedB5yWZMqQ7teLaJcab+n+XsMTu0RX0fbzWnw6zAD2pO3j9QTPg3vPhgcOg6XXwlGXwPP3h6Wr4SGAK9r3ugtY3a1hez7wb5t+TlWtHe79R+CFwB1V9ZGNT6RtiyFJEuCaLw3v48AJSf48yVFJXg/8V+BTGw/oFuDP38r3/TRwIPDpJMcmOYW2luzMqnq4O2YR8Pxu09b9uoXunwL2Bs55PRxyARz4Xnj2LDh9wTB3PE6B9a+BW98H3zsdZh8A9yzq1ob9BE49HU76w7YW7RPAocD/6U597HOS/FKSI5KclOSzSfYc4Xe8ETgoye9257+TLa/3kiRNIoYvPUFVXUm74/E02p1/HwU+SAtPG+3HE7tjW3rfO4CX0+50vAb4PHAWbWH+Rh+jdb/m0rpbh1TVUlpHacO/wedeAR/6LLxzGqzdb8hlzuEcCI++Dm56SevmcRKc/1V46ZlwPvCbwDtoO+Ez9HOAbwM30ALZaka4n1hVnQf8HS3YXQf8GvChkZwrSZocUvVkS22kMSh5AW2R/LAblb4PfvtvW6gbzkxgTtoi/VnA8bTu2Vxa0Lyj/I9CkrQDueZL49E84Fm0RfMj6kh1ptPWky2oNmT7h8APkzyVFsReA0zrZjfOAZYZxCRJo83wpfGnagXJd2ijj0Y6Ymg6res1mxa8hrxd3U3b9f4HwNNo3bA3AOu6IHZDd4wkSdvN8KXxqWoxyWzaJrDrgXt54pYS0Dpd+3aPs6na7B2bXZfrLuCuJBfRbg44Hnhzkodp3bAbquq+Uf0ukqRJxfCl8asFsHNpC/+fRfv3ee1T2xqumcBOtM1WrwcWbNrxevK3rqItxL8jyQXAIbQg9vYkK+gFsQdH8ytJkiY+F9xrYmjDuQ8GZrwV3vXFdmfmCmAJVWtG72MyhTaHcRZtx/67aUFsblUNtzO+JEmPY/jShJPkjKo6ow+fM5U2duh42ia0S2lBbF5VPbKjP1+SND552VHaRt3Q7xuBG5PsBBxN64idnOQ2WhCbX1Vbc0emJGmCM3xJo6AbRzQXmJtkOq0TdjzwiiS30oLYTTWKl0AlSeOT4UsaZV2n63rg+iS7AscAvwC8KsnNtCB2c1WtG2CZkqQBMXxJO1C39utq4Ooku9MW6Z8AnJpkAS2ILewuYUqSJgHDl9QnVbUK+Cnw025Q9yzgV4DXJplHC2KLq2rD4KqUJO1ohi9pAKpqJXA5cHmSGbQg9jJgzyQ30ILY7Y43kqSJx/AlDVg3Z/Iy4LIk+9IW6r8a2HlIELvTICZJE4PhSxpDqupe4AdJLqHt0j8LeB3AxoHfVbV8gCVKkraT4Usag7ou1zJgWZLvAQfQOmK/m2Q1cAMtiN07wDIlSdvA8CWNcV0QWwosTfJd2hilWcDbkqykN2dyxLMrJUmDY/iSxpEuiN0G3JbkO8ChtI7YHyS5l96cyZUDLFOS9CQMX9I41W1JsRBYmOR84AhaR+wlSe6iF8QeHmCZkqRNGL6kCaDbpPUm4KYk04CjaB2xk5LcTm/O5KMDLFOShOFLmnC6sUXzgflJdqbNmZwFvDzJIloQW+CcSUkaDMOXNIF1AWsOMCfJLsAzgZ8DXrnJnMm1AyxTkiYVw5c0SXSXHK8Frk2yG23O5PNpcyZvpAWxW5wzKUk7luFLmoS6Rfg/A36WZA/gOOBFwGuSzKcFsUXOmZSk0Wf4kia5qnoI+AnwkyR709aHnQTsnWQuLYjd5ngjSRodhi9Jj6mqB4AfAT9K8hRaEHsFsOuQOZNLDWKStO0MX5KGVVX3AT8EfpjkqbStK14LTBkSxJYZxCRp6xi+JG1RVd0NfC/J94H9aR2xNwDrhgz8vmeAJUrSuGH4kjRiXZfrTuDOJBcBB9E6Ym9JsorenMn7B1imJI1phi9J26QLYrcDtye5ADiE1hH7vSQr6AWxBwdYpiSNOYYvSdut25JiEbAoybeAw2kdsROTLKc3Z3LV4KqUpLHB8CVpVHVB7BbgliSzgSNpHbGXJllKC2LzquqRAZYpSQNj+JK0w3RzJhcAC5LsBBxN64idnGQxvTmTqwdYpiT1leFLUl908yPnAnOTTKfNmTweOCXJrbQgdqNzJiVNdIYvSX3XdbquA65LsitwDPALwKuS3ERvzuS6AZYpSTuE4UvSQHVrv64Grk6yO23O5C/T5kwuoAWxhQ78ljRRGL4kjRnd3ZBXAlcm2YsWxH4F+I0k8+jNmXTgt6Rxy/AlaUzq9ge7HLg8yT60OyZ/Hdh9yMDv2x1vJGm8MXxJGvO6HfMvBS5Nsh8tiJ0KTBsyZ/Iug5ik8cDwJWlc6WZI/iDJJcBM2h2TpwEbNgaxqlo+yBol6ckYviSNS12XaxmwLMnFwAG0IPbGJI/SG2907wDLlKQnMHxJGve6ILYUWJrku8DBtCD2tiQP0gtiDwywTEkCDF+SJpguiN0G3Jbk28ChtCD2jiT30pszuXKAZUqaxAxfkiasbkuKhcDCJOcDR9CC2EuS3ElvzuTDAyxT0iRj+JI0KXSbtN4E3JRkGm3O5CzgZUmW0ILY/Kp6dIBlSpoEDF+SJp1ubNE8YF6SnYFn0DpiL0+ykN6cyTUDLFPSBGX4kjSpdQFrDjAnyS60OZPPps2ZvLl77SbnTEoaLYYvSep0lxyvAa5JshtwLPB84NQhcyZvdc6kpO1h+JKkYXSL8H8G/CzJnrQ5kyfS5kzOpwWxRc6ZlLS1DF+StAXdthRXAFck2Zu2UP8kYK8hcyaXON5I0kgYviRpK3Qbtf4I+FGSp9AW6r8S2GXInMmlBjFJm2P4kqRtVFX3AZcAlySZSeuIvRaYkmQOLYgtN4hJGsrwJUmjoBvmvTzJ94H9aR2x3wHWDBn4fc8AS5Q0Rhi+JGkUdV2uO4E7k1wIPJ3WEXtLklX05kzeP8AyJQ2Q4UuSdpAuiC0BliS5ADiE1hH7/ST30Zsz+eAAy5TUZ4YvSeqDbkuKRcCiJN8CDqd1xF6cZDm9ILZqcFVK6gfDlyT1WbdJ683AzUlmA0fSOmIvTXIHvTmTjwywTEk7iOFLkgaoG1u0AFiQZCfanMlZwK8nWUwLYguqavUAy5Q0igxfkjRGVNVa4AbghiTTgWfSOmKnJLmF3pzJtQMsU9J2MnxJ0hjUdbquA65LsittzuRzgVcnuYkWxG5x4Lc0/hi+JGmM69Z+XQVclWQPWhD7ZeA1Q+ZMLnTOpDQ+GL4kaRypqoeAK4Erk+xFWx/2q8CMJPNoQew2g5g0dhm+JGmc6vYH+zHw4yT70C3UB3YfMmfyDscbSWOL4UuSJoBux/xLgUuT7EdbqP8aYNqQIHaXQUwaPMOXJE0w3QzJ7yf5ATCTFsROAzZsHPhdVXcPskZpMjN8SdIE1XW5lgHLklwMHEgLYm9K8ghtW4s5VXXfAMuUJh3DlyRNAl0QuwO4Y8icyVnA25M8QG/g9wMDLFOaFAxfkjTJdEFsMbA4ybeBw2gdsf+S5G56cyYfGlyV0sRl+JKkSazbkuJW4NYhcyZnAb+a5E5aEJtXVQ8PsExpQjF8SZKAxwZ+3wjc2M2ZPIrWEXtZktvozZl8dIBlSuOe4UuS9ATd/Mh5wLwkO9PmTM4CXpFkIS2I3VhVawZYpjQuGb4kSU+qC1jXA9cn2QU4BngO8KohcyZvds6kNDKGL0nSiHWXHK8BrkmyG3AccAJwapIbaUHs1u4SpqRhGL4kSdukW4T/U+CnSfakBbETgd8YMmdysXMmpcczfEmStltVrQSuAK5IMoO2PuxlwJ5J5tKC2BLHG0mGL0nSKKuqFcBlwGVJ9qUFsVcCu2wcbwTcaRDTZGX4kiTtMFV1L3AJcEmSjXMmXwcwZOD3coOYJhPDlySpL6pqOXBxku8B+9OC2O8Aa4YM/L53kDVK/WD4kiT1VdfluhO4M8mFwNNpQextSVbSG/i9YoBlSjuM4UuSNDBdEFsCLEnyHdrA7+OBP0hyH705kw8OsExpVBm+JEljQrclxSJgUZJvAYfTgtiLkyyjN2dy1eCqlLaf4UuSNOZ0m7TeDNycZBptzuQs4KQkd9ALYqMzZzKZTrv8uQ+wM7AGuB9YgiOUNMoMX5KkMa0bWzQfmN/NmTya1hH79SSL6M2ZXL3Vb972JDuWFuym0ULXemAqLYSto90MMB/XoGmUTBl0AZIkjVRVramqG6rqHOB/A3OBZwF/nOS0JMcl2SnJYUkqyfM2+2bJocBptJ357wHuAO4G7use7/hPcNK+8O/Aad3xWy3JW5M8tC3namIyfEmSxrQkU5N8JMnCJI92j38BrK+qa6vq/wGfoF2mfB7w34GTu9OH//9cC1Kn0ILWclq36wk2wIZ1sK477pRtDGDnAEdsw3maoLzsKEka6/4EOB14C3A98HPAl4DVwEcAquoR4CrgqiR7AC/tzn1jkgNolyYXVtWG7lLjybTQNdJLlau7408mOXdrLkF2tT2yude7NW3r3Wh28rDzJUka634ZOK+qzquqRVX1TeCbwC8Nd3BVPQRc2/15LfC/aGvG7krygdnwKxtap2v1P8LBB8OHdoJzdoEvHwXvvRhmbKaO1bTznjn0ySR/nWRBkkeSLEryt0l2GfL64y47JjkjyZzu+Vu69919W/7BaHwyfEmSxrpLgZckOQYgyXHArwLnbzygCzTDdY4+DHyUti7s68AHr4Y3/QX88v+EE98Nf7M/LP6/8N8/Bh9cA7ueBh9YDdlMLfcCz6It/N9oFfCfaQv33wW8AfjzLXynw2m7+78eeDYwOndtalwwfEmSxrq/Ab4MzE2ylrYD/peq6tNDjrkHWDDMuf+nqs6tqvnAf9kJlv4Qdn09XPwf8Asz4IFXwP2BfU+EFV+F/30vPOPjbWuL4aynLdk5eOMTVfWRqrqs68qdD/wl8Ntb+E47A2+qqquqak53R6cmCdd8SZLGut8C3kzrFN0APAf4hyQLq+pzAFV1JnDmMOf+eOMvVbXhoGTuItj/WHhwOex1N+z/UXgTrdM1pXtkDhwA3LSZetYy5NJkktcB76EFtj1o21RM3cJ3ur2qlm3hGE1Qhi9J0lj3d8DHqurs7u/r0+46fD/wua15o6kwtaAACnIw/OxtcMFyOOgR2Hs6PPQc+OlL4MmC0XpgOkCSE4CzaZc3/whYAbwa+NgWSnGX/knM8CVJGut244lbQaxnZEtnTgAuBkhbBX/MC2DueXDYvrDuFjhuN/jBqfDTF8LSPTaz5cQmptK7S/KFwB1V9ZGNL2Yb9wPT5OGaL0nSWHce8KdJTuk2T/0N4I+Br208IMm7k8wf5tx3JnldkuOBf14NB70YchMc8lY4by1s+DS86Gew62zY773w7Flw+gLY9Unq2YnW4QK4ETgoye8mOSLJO9nyei9Ncna+JElj3R/S9vP6NDATuBP4R9oWEhvtxyZbQHT+gXa34xHAPbvBe/8I1uzV3mP9U+F9H4S3nAEfXg877Q53Hw5X79fWdQ1nKm3T1SUAVXVekr+jbfK6K3AB8KGuVmlYcU83TTRJzqiqMwZdh6T+SzKVto3D8bQwtozeEO5V3UEvoI0UWr4NHzETmEPVFaNSsCYlO1+SpHEtyRTgUFrgOpY2CmgOcFFVrRzmlHm0fb+mM/Id7umOn8rwW1pII2b4kiSNO0kCPJ0WuGYBK2mB67O1pdE/VStIvkOb7TjSEUPTaV2v2VszWkgajuFLkjQudIHrAHqBaw0tcH2hqu7dqjerWkwymzbjcT1t5/rh7nScCuzbPc6mavE2fwGpY/iSJI1pSWbSAtfx3VNzgP8HLN+uYdQtgJ1LWxv2LNr/E9fSQthU2l2N62jDvBfY8dJoMXxJksacJPvSulvH0y753QD8G3DndgWuTbVAdQXJ1bSRQTPorQVbASyhas2ofZ6E4UuSNEYkmUEvcO1JC1z/ASwZ1cA1nBawbtmhnyF1DF+SpIFJsidt24fjaWur5tH2ylpcVRsGWZu0oxi+JEl9lWQ3eoFrf2A+cAlwa1WNZLyPNK4ZviRJO1wbq8gxtMB1MHATcDlwc1WtG2RtUr8ZviRJO0SSnWl3Eh4PHAYsBK4Bzi0XsWsSM3xJkkZNkp2Ao2iB6yjgNtrWEF+rqkcHWZs0Vhi+JEnbpZuneCQtcD2DNrR6DjC7qh4eZG3SWGT4kiRttW6e4mG0wHUMcA8tcF1QVQ8NsDRpzDN8SZJGpBvvcwgtcB0HPEALXJ+pqgcGWZs0nhi+JEmb1QWuA+nNU3yEFrg+V1X3DbI2abwyfEmSHqcLXE+jF7g20ALXl6vq7kHWJk0Ehi9JEgBJ9qM3wHoaLXCdC9y1w8f7SJOI4UuSJrEk+9Cbp7g7bZ7i14E7DFzSjmH4kqRJJsle9ALXDGAu8G3gNucpSjue4UuSJoEku9ObpziTNk/xYmChgUvqL8OXJE1QSXYFjqUFrgOBG4EfAbc4T1EaHMOXJE0gSabTm6d4KHAL8FPgpqpaO8jaJDWGL0ka57p5is+gBa4jgMW0OxX/vapWD7I2SU9k+JKkcSjJNHrzFI8G7qAFrm9W1SODrE3SkzN8SdI40Q2wPpwWuJ4JLKcFrm9X1apB1iZp5AxfkjSGdQOsN85TPBa4nxa4Lq6qBwdZm6RtY/iSpDGmG+/zdNpeXLOAVbTA9U9Vdf8ga5O0/QxfkjQGdIFrf3rzFNcB1wNfqqp7BlmbpNFl+JKkAUryVHrzFKfQOlxnAcsd7yNNTIYvSeqzJE+hF7h2oQWurwJLDVzSxGf4kqQ+SLI3vXmKe9HmKf4HsMTAJU0uhi9J2kGS7EEvcO1Lm6d4IbDIeYrS5GX4kqRRlGQ3evMUDwAWAJcAt1bV+kHWJmlsMHxJ0nZKsgtwDC1wHQzcDPyENk/RAdaSHsfwJUnbIMnO9OYpHg4sBK4Bzq2qNYOsTdLYZviSpBHq5ikeTQtcRwK30+5U/HpVPTrI2iSNH4YvSXoS3TzFI+jNU7yTFrhmV9XDg6xN0vhk+JKkTXTzFA+j3al4LHAvLXBdWFUrB1iapAnA8CVJPDbe52Bah+s44EFa4PpMVT0wyNokTSyGL0mTVhe4DqS3F9ejtMD1+aq6b5C1SZq4DF+SJpUucM2kN96naIHrK1W1fJC1SZocDF+SJoUk+9HrcO1MC1znAnc53kdSPxm+JE1YSWbQ63DtTpun+E3gdgOXpEExfEmaUJLsRVswfzzwFFrg+jZwm/MUJY0Fhi9J416S3ekFrpm0eYrfBxY6T1HSWGP4kjQuJdmV3jzFg4CbgB8BtzhPUdJYZviSNG4kmU7bZX4WbRPUW4GrgLOrau0AS5OkETN8SRrTkuzE4+cpLqbdqfjVqlo9yNokaVsYviSNOd0A6yNpHa5nAEtpgeu8qnpkkLVJ0vYyfEkaE7p5iofTOlzHAMtpgeuCqnpokLVJ0mgyfEkamG63+UNpHa7jgBW0wPW9qnpwkLVJ0o5i+JLUV13gOojW4ZoFrAJuAP6pqu4fZG2S1A+GL0k7XBe4nkZvt/l1tA7Xl6rqnkHWJkn9ZviStMMkeSq9wDWF1uE6G1jmeB9Jk5XhS9KoSvIUegOsd6UFrq8CSw1ckmT4kjQKkuxNL3DtTZuneD5tnqKBS5KGMHxJ2iZJ9qA3T/GpwDzgQmCRA6wlafMMX5JGLMluwLG0LteBwI3ApbR5ig6wlqQRMHxJelJJdqHNUzweOAS4GbgSuNl5ipK09Qxfkp4gyc60sT6zgCOARcB1wL9W1ZoBliZJ457hSxLw2DzFo2gdrqOA22l7cX2jqh4dZG2SNJEYvqRJLMlUWmdrFu3S4jJa4PpWVa0aZG2SNFEZvqRJphtgfSitw3UscC9tL66LqmrlIGuTpMnA8CVNAt14n4NpHa5ZwEpah+uzVbVikLVJ0mRj+JImqC5wHUBvgPUaWuD6QlXdO8jaJGkyM3xJE0ySmfTmKUILXP9SVcsHV5UkaSPDlzQBJNmXXuDambaG69+AOx3vI0lji+FLGqeSzKA3T3FPWuD6JnC7gUuSxi7DlzSOJNmT3qL5fWnzFC8AFjtPUZLGB8OXNMYl2Z22JcTxwP7AfOAHwELnKUrS+GP4ksagbp7ixgHWT6fNU7ycNk9x3SBrkyRtH8OXNEZ08xQ3DrA+DLgVuBo413mKkjRxGL6kAUqyE3A0rcN1FHAbbWuIrzlPUZImJsOX1GfdPMUjaR2uZwBLaXcqzq6qhwdZmyRpxzN8SX3QzVM8jN48xbtpHa4LquqhAZYmSeozw5e0g3TjfQ6hBa7jgBW0Dtf/raoHBlmbJGlwDF/SKOoC14H05ik+Qutwfa6q7htkbZKkscHwJW2nLnA9jV7g2kALXF+uqrsHWZskaewxfEnbKMl+9OYpTqMFrnOAZY73kSRtjuFL2gpJ9qHX4dqdtobr68AdBi5J0kgYvjQxJNNpO8Hv8xY4mOR5wP3AErZzg9Ike9EbYD0DmAt8C7jNwCVJ2lqGL41vyQx6Y3imAWtmwVOAnwN2BtaRzAHmU7Vi5G+bPWh3KM4CZtKduKMbAAAHW0lEQVTmKV5Mm6foAGtJ0jabMugCNHEkqSSv2873eGuSke17lRwKnEYLSfcAdwB3/xD2DHzux/Bg9/ws4LTu+Cf77Gd23+FDwLtpnbTLgI9V1Teq6pYdGbySnJEWFCVJE5idLz1BkkXAcEHl/Ko6pc/lDK8FqVOA5cDqoS8dCfdcBG/+RVjZPbUcmA6cQjKbqsW9t8l0evMUn909PQ/4elWt3bbS8n1gTlW9e1vOlyRNbIYvDecXgalD/j4A+Blw7mDK2US71HgywwQvgJ1gw6+2DU2HWt0df/J5yddeDfvTOmZHAIuA64GfAu+jXVrcpuAlSdKWeNlRT1BVd1fVXRt/gFfQLuGNJHztn2R2koeTLE7yxqEvJjkoydlJ7u9+Zic5emvqOwE+vB98cBp8ZVf44nHwh9e2Ow8BmA8zA9/8MewJcBq8dBqc+2Z48b7wV6+GZcArgRuBT1TVWVV1PbBxYf6hSb7bfYe5SX5tk+9wYpIrkjyaZFmSjyfZuXvti8CLgdO7S5iV5LDuteO677syyfIkZyXZf2u+uyRp/DN86Ul1G4i+HfhKVT0y5Pm3Dg0WQ3wY+CbwHOCzwD+n3XlIkt2A7wGP0gLKC4A7gQu710ZS0PRD4Gm/D589B07/Y/jYUjj69fAHmx66Dqb8AA64D47YANNnw+t+C846Cz7yKvhMVV099DsN8VHgk7TLkFcCZ3cL8ElyEO1Ox6uBn+/+2fw28Ffduf8N+DHwBVrH8ABgSZIDgEtoe4E9HzgJ2AP4Rjf3UZI0SXjZUVvya8DhwD9u8vwDwAJg08tzX62qz3S/fzTJS4D3AG8E3gAEeNvGLRqSvIN2OfCVjKyz9vRz4fu0xfX8Jix/CL54JnxgNXxiCrAedqG92SsOgHt2gocL8n74m/fCLcBBb4B9gM2N+/l4VZ3X1fdnwJtpYfJS4F3AUuBd3eL7eUn+FPhMkg9W1QNJ1gAPd11Duvd5J3BtVf3JkOfe3NXwPOAnI/jukqQJwPClLfl94Mqqunbok1X1NeBrwxz/42H+3rhI/7m0ILeyNdQesxtw5Ajr2ecv4Ygz4R0r4OC1sFvB1A0w7QewzwyYshr2BTgJvncq3HUavDSw/l2wsHuPtbT9ujbnuiG/L+0eZ3aPxwKXb3LX46W0bS2O2uTcoZ4LnLiZOzmPxPAlSZOG4UublWQmcCpw+ii95RTgGloHbFMjGjr9BTj4Q/Ce4+E774R/ORRWXghHfhn+x0qY9jJYfhD8EDhmJqwa8sFrd2szFwHW0+5+3JzHunlVVV1QHMmlwSfbcHUKMBt47zCvLRvBe0uSJgjDl57MW2l3CZ61FeecAHx+k7/ndb9fRVsfdU9txYanQ/07HL0Bpv0I/mljmPpKuzvzMQf2OlybM5Vh7pIcoXnAaUmmDOl+vYi2WP+W7u81PP5uUWjf/TRgsXdSStLk5kJfDatbaP97wNlV9YRLZUl+I8n8bgH6UK9N8vtJjk7yfuClwCe61/6F1uX5RpIXJzm8u3Pw70d6x+NMmFOQU+HV/wpPeyOceGnrzm2NnXjiVhQj9WngQODTSY5Ncgrw18CZVfVwd8wi4PlJDkuyX7eg/lPA3sA5SX4pyRFJTkry2SR7bmMtkqRxyPClzfkV4GieuNB+o71pm5PutMnzZwC/SVv79E7a4vorAbpwciJwK/CvtJE9X6Itfr9/JEV9Hi76LTj7MnjNG+BTF8HLTnt8p21LpgLrgCVbcc5jquoO4OW0Ox2vaSVxFvBnQw77GK37NRe4GzikqpYCL6R1675NG8j9KVoHblu7cJKkcSjOBda4k7yAtkHq8m04eyYwh6orRrcoSZJGxs6XxqN5tA7Wky2aH8707rwFo16RJEkjZPjS+NMW63+H1sUaaQCb3h3/HbZxsb8kSaPBy44av9pw7ZNpW0fc2z1uaipt36+ptOC1eJhjJEnqG8OXxrc2ZPuZwLNoW6espYWwqbSbAdbRhmYvsOMlSRoLDF+aGNpg64NpO9dPp91BuAJYQtWaJztVkqR+MnxJkiT1kQvuJUmS+sjwJUmS1EeGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqI8OXJElSHxm+JEmS+sjwJUmS1EeGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqI8OXJElSHxm+JEmS+sjwJUmS1EeGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqI8OXJElSHxm+JEmS+sjwJUmS1EeGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqI8OXJElSHxm+JEmS+sjwJUmS1EeGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqI8OXJElSHxm+JEmS+sjwJUmS1EeGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqI8OXJElSHxm+JEmS+sjwJUmS1EeGL0mSpD4yfEmSJPWR4UuSJKmPDF+SJEl9ZPiSJEnqo/8PM4cnRoC4bacAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAFBCAYAAABJi8prAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xu4XWV17/HvyBUCSIAQCBASQkByodSqCGKlVZAqrdZbqtYibU/bA2jVU+qtQkOt1So9UA7QiqJ4K5CqCBgVBAqKCqKAGhMiAokhgYQAwZCEXMf5452btdhZ2Vk72ZlrX76f59nPWnuueRlri/Lzne98R2QmkiRJqsewThcgSZI0lBi+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JK0TRExOyKy+pnd7md9XMMVTdeZvKuuU11rdvVz+q68Tl+IiEVdf5dO1yKpd0Z0ugBJ6kf+sXq9Dbiig3VIGsQc+ZKkmkTEbp2uQVLnGb4k9ZlutwhfGhFfiognI+LxiPhqRBzYbf9REfF3EfHjiFgdEesi4v6I+LftXOf3Wt3y7GH7MRHxtYhYGhHrq3rujohPRcTIiDi92+27E5vOc2vTeaZFxBer82yIiBUR8ZWI+K0e/g6/W+3zFLCgaZ/jIuKaiFgeERsjYll13ORu54qI+EBE/Lr6+9weES9q4z8OSf2Utx0l7SpzgbFNv78B2Bs4CZ4dBboJOKHbcVOBNwJ/1xdFRMSY6jrjmjbvW/28oN3rRMTLgBuB3Zs271/VempEvCozv9fi0GuA/ar3T1bnmgX8FzC8ab8JwDuAP4qIl2bmwmr7P9K4HQrl73ULEO3ULan/ceRL0q7yEHA4cCSwotr2yoiYUL3/WxrBaz5wPLAHMAO4tA/rmEYjeL0P2I0Sml4GfAzYlJlXZGZzmLktM6P6+b1q26cpwWsx8EJgNCW8PVad85JtXP83lO+2O/CaKgz+ByV43Q0cVZ3r94ENlFD4SYCIGAu8vzrPeuAUSqD9NLDnDvwtJPUDjnxJ2lXOzcwHASLie5QRIoBJwCPAa5v2PTMz76jez69++soSYBPlf+/eQglB9wF3Z+aH2jlBRBxBCUlQ6v9Ji92OjogDM/PRbts/3PzdIuJkSsAC+J2qlu5Orl6PpwQ7gG9m5o1VPR8G3gWMbKd+Sf2LI1+SevJM0/sx3T5r/n1di2MXNr1f0/S+K0wc0LStr8LWVv+HMjNXAP8bWEkJO+cBVwP3R8T3IuJ5bZx3fJvX36/Ftnt24Fy7RcQe3c73cNebzFwHPN5mTZL6GcOXpJ4saXo/o9tnM7exX5eNTe9brUW1vOn9tF7Wtb7pffMThFNa7ZyZlwMHAkdTRuAuqj56GXBWG9db0fT+pqZbks/+AMMy8xctju0eTJvP9ZkezrWGEhi7HNL1JiJ2p3XQkzQAGL4k9eQmyjwkgD+IiHMi4hURcQ5l/hGUIHTzDpz7uqb3l0TEsRGxe0Q8PyL+fjvHLm56f3JE7BYRBwDv7b5jRIyLiPMpt/BWANcD1zbtcmjT+67RpEkRsU/Xxsy8H/hl9esrI+I9ETG2uu4xEXEucNV2au7yA6qJ98BpEfG2iNgzIvaIiJdExCeBC6vP76Ax+viaiHhVNVL3z3jLURqwDF+Stikzl9N40m4Y8E+UoPVPNP73Y3a1X29dBHy/ej8TuBNYS5kD1eNoVGYuA26tfv0dSmhaynODVJfdKE80fo8y2raB54bFG5red83Nmgw80W3Jir+mBKEALqAEqHXAvZRbmc23UXuqfQ3l+20BRgFfBlYDT1fXP5vyVCiZuQr41+rQ0VWtTwFnUP5WkgYgw5ekHmXmxynLRNwEPAFsrl5vAl5ffb4j530GeAXw95QJ7Gsoo2i/Ar7axin+lDKCtYoSij4PnN5ivycpYekuym28zZSw8wPgbZn59aZ93wV8k8bIVHO9t1GecvwCZf7VRsrf4WfA/wPamrxfnetKyi3Pr1IC4SbKU5M/poSt5nXOzgM+WF1zPSWknlztL2kAikzbgkmSJNXFkS9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqNKLTBUh9ImI0cAiwDzAK2AA8CSwhc0MnS5MkqZnhSwNbxFhgGjCD8s/zBmAzMJwSwjYRMQ+4j8xVHatTkqSKtx3VJyJickRkRLyoxotOAmYB038XTtobPgg8BjxRvS4FVgIzboIz+qK+iJgdJcztzDlOj4ind+YckqSBy/ClliJiQkR8PiIei4hnImJ+RJxYw3VvjYiL29hxEnAqJWit2AJbtrHnZmDFWlgF8AY4sK9q3QlXA1M6XYQkqTMMX9pKlFt53weCEnCmAe8CVnSyrmeV+k6h1LO+nUM2ltuRvBqOr47vmMxcl5nb/FtGxIiIiDprkiTVx/ClVt4HPJKZp2XmjzLzocy8OTMXtHHspIj4TkSsrUbLTm7+MCJeHhF3VqNpyyPigogYVX12BXAicFZ1izAjYnL12fSImBsRq0fBgzPhL2+B3Xv7xe6B/cfCrdX174uIV3Wr79nrRMSKiLgyIno1WhYRH4+IhRGxLiIWRcQnImK3ps+fc9ux61Zmtf0BSqDco7ffTZI0MBi+1MofA3dGxNVVALk3It7ZPBpTBYZscexHgYuAY4C7gKsiYs/qmIOBbwH3AC8A/hJ4K/Cx6th3Az8EPgdMqH6WRMQE4LvAvElwwmfhotUwbBZ8eH0ZnWvb5+ANp8Mdh8CLge8A11Z10Xwd4FjgJGDPap/e/HdlDfAXlBHDM4G3AP+wnWMOA94GvJnyt3umF9eTJA0ghi+1MoUSGh6k3N77d+DjwFlN+6wEFrY49oLMvD4z7wc+BOwL/Hb12ZnAMuDMzFyQmd8APgC8MyLGZOZTlNuDazPz0epnM3AG8NPMfP8iWPN2ePTr8G+Pw5EXwNTefLEXw7cuhHuXwFpK2FtSnZ/m61T1/Qw4jRLE2p6on5kfyczvZ+aizPwm8C+UkNmTUcCfZebdmTkvMzf15ntJkgYOl5pQK8OAH2fmB6vf74mIIyjh62KAzLy46303P2t6v6x6HV+9TgPuyMzmyfG3U4LH1G7HNnsh8PKIeHoYDBsGwxISYF4ZHbu/3S92HNwHbATGZuaWiLgTmN79Oi0OPRz4UTvXiIg3Ae+pvtOelGUvhm/nsIczc3k755ckDWyGL7XyCDC/27YFlJGi7dnY9SYzs7pT2c4Ia6tbmF2GAXOBsy+CFxwCk9fAUwAvrJ5ibPYFmHlauXW4LZuB0T1dp8VnbQWjiDgOuAo4D3gvpb7XAudv59A17ZxfkjTwGb7UyveB53fbdiSweCfPuwCYFRHDmka/Xka51fhA9fsGth4lupuyntfis2AssBdlHa+W5sHRbCN83Vm+16PA+moO27HAV7pfJzM3tjq+DScASzPzI10boiyLIUkS4JwvtXYBcFxE/ENETI2INwN/C1zStUM1Af++Xp73UuAg4NKImBYRp1Lmkl2cmWurfRYBx1aLto6rJrpfAuwNXP1mOPRGOOhsOGYGnLWwl088/ghefTYc++7ShuhCYBLwH9XHz14nIl4SEVMi4qSIuCwi9mrzEr8EDo6IP62OP4Ptz/eSJA0hhi9tJTPvojzxOIsygvRR4BxKeOoyjq1Hx7Z33qXAqylPOt4LfBa4kjIxv8v5lNGv+ZTRrUMzcxllRGnLV+Dy18C5l8EZI2DjuKbbnO14I3zxSnjlRfAN4A+A12fmw1V9z14H+DbwC0ogW0+b64ll5vXAJynB7mfAycC5valRkjS4RWZPU22kfijieMok+ZYLlb4P3vqJEupaGQ/MI/POXVWeJEk9ceRLA9ECyrywVpPmezK6Oq7VEhmSJNXC8KWBJ3MVcANlFKvdADa62v+G6nhJkjrC8KWBKXMxZVmIfSmhalvraA2vPt8XmFsdJ0lSx7jUhAauzMVEzKFM/D+a8s/zxv3LE5DjgZHAJuDnwEJHvCRJ/YHhSwNbCVR3EnEPMBEY+wt4KeVpylXAEjI3dLJESZKa+bSjBp2ImJ2ZsztdhyRJrTjnS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBoZviRJkmpk+JIkSaqR4UuSJKlGhi9JkqQaGb4kSZJqZPiSJEmqkeFLkiSpRoYvSZKkGhm+JEmSamT4kiRJqpHhS5IkqUaGL0mSpBqN6HQBkiR1XMRo4BBgH2AUsAF4ElhC5oZOlqbBx/AlSRq6IsYC04AZlH8nbgA2A8MpIWwTEfOA+8hc1bE6Nah421GSNOhExOSIyIh4UQ87TQJmAdOBlcBS4DHgiep16e/CSfvBV4FZ1f47UsvpEfH0jhyrwcnwJUnq1yJieER8JCIeiohnqtd/jogdv3tTgtSplKC1gjLatZUtsGUTbKr2O3UHA9jVwJQdLVWDj7cdJUn93fuBs4B3AD8Hfgv4PLAe+Eivz1ZuNZ5CCV3r2zxqfbX/KUTM6c0tyMxcB6zbdjkxAticmdnuOTWwOfIlServXgpcn5nXZ+aizLwOuA54SRvHHhkRt1cjZvdFxKsoc7w2A+s/DRMnwrkj4erd4ItT4exbYOw2zrW+Ou75zRsj4uMRsTAi1kXEooj4RETs1vT5c247RsTsiJhXbX+gOu8evfmDaGAzfEmS+rvbgd+PiKMAImI68Argm107VIGm1cjRJ4CLgN8GvgNcezucADx+E+zzTvj4gbD4P+HvzodzNsDus+DD6yG2UcvjwNFEjGratgb4C0qoOxN4C/AP2/lOhwFvA94MHAM8s539NYh421GS1N/9K7AXMD8iNlP+3fXRzLy0aZ+VwMIWx/5HZs4BiIh3j4Q//Ai84gZYeC68el946K5yCxOAE+D//g5ceQFM/QDc3+J8XdefCDwAkJnNtz4XRcS/AGcD5/TwnUYBf5aZy7f35TX4GL4kSf3dnwCnUUaKfkEZxfr3iHgoMy8HyMyLgYtbHPvDrjeZueXgiPmL4ECAJTB1OcwYAXOa9h8GMA8m0Dp8AWyk6dZkRLwJeA8wFdiTskzF8O18p4cNXkOX4UuS1N99Ejg/M6+qfv95lKcOPwhc3psTDYfhCQmQEBPhJ38ON66Ag9fB3nvDiukw/0ToKRhtBkYDRMRxwFXAecB7gVXAa4Hzt1PKmt7UrcHF8CVJ6u/GsPVSEJtpb97yccAtAFFmwR91PMy/HibvB5segOlj4LbXwY9PgGV7bmPJiW6G03hK8gRgafOtx9jB9cA0dDjhXpLU310PfCAiTq0WT3098H+Aa7p2iIh3RsR9LY49IyLeFBEzgS+sh4NPhLgfDj0drt8IWy6Fl/0Edp8L486GY2bAWQth9x7qGUkZ4QL4JXBwRPxpREyJiDOAt/bJt9ag5ciXJKm/exdlPa9LgfHAI8CngX9q2mcc3ZaAqPw78FHKIqcrx8DZ74UNzyvn2Lw/vO8ceMdsOG8zjNwDHjsM7hlX5nW1Mpyy6OoSgMy8PiI+CVxICWw3AudWtUothWu6abCJiNmZObvTdUiqX0QMpyzjMJMSxpYD84AFmbmm2ul4SkuhFTtwifHAPDLv7JOCNSQ58iVJGtAiYhgwiRK4plFaAc0Dbs7M1S0OWQAcTZk03+4K91T7D6f1khZS2wxfkqQBJyICOIQSuGYAqymB67LcXuufzFVE3EDp7dhui6HRlFGvub1pLSS1YviSJA0IVeCaQCNwbaAErs9l5uO9OlnmYiLmUno8bqasXN/qScfhwH7V61wyF+/wF5Aqhi9JUr8WEeMpgWtmtWke8F/Aip1qRl0C2BzK3LCjKf9O3EgJYcMpTzVuojTzXuiIl/qK4UuS1O9ExH6U0a2ZlFt+vwC+AjyyU4GruxKo7iTiHkrLoLE05oKtApaQuaHPridh+JIk9RMRMZZG4NqLEri+ASzp08DVSglYD+zSa0gVw5ckqWMiYi/Ksg8zKXOrFlDWylqcmVs6WZu0qxi+JEm1iogxNALXgcB9wHeBBzOznfY+0oBm+JIk7XKlrSJHUQLXROB+4A7gV5m5qZO1SXUzfEmSdomIGEV5knAmMBl4CLgXmJNOYtcQZviSJPWZiBgJTKUErqnArylLQ1yTmc90sjapvzB8SZJ2StVP8XBK4DqS0rR6HjA3M9d2sjapPzJ8SZJ6reqnOJkSuI4CVlIC142Z+XQHS5P6PcOXJKktVXufQymBazrwFCVwfSozn+pkbdJAYviSJG1TFbgOotFPcR0lcF2emU90sjZpoDJ8SZKeowpcB9AIXFsogeuLmflYJ2uTBgPDlyQJgIgYR6OB9QhK4JoDPLrL2/tIQ4jhS5KGsIjYh0Y/xT0o/RS/Diw1cEm7huFLkoaYiHgejcA1FpgPfBv4tf0UpV3P8CVJQ0BE7EGjn+J4Sj/FW4CHDFxSvQxfkjRIRcTuwDRK4DoI+CXwA+AB+ylKnWP4kqRBJCJG0+inOAl4APgxcH9mbuxkbZIKw5ckDXBVP8UjKYFrCrCY8qTiVzNzfSdrk7Q1w5ckDUARMYJGP8UjgKWUwHVdZq7rZG2Semb4kqQBompgfRglcD0fWEEJXN/OzDWdrE1S+wxfktSPVQ2su/opTgOepASuWzLzN52sTdKOMXxJUj9Ttfc5hLIW1wxgDSVwfSYzn+xkbZJ2nuFLkvqBKnAdSKOf4ibg58DnM3NlJ2uT1LcMX5LUQRGxP41+isMoI1xXAits7yMNToYvSapZROxLI3DtRglcXwOWGbikwc/wJUk1iIi9afRTfB6ln+I3gCUGLmloMXxJ0i4SEXvSCFz7Ufop3gQssp+iNHQZviSpD0XEGBr9FCcAC4HvAg9m5uZO1iapfzB8SdJOiojdgKMogWsi8CvgR5R+ijawlvQchi9J2gERMYpGP8XDgIeAe4E5mbmhk7VJ6t8MX5LUpqqf4hGUwHU48DDlScWvZ+YznaxN0sBh+JKkHlT9FKfQ6Kf4CCVwzc3MtZ2sTdLAZPiSpG6qfoqTKU8qTgMepwSumzJzdQdLkzQIGL4kiWfb+0ykjHBNB35DCVyfysynOlmbpMHF8CVpyKoC10E01uJ6hhK4PpuZT3SyNkmDl+FL0pBSBa7xNNr7JCVwfSkzV3SyNklDg+FL0pAQEeNojHCNogSuOcCjtveRVCfDl6RBKyLG0hjh2oPST/E64GEDl6ROMXxJGlQi4nmUCfMzgX0pgevbwK/tpyipPzB8SRrwImIPGoFrPKWf4q3AQ/ZTlNTfGL4kDUgRsTuNfooHA/cDPwAesJ+ipP7M8CVpwIiI0ZRV5mdQFkF9ELgbuCozN3awNElqm+FLUr8WESN5bj/FxZQnFb+Wmes7WZsk7QjDl6R+p2pgfThlhOtIYBklcF2fmes6WZsk7SzDl6R+oeqneBhlhOsoYAUlcN2YmU93sjZJ6kuGL0kdU602P4kywjUdWEUJXP+Tmb/pZG2StKsYviTVqgpcB1NGuGYAa4BfAJ/JzCc7WZsk1cHwJWmXqwLXATRWm99EGeH6fGau7GRtklQ3w5ekXSYi9qcRuIZRRriuApbb3kfSUGX4ktSnImJfGg2sd6cErq8BywxckmT4ktQHImJvGoFrb0o/xW9S+ikauCSpieFL0g6JiD1p9FPcH1gA3AQssoG1JG2b4UtS2yJiDDCNMsp1EPBL4HZKP0UbWEtSGwxfknoUEbtR+inOBA4FfgXcBfzKfoqS1HuGL0lbiYhRlLY+M4ApwCLgZ8B/Z+aGDpYmSQOe4UsS8Gw/xamUEa6pwMOUtbiuzcxnOlmbJA0mhi9pCIuI4ZSRrRmUW4vLKYHrW5m5ppO1SdJgZfiShpiqgfUkygjXNOBxylpcN2fm6k7WJklDgeFLGgKq9j4TKSNcM4DVlBGuyzJzVSdrk6ShxvAlDVJV4JpAo4H1Bkrg+lxmPt7J2iRpKDN8SYNMRIyn0U8RSuD6cmau6FxVkqQuhi9pEIiI/WgErlGUOVxfAR6xvY8k9S+GL2mAioixNPop7kUJXNcBDxu4JKn/MnxJA0hE7EVj0vx+lH6KNwKL7acoSQOD4Uvq5yJiD8qSEDOBA4H7gNuAh+ynKEkDj+FL6oeqfopdDawPofRTvIPST3FTJ2uTJO0cw5fUT1T9FLsaWE8GHgTuAebYT1GSBg/Dl9RBETESOIIywjUV+DVlaYhr7KcoSYOT4UuqWdVP8XDKCNeRwDLKk4pzM3NtJ2uTJO16hi+pBlU/xck0+ik+RhnhujEzn+5gaZKkmhm+pF2kau9zKCVwTQdWUUa4/jMzn+pkbZKkzjF8SX2oClwH0einuI4ywnV5Zj7RydokSf2D4UvaSVXgOoBG4NpCCVxfzMzHOlmbJKn/MXxJOygixtHopziCEriuBpbb3keStC2GL6kXImIfGiNce1DmcH0dWGrgkiS1w/ClwSFiNGUl+H3eAROJeBHwJLCEnVygNCKeR6OB9VhgPvAt4NcGLklSbxm+NLBFjKXRhmcEsGEG7Av8FjAK2ETEPOA+Mle1f9rYk/KE4gxgPKWf4i2Ufoo2sJYk7bBhnS5Ag0dEZES8aSfPcXpEtLfuVcQkYBYlJK0ElgKPfQ/2Crj8h/CbavsMYFa1f0/Xfn71Hc4F3kkZSfs+cH5mXpuZD+zK4BURs6MERUnSIObIl7YSEYuAVkHlm5l5as3ltFaC1KnACmB980eHw8qb4bQXw+pq0wpgNHAqEXPJXNw4TYym0U/xmGrzAuDrmblxx0qLW4F5mfnOHTlekjS4Gb7UyouB4U2/TwB+AszpTDndlFuNp9AieAGMhC2vKAuaNltf7X/K9RHXvBYOpIyYTQEWAT8Hfgy8j3JrcYeClyRJ2+NtR20lMx/LzEe7foDXUG7htRO+DoyIuRGxNiIWR8Tbmz+MiIMj4qqIeLL6mRsRR/SmvuPgvHFwzgj40u5wxXR410/Lk4cA3AfjA677IewFMAteOQLmnAYn7gcfey0sB/4Q+CVwYWZemZk/B7om5k+KiO9U32F+RJzc7Tu8PCLujIhnImJ5RFwQEaOqz64ATgTOqm5hZkRMrj6bXn3f1RGxIiKujIgDe/PdJUkDn+FLPaoWEP1L4EuZua5p++nNwaLJecB1wG8DlwFfiPLkIRExBvgf4BlKQDkeeAS4qfqsnYJGHwoH/BVcdjWc9X/g/GVwxJvhr7vvugmG3QYTnoApW2D0XHjTn8CVV8JH/gg+lZn3NH+nJh8FLqLchrwLuKqagE9EHEx50vEe4AXV3+atwMeqY98N/BD4HGXEcAKwJCImAN+lrAV2LHASsCdwbdX3UZI0RHjbUdtzMnAY8Olu258CFgLdb899LTM/Vb3/aET8PvAe4O3AW4AA/rxriYaI+BvK7cA/pL2RtUPmwK2UyfW8EVY8DVdcDB9eDxcOAzbDbpSTvWYCrBwJaxPig/CvZ8MDwMFvgX2AbbX7uSAzr6/q+xBwGiVM3g6cCSwDzqwm3y+IiA8An4qIczLzqYjYAKytRg2pznMG8NPMfH/TttOqGl4E/KiN7y5JGgQMX9qevwLuysyfNm/MzGuAa1rs/8MWv3dN0n8hJcitLgNqzxoDHN5mPfv8C0y5GP5mFUzcCGMShm+BEbfBPmNh2HrYD+Ak+J/XwaOz4JUBm8+Eh6pzbKSs17UtP2t6v6x6HV+9TgPu6PbU4+2UZS2mdju22QuBl2/jSc7DMXxJ0pBh+NI2RcR44HXAWX10ymHAvZQRsO7aajr9OZh4LrxnJtxwBnx5Eqy+CQ7/Ivz9ahjxKlhxMHwPOGo8rGm68MYxpeciwGbK04/b8uxoXmZmFRTbuTXY04Krw4C5wNktPlvexrklSYOE4Us9OZ3ylOCVvTjmOOCz3X5fUL2/mzI/amX2YsHTZl+FI7bAiB/AZ7rC1JfK05nPOqgxwrUtw2nxlGSbFgCzImJY0+jXyyiT9R+oft/Ac58WhfLdZwGLfZJSkoY2J/qqpWqi/f8CrsrMrW6VRcTrI+K+agJ6szdExF9FxBER8UHglcCF1WdfpozyXBsRJ0bEYdWTg//W7hOP42FeQrwOXvvfcMDb4eW3l9G53hjJ1ktRtOtS4CDg0oiYFhGnAh8HLs7MtdU+i4BjI2JyRIyrJtRfAuwNXB0RL4mIKRFxUkRcFhF77WAtkqQByPClbfk94Ai2nmjfZW/K4qQju22fDbyRMvfpDMrk+rsAqnDycuBB4L8pLXs+T5n8/mQ7RX0Wbv4TuOr78MdvgUtuhlfNeu5I2/YMBzYBS3pxzLMycynwasqTjveWkrgS+FDTbudTRr/mA48Bh2bmMuAEymjdtykNuS+hjMDt6CicJGkACvsCa8CJOJ6yQOqKHTh6PDCPzDv7tiiB8ZBPAAAB80lEQVRJktrjyJcGogWUEayeJs23Mro6bmGfVyRJUpsMXxp4ymT9GyijWO0GsNHV/jewg5P9JUnqC9521MBVmmufQlk64vHqtbvhlHW/hlOC1+IW+0iSVBvDlwa20mT7+cDRlKVTNlJC2HDKwwCbKE2zFzriJUnqDwxfGhxKY+uJlJXrR1OeIFwFLCFzQ0+HSpJUJ8OXJElSjZxwL0mSVCPDlyRJUo0MX5IkSTUyfEmSJNXI8CVJklQjw5ckSVKNDF+SJEk1MnxJkiTVyPAlSZJUI8OXJElSjQxfkiRJNTJ8SZIk1cjwJUmSVCPDlyRJUo0MX5IkSTUyfEmSJNXI8CVJklQjw5ckSVKNDF+SJEk1MnxJkiTVyPAlSZJUI8OXJElSjQxfkiRJNTJ8SZIk1cjwJUmSVCPDlyRJUo0MX5IkSTUyfEmSJNXI8CVJklQjw5ckSVKNDF+SJEk1MnxJkiTVyPAlSZJUI8OXJElSjQxfkiRJNTJ8SZIk1cjwJUmSVCPDlyRJUo0MX5IkSTUyfEmSJNXI8CVJklQjw5ckSVKNDF+SJEk1MnxJkiTVyPAlSZJUI8OXJElSjQxfkiRJNTJ8SZIk1cjwJUmSVCPDlyRJUo0MX5IkSTUyfEmSJNXI8CVJklQjw5ckSVKNDF+SJEk1+v+urJaU4ZfB+wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAFBCAYAAABJi8prAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm43VV97/H3NyOEKUgIBAgJIQxJsLZXRbhYcECo0jpVo20pxbbePgyttlIVFQqlqFVaLEXaOmJtL2AHq0gtjujFUrTWgZABBIIhAUKAYEhCxu/9Y/0Oe+fk5GSf5GTtM7xfz3Oes7P3b/juPAof1lq/9Y3MRJIkSXWM6XYBkiRJo4nhS5IkqSLDlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEkadBFxTERcGxGLIuLpiFgTEYsj4uMRcVJzzNKIyIhYuodrmRkRlzU/L9mT99pdTa3Z/NzW7Xok7Rnjul2ApJElIt4C/A0wsddHxzU/BwOvrVjSTOBP2v58W8V7S9J2HPmSNGgi4mXAJyjBK4E/A6Y3fz4WeA/wZNcK3EMiYq9u1yBp+DB8SRpMH6D1z5VrMvOSzHwoMzdm5r2Z+QHgrTs6uZka7Jl2e0kH778+Iv5fRDwWERsi4pGI+HZEvKP5/Hrgm223+JO261zWdp1fiohbI+KJiNjYTIn+dURM6VXfs1OlEfGLEfGfEbEe+Nu2Y369qeGppqZ7IuLKiJjU61r7R8QnIuLJiPhZRNwAHNLh37OkYcxpR0mDIiKmAie2vfXhvo7LzM2DdL8XAf/Etv8ReUjzMwn4iw6v8w7gql5vzwAuBM6KiJMyc2Wvzw8GvgJsM+IVEX/dnNfuGMqI3xkRcWpmro+IAP4NeGnbcW8GTuukZknDmyNfkgbLzLbXP8vM5Xv4fi+m9c+wk4EJwBHArwA3AGTmuWwbcC7PzGh+LouI6ZTROoD/oISuvShBCOAo4H193HsS8G1gFrAvcGXzIEFP8LoeOLQ57o+b914AnNe8PqOtrp8AxwPTgHs7/vaShi1HviQNVw+0vX43cDuwGLgzM7/U4TV+CRjf9vrBPo45YwfnviUzVzSv742Ic9s+O7f56etafwm8rO29v87MJQAR8afA1zopXNLw5ciXpMGytO31/hFx2CBeu6//UPw8cB2wAXgNZZrzZuDhiLi2w+tO7eCYg/p4b2Vb8NqVa7Vf86EdvJY0Qhm+JA2KZl3Ud9ve+uO+jouI/kbcN7S9bl9PNauP+2VmXgAcCLwIOBv4MjAWuCAiTu45tJ/7ta/lel/blOSzP/Qdqtbv5Fpn7+BaPWviVrUde8QOXksaoQxfkgbTe4Gtzes/aJ5SPCwixjcbr74H+Hg/57dP+/0KPLuw/nW9D4yI0yLiXZQtLO4B/gW4o+2QI5vfj7e9Nyci2vcfuxXoeQDgHc1Tj5OaJxFPi4i/Bd7V7zduaZ/qvCIiTomIiRHxnOa6/xf4jebz9icwL4yI4yLiUOCSDu8laRiLzP7+o1CSBiYifpcyHTh+B4d8ITNf2+xsPwN4MDNnNudOpqzlmtwc+zRlQfs6yuJ1gJdm5m0RcTbw2R3cYy1wXGYub/bgWgZM6XVMz3XeCfx5P1/p8sy8rKlvu5rbRcR1tBbV9+UtmXl987TjN4CX9Pp8VVud38rM3p9LGgEc+ZI0qDLzE8DPUQLYPZQpurXAEuCTwAf7OXc18CrK9OV6yqjVJcDVfRz+feDTwCLgKWALJbx8CXhJz9OWmfkM8Kbm+HV93PNDzT2/3NxvM/AI8J+UnfE/M4Dvfj5l+vNbTU2bKOu4vgm8s7kHWf6r97XAp5rjngb+lT5G+CSNPI58SZIkVeTIlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEmSJFVk+JIkSarI8CVJklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEkVGb4kSZIqMnxJkiRVZPiSJEmqyPAlSZJUkeFLkiSpIsOXJElSRYYvSZKkigxfkiRJFRm+JEmSKjJ8SZIkVWT4kiRJqsjwJUmSVJHhS5IkqSLDlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEmSJFVk+JIkSarI8CVJklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEkVGb4kSZIqMnxJkiRVZPiSJEmqaFy3C5AGRcRE4AjgQGACsBF4ElhG5sZuliZJUjvDl4a3iMnAHGAe5X/PG4EtwFhKCNtMxAJgMZmru1anJEkNpx01KCJiZkRkRLyg4k1nAPOBub8Ipx8AFwOPAU80v5cDq4B5X4PzBqO+iLgsSpjbnWucGxFP7841JEnDl+FLfYqIaRHxmYh4LCKeiYiFEXFahfveFhHXdnDgDOAsStBauRW27uDILcDKdbAa4PVw6GDVuhtuAmZ1uwhJUncYvrSdKFN53wGCEnDmAL8PrOxmXc8q9Z1JqWdDJ6dsKtORvBJObs7vmsxcn5k7/LuMiHERETVrkiTVY/hSX94JPJyZ52TmdzPzgcz8emYu6uDcGRHx1YhY14yWvaL9w4g4NSLubEbTHo2IqyNiQvPZ9cBpwAXNFGFGxMzms7kRcUtErJkA958Av/MN2HugX+wHcPBkuK25/+KIOKNXfc/eJyJWRsQNETGg0bKI+GBELImI9RGxNCI+FBF7tX2+zbRjz1Rm8/59lEC5z0C/myRpeDB8qS+vBe6MiJuaAPLDiLiwfTSmCQzZx7lXAtcAzwO+B9wYEfs25xwOfBn4AfALwO8AvwZ8oDn3bcAdwKeBac3PsoiYBnwbWDADTvkUXLMGxsyH920oo3Md+zS8/lz4ryPghcBXgS80ddF+H+BE4HRg3+aYgfx/ZS3w25QRw/OBNwPv3ck5RwG/DryR8nf3zADuJ0kaRgxf6sssSmi4nzK991fAB4EL2o5ZBSzp49yrM/PmzLwXeA/wHODnm8/OB1YA52fmosz8EvBu4MKImJSZT1GmB9dl5iPNzxbgPOBHmfmupbD2bHjk3+AvHodjr4bZA/liL4QvfwR+uAzWUcLesub6tN+nqe/HwDmUINbxQv3MvCIzv5OZSzPz34H3U0JmfyYAv5mZ/5OZCzJz80C+lyRp+HCrCfVlDPDfmXlx8+cfRMQxlPB1LUBmXtvzupcft71e0fye2vyeA/xXZrYvjr+dEjxm9zq33fOBUyPi6TEwZgyMSUiABWV07N5Ov9hJsBjYBEzOzK0RcScwt/d9+jj1aOC7ndwjIt4AvL35TvtStr0Yu5PTHsrMRzu5viRpeDN8qS8PAwt7vbeIMlK0M5t6XmRmNjOVnYyw9jWF2WMMcAtw0TXwC0fAzLXwFMDzm6cYB2gLMLG/+/TxWUfBKCJOAm4ELgf+kFLfq4GrdnLq2k6uL0ka/gxf6st3gON6vXcs8OBuXncRMD8ixrSNfr2YMtV4X/PnjWw/SvQ/lP28HrwAJgP7UfbxGrA7y/d6BNjQrGE7Efjn3vfJzE07uMTOnAIsz8wret6Isi2GJEmAa77Ut6uBkyLivRExOyLeCPwB8NGeA5oF+IsHeN3rgMOA6yJiTkScRVlLdm1mrmuOWQqc2GzaOqVZ6P5R4ADgpjfCkV+Bwy6C582DC5YM8InH78IrL4IT31baEH0EmAH8TfPxs/eJiBdFxKyIOD0iPhYR+3V4i3uAwyPiN5rzz2Pn670kSaOI4UvbyczvUZ54nE958u9K4BJKeOoxhe1Hx3Z23eXAKylPOv4Q+BRwA2Vhfo+rKKNfCymjW0dm5grKiNLWf4ZPvgou/RicNw42TWmb5uzEr8Jnb4CXXwNfAn4JeF1mPtTU9+x9gP8A7qYEsg10uJ9YZt4MfJgS7H4MvAK4dCA1SpJGtsjsb6mNNARFnExZJL8rm75OBRaQeefgFiVJUmcc+dJwtIiyLqyvRfP9mdic19cWGZIkVWH40vCTuRq4lTKK1WkAm9gcf2tzviRJXeG0o4av8hThmZStIx5vfvc2Fjio+X0rmbv7xKYkSbvF8KXhrTTJPg54LmXrlE2UEDYWGA9sBu4CljjiJUkaCgxfGhlKc+7plH3AJlKeTlwNLCNzYzdLkySpneFLkiSpIhfcS5IkVWT4kiRJqsjwJUmSVJHhS5IkqSLDlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEmSJFVk+JIkSarI8CVJklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEkVGb4kSZIqMnxJkiRVZPiSJEmqyPAlSZJUkeFLkiSpIsOXJElSRYYvSZKkigxfkiRJFRm+JEmSKjJ8SZIkVWT4kiRJqsjwJUmSVJHhS5IkqSLDlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEmSJFVk+JIkSarI8CVJklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEkVGb4kSZIqMnxJkiRVZPiSJEmqyPAlSZJUkeFLkiSpIsOXJElSRYYvSZKkigxfkiRJFRm+JEmSKjJ8SZIkVWT4kiRJqsjwJUmSVJHhS5IkqSLDlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEmSJFVk+JIkSarI8CVJklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEkVGb4kSZIqMnxJkiRVZPiSJEmqyPAlSZJUkeFLkiSpIsOXJElSRYYvSZKkigxfkiRJFRm+JEmSKjJ8SZIkVWT4kiRJqmhctwuQJKnrIiYCRwAHAhOAjcCTwDIyN3azNI08hi9J0ugVMRmYA8yj/DtxI7AFGEsJYZuJWAAsJnN11+rUiOK0oyRpxImImRGREfGCfg6aAcwH5gKrgOXAY8ATze/lvwinHwT/Asxvjt+VWs6NiKd35VyNTIYvSdKQFhFjI+KKiHggIp5pfv9ZROz67E0JUmdRgtZKymjXdrbC1s2wuTnurF0MYDcBs3a1VI08TjtKkoa6dwEXAL8F3AX8HPAZYANwxYCvVqYaz6SErg0dnrWhOf5MIj43kCnIzFwPrN9xOTEO2JKZ2ek1Nbw58iVJGur+N3BzZt6cmUsz84vAF4EXdXDusRFxezNitjgizqCs8doCbPg4TJ8Ol46Hm/aCz86Gi74Bk3dwrQ3Nece1vxkRH4yIJRGxPiKWRsSHImKvts+3mXaMiMsiYkHz/n3NdfcZyF+IhjfDlyRpqLsdeGlEHA8QEXOBlwH/3nNAE2j6Gjn6EHAN8PPAV4Ev3A6nAI9/DQ68ED54KDz4t/COq+CSjbD3fHjfBogd1PI48FwiJrS9txb4bUqoOx94M/DenXyno4BfB94IPA94ZifHawRx2lGSNNT9ObAfsDAitlD+3XVlZl7XdswqYEkf5/5NZn4OICLeNh5++Qp42a2w5FJ45XPgge+VKUwAToG//F9ww9Uw+91wbx/X67n/dOA+gMxsn/pcGhHvBy4CLunnO00AfjMzH93Zl9fIY/iSJA11bwLOoYwU3U0ZxfqriHggMz8JkJnXAtf2ce4dPS8yc+vhEQuXwqEAy2D2ozBvHHyu7fgxAAtgGn2HL4BNtE1NRsQbgLcDs4F9KdtUjN3Jd3rI4DV6Gb4kSUPdh4GrMvPG5s93RXnq8GLgkwO50FgYm5AACTEdvv8W+MpKOHw9HHAArJwLC0+D/oLRFmAiQEScBNwIXA78IbAaeDVw1U5KWTuQujWyGL4kSUPdJLbfCmILna1bPgn4BkCUVfDHnwwLb4aZB8Hm+2DuJPjWa+C/T4EV++5gy4lextJ6SvIUYHn71GPs4n5gGj1ccC9JGupuBt4dEWc1m6e+Dvgj4PM9B0TEhRGxuI9zz4uIN0TECcDfb4DDT4O4F448F27eBFuvgxd/H/a+BaZcBM+bBxcsgb37qWc8ZYQL4B7g8Ij4jYiYFRHnAb82KN9aI5YjX5Kkoe73Kft5XQdMBR4GPg78adsxU+i1BUTjr4ArKZucrpoEF/0hbNy/XGPLwfDOS+C3LoPLt8D4feCxo+AHU8q6rr6MpWy6ugwgM2+OiA8DH6EEtq8Alza1Sn0K93STJI0UETGWso3DCZQw9iiwAFiUmWubg06mtBRauQu3mAosIPPOQSlYo5IjX5KkYS0ixgAzKIFrDqUV0ALg65m5po9TFgHPpSya73SHe5rjx9L3lhZSxxz5kiQNOxERwBGUwDUPWEMJXHdnJ61/Wr0dO20xNJEy6nULmQ/uYtkSYPiSJA0TTeCaRitwbaQErgWZ+fguXHAGpcfjFsrO9X096TgWOKj5favBS4PB8CVJGtIiYiolcJ3QvLWAstnqyt1uRl2abB9HmYYcR1lov4UStsZTFtffBSwZSDNtqT+GL0nSkBMRB1FGt06gTPndTQldD+924Or7hhMoLYMm01oLthpYRubGQb+fRjXDlyRpSIgyCtUTuPajBK67gWV7JHBJXWL4kiR1TUTsR9n24QTK2qpFlBGuBzNzazdrk/YUw5ckqaqImEQrcB0KLKaMcN2fmZ2095GGNcOXJGmPK20VOZ4SuKYD91JGuH6SmZu7WZtUm+FLkrRHRFnEfhwlcM0EHqAErnvSRewaxQxfkqRBExHjgdmUwDUb+CklcC3JzGe6WZs0VBi+JEm7pemneDQlcB1LaVrd009xXTdrk4Yiw5ckacCafoozKYHreGAVJXAtzMynu1iaNOQZviRJHWna+xxJCVxzgado9VN8qpu1ScOJ4UuStENN4DqMVj/F9bQC1xPdrE0argxfkqRtNIHrEFqBayutBtaPdbM2aSQwfEmSAIiIKbQaWI+j1cD6Edv7SIPH8CVJo1hEHEirn+I+tBpYLzdwSXuG4UuSRpmI2J9W4JoMLKSErp/aT1Ha8wxfkjQKRMQ+tPopTqX0U1wAPGDgkuoyfEnSCBURewNzKIHrMOAeygjXffZTlLrH8CVJI0hETKTVT3EGcB9lhOvezNzUzdokFYYvSRrmmn6Kx1IC1yzgQVr9FDd0szZJ2zN8SdIwFBHjaPVTPAZYTglcizNzfTdrk9Q/w5ckDRNNA+ujKIHrOGAlrX6Ka7tZm6TOGb4kaQhrGlj39FOcAzxJK3D9rJu1Sdo1hi9JGmKa9j5HUPbimgespdVP8clu1iZp9xm+JGkIaALXobT6KW4G7qIErlXdrE3S4DJ8SVIXRcTBtPopjqFpYA2stL2PNDIZviSpsoh4Dq3AtRetBtYrDFzSyGf4kqQKIuIAWv0U96f0U1wALDNwSaOL4UuS9pCI2JdW4DqIVj/FpfZTlEYvw5ckDaKImESrn+I0YAklcN2fmVu6WZukocHwJUm7KSL2Ao6nBK7pwE9o9VO0gbWkbRi+JGkXRMQEWv0UjwIeoASuezJzYzdrkzS0Gb4kqUNNP8VjKIHraOAhWv0Un+lmbZKGD8OXJPWj6ac4i1Y/xYcpgWtRZq7rZm2ShifDlyT10vRTnEl5UnEO8DitfopruliapBHA8CVJPNveZzplhGsu8DNa/RSf6mZtkkYWw5ekUasJXIfR2ovrGZr2Ppn5RDdrkzRyGb4kjSpN4JpKq71P0gpcK7tZm6TRwfAlaVSIiCm0Rrgm0Gpg/YjtfSTVZPiSNGJFxGRaI1z70Oqn+JCBS1K3GL4kjSgRsT9lwfwJwHNoBa6f2k9R0lBg+JI07EXEPrQC11Ra/RQfsJ+ipKHG8CVpWIqIvWn1UzwcuJcSuO6zn6KkoczwJWnYiIiJlF3m51E2Qb2fVj/FTV0sTZI6ZviSNKRFxHi27af4ICVwLcnMDd2sTZJ2heFL0pDTNLA+mjLCdSywglY/xfXdrE2SdpfhS9KQ0PRTPIoywnU8sJJW4Hq6m7VJ0mAyfEnqmma3+RmUEa65wGpa/RR/1s3aJGlPMXxJqqoJXIdTRrjmAWuBuyntfZ7sZm2SVIPhS9Ie1wSuQ2jtNr+ZVj/FVd2sTZJqM3xJ2mMi4mBagWsMzQgX8KjtfSSNVoYvSYMqIp5Dq4H13rQC1woDlyQZviQNgog4gFbgOoBt+yn6DxlJamP4krRLImJfWv0UDwYWUQLXUhtYS9KOGb4kdSwiJgFzKKNchwH30OqnaANrSeqA4UtSvyJiL0o/xROAI4GfUALXT+ynKEkDZ/iStJ2ImEBp6zMPmAUspdVPcWMXS5OkYc/wJQl4tp/ibMoI12zgIUrgWpyZz3SzNkkaSQxf0igWEWMpI1vzKFOLj9Lqp7i2m7VJ0khl+JJGmaaB9QzKCNcc4HHKXlx3Z+aabtYmSaOB4UsaBZr2PtMpI1zzgDW0Gliv7mZtkjTaGL6kEaoJXNNoNbDeSKuf4uPdrE2SRjPDlzTCRMRUWv0UoRW4VnavKklSD8OXNAJExEG0AtcEWv0UH7a9jyQNLYYvaZiKiMm0+inuRytwPWTgkqShy/AlDSMRsR+tRfMH0eqn+KD9FCVpeDB8SUNcROxD2RLiBOBQYDElcD1gP0VJGn4MX9IQ1PRT7GlgfQTb9lPc3M3aJEm7x/AlDRFNP8WeBtYzgfspgete+ylK0shh+JK6KCLGA8dQRrhmAz+l1cDafoqSNAIZvqTKmn6KR1NGuI4FVlCeVFyUmeu6WZskac8zfEkVNP0UZ9Lqp/gYZYRrYWY+3cXSJEmVGb6kPaRp73MkJXDNBVbTamD9VDdrkyR1j+FLGkRN4DqMVj/F9bQaWD/RzdokSUOD4UvaTU3gOoRW4NpKq5/iY92sTZI09Bi+pF0UEVNo9VMcRxO4gEdt7yNJ2hHDlzQAEXEgrRGufWj1U1xu4JIkdcLwpZEhYiJlJ/gDgQnARuBJYBm7uUFpROxPq4H1ZGAhJXD91MAlSRoow5eGt4jJtNrwjKOEri3AWEoI20wJSovJXN35ZWNfyhOK84CpbNtP0QbWkqRdNqbbBWjkiIiMiDfs5jXOjYjO9r2KmAHMp4SkVcBy4LGL4bCAT94BP2venwfMb47v797HNd/hUuBCykjad4CrMvMLmXnfngxeEXFZRCzYU9eXJA0N47pdgIaeiFgK9BVU/j0zz6pcTt9KkDoLWAlsaP/oIlj8CjjnhbCmeWslMBE4i4hbyHywdZmYSKuf4vOatxcB/5aZm3attLiN8qTjhbtyviRpZDN8qS8vpEzb9ZgGfB/4XHfK6aVMNZ5JH8EL4CDY/LKyoWm7Dc3xZ94c8flXw6GUEbNZwFLgLuC/gXdSphZ3KXhJkrQzTjtqO5n5WGY+0vMDvIoyhddJ+Do0Im6JiHUR8WBEnN3+YUQcHhE3RsSTzc8tEXHMQOo7CS6fApeMg3/YG66fC7//o/LkIQAXwwkBX7wD9gOYDy8fB587B047CD7wangU+GXgHuAjmXlDZt5FWS8GMCMivtp8h4UR8Ype3+HUiLgzIp6JiEcj4uqImNB8dj1wGnBBM4WZETGz+Wxu833XRMTKiLghIg4dyHeXJA1/hi/1q9lA9HeAf8jM9W3vn9seLNpcDnwR+HngY8DfR8QLmnMmAd8EnqEElJOBh4GvNZ91UtDEI+GQt8LHboIL/giuWgHHvBH+T+9DN8OYb8G0J2DWVph4C7zhTXDDDXDFr8DfZeYP2r9TmyuBayjTkN8DbmwW4BMRhwNfBn4A/ELzd/NrwAeac98G3AF8mjJiOA1YFhHTgG9TFu2fCJwO7At8oen7KEkaJZx21M68AjgK+Hiv958ClgC9p+f+NTP/rnl9ZUS8FHg7cDbwZiCAt/Rs0RARv0eZDvxlOhtZO+JzcBtlcT2/CiufhuuvhfdtgI+MKYUdSLnYq6bBqvGwLiEuhj+/CO4DDn9zOWZH7X6uzsybm/reA5xDCZO3A+cDK4Dzm8X3iyLi3cDfRcQlmflURGwE1jWjhjTXOQ/4UWa+q+29c5oaXgB8t4PvLkkaAQxf2pm3At/LzB+1v5mZnwc+38fxd/Tx555F+s+nBLk1ZUDtWZOAozus58D3w6xr4fdWw/RNMClh7FYY9y04cDKMWQnHAJwO33wNPDIfXh6w5Xx4oLnGJsp+XTvy47bXK5rfU5vfc4D/6vXU4+2UbS1m9zq33fOBU3fwJOfRGL4kadQwfGmHImIq8BrggkG65Bjgh5QRsN46ajr9aZh+Kbz9BLj1PPjHGbDma3D0Z+GP18C4M2DlMSXIvHYqrG278aZJpecilH3AJvZzm2dH8zIzm6DYydRgf5vmjQFuAS7q47NHO7i2JGmEMHypP+dSnhK8YQDnnAR8qtefFzWv/4eyPmpVDmDD03b/AsdshXH/CZ/oCVP/UJ7OHIix9PGUZIcWAfMjYkzb6NeLKYv172v+vJFtnxaF8t3nAw/6JKUkjW4u9FWfmoX2vwvcmJnbTZVFxOsiYnGzAL3d6yPirRFxTERcDLwc+Ejz2T9SRnm+EBGnRcRRzZODf9HpE49TYUFCvAZe/U9wyNlw6u1ldG4gxrP9VhSdug44DLguIuZExFnAB4FrM3Ndc8xS4MSImBkRU5oF9R8FDgBuiogXRcSsiDg9Ij4WEfvtYi2SpGHI8KUdeQll7VTvhfY9DqBsTjq+1/uXAb9KWft0HmVx/fcAmnByKnA/8E+Ulj2foSx+f7KToj4FX38T3PgdeO2b4aNfhzPmbzvStjNjKS2Hlg3gnGdl5nLglZQnHX9YSuIG4D1th11FGf1aCDwGHJmZK4BTKKN1/0FpyP1Rygjcro7CSZKGIXs7aviJOJmyQerKXTh7KrCAzDsHtyhJkjrjyJeGo0WUEaz+Fs33ZWJz3pJBr0iSpA4ZvjT8lMX6t1JGsToNYBOb429lFxf7S5I0GJx21PBVmmufSdk64vHmd29jgYOa37e2N9WWJKkbDF8a3kqT7eOA51K2TtlECWFjKQ8DbKY0zV7iiJckaSgwfGlkKI2tp1N2rp9IeYJwNbCMzI39nSpJUk2GL0mSpIpccC9JklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEkVGb4kSZIqMnxJkiRVZPiSJEmqyPAlSZJUkeFLkiSpIsOXJElSRYYvSZKkigxfkiRJFRm+JEmSKjJ8SZIkVWT4kiRJqsjwJUmSVJHhS5IkqSLDlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEmSJFVk+JIkSarI8CVJklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEmr+ZROAAAAp0lEQVQVGb4kSZIqMnxJkiRVZPiSJEmqyPAlSZJUkeFLkiSpIsOXJElSRYYvSZKkigxfkiRJFRm+JEmSKjJ8SZIkVWT4kiRJqsjwJUmSVJHhS5IkqSLDlyRJUkWGL0mSpIoMX5IkSRUZviRJkioyfEmSJFVk+JIkSarI8CVJklSR4UuSJKkiw5ckSVJFhi9JkqSKDF+SJEkVGb4kSZIqMnxJkiRV9P8BKaBYZKIn21cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bad_diff_all_ids = {6, 7, 8}\n",
    "show_cluster_graphs(\n",
    "    df,\n",
    "    golden_pairs_set, dedupe_found_pairs_set, dedupe_unclustered_found_pairs_set,\n",
    "    bad_diff_all_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Remember the `recall_weight` parameter of the `deduper.threshold` method? That's what it controls: how aggressive we want to be on finding or dropping matches while clustering. Check Dedupe docs for more details on how it [performs clustering](https://docs.dedupe.io/en/latest/Grouping-duplicates.html) and how it computes [a good threshold](https://docs.dedupe.io/en/latest/Choosing-a-good-threshold.html#)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Finally, it's worth mentioning there's a web-based [product version of Dedupe](https://dedupe.io/). If you don't want to write code for deduping a dataset, check it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src=\"https://dedupe.io/images/tutorials/getting-started-with-dedupe-io/img-5.jpg\" width=800></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Once we have the clusters, how to consolidate data from many records into one? Check for material on **Data Fusion**:\n",
    "  - [Open issue at Python Record Linkage Toolkit](https://github.com/J535D165/recordlinkage/issues/27)\n",
    "  - Christen, 6.12 Merging Matches [2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What if new records arrive? Should we merge, unmerge, move records from clusters? Check for material on **Incremental Record Linkage**:\n",
    "  - Dedupe approach is to either add to an existing cluster or create a new cluster. You can use a [Gazetteer](https://docs.dedupe.io/en/latest/API-documentation.html#gazetteer-objects) with previously deduped data, then merge new matches into it, and index new nonmatches.\n",
    "  - Other approaches, check papers:\n",
    "      * [Online Correlation Clustering](https://arxiv.org/pdf/1001.0920.pdf)\n",
    "      * [Incremental Record Linkage](http://www.vldb.org/pvldb/vol7/p697-gruenheid.pdf)\n",
    "      * [Heuristic-based approaches for speeding up incremental record linkage](https://www.sciencedirect.com/science/article/pii/S0164121217302972)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Also worth checking the **Privacy** implications of Record Linkage:\n",
    "- Christen, chapter 8 [2]\n",
    "- [k-anonymity](https://desfontain.es/privacy/k-anonymity.html), [k-map](https://desfontain.es/privacy/k-map.html), [δ-presence](https://desfontain.es/privacy/delta-presence.html)\n",
    "- [Why differential privacy is awesome](https://desfontain.es/privacy/differential-privacy-awesomeness.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Referenced by this talk**:\n",
    "    * [1] [Mike Mull, The Art and Science of Data Matching](https://www.youtube.com/watch?v=Y-nYEOgq3YE) (slides [here](https://github.com/mikemull/Notebooks/blob/master/PyDataNYCSlides.ipynb))\n",
    "    * [2] [Peter Christen, Data Matching](http://www.springer.com/us/book/9783642311635)\n",
    "    * [3] [Mikhail Yuryevich Bilenko, Learnable Similarity Functions and Their Application to Record Linkage and Clustering](http://www.cs.utexas.edu/~ml/papers/marlin-dissertation-06.pdf)\n",
    "- Other **talks** about Record Linkage with Python:\n",
    "    * [Andrew Rowe, Big Data Deduplication and data matching using Python](https://www.youtube.com/watch?v=Z6mlvrYEYnk)\n",
    "    * [Rhydwyn Mcguire, Record linkage: Join for real life](https://www.youtube.com/watch?v=cEcVIjyHfiQ)\n",
    "- [Basics of Entity Resolution with Python and Dedupe](https://medium.com/district-data-labs/basics-of-entity-resolution-with-python-and-dedupe-bc87440b64d4)\n",
    "- [Automating your Data Cleanup with Python](https://blog.kjamistan.com/automating-your-data-cleanup-with-python/)\n",
    "- [Comparison of Data Matching software](https://github.com/J535D165/data-matching-software)\n",
    "- [Compilation of Record Linkage resources](https://github.com/ropeladder/record-linkage-resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Thank you!\n",
    "flavio@vinta.com.br  \n",
    "[@flaviojuvenal](https://twitter.com/flaviojuvenal)  \n",
    "[vinta.com.br](https://www.vinta.com.br?dedupe-slides=1)\n",
    "\n",
    "Special thanks to Russell Keith-Magee [@freakboy3742](https://twitter.com/freakboy3742), Forest Timothy Gregg [@forestgregg](https://twitter.com/forestgregg), and Jonathan de Bruin [@J535D165](https://github.com/J535D165)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}