5SSG2060_Practical_1_Web Mapping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a Simple Interactive Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red;\">Important Note for Mac Users</span>\n",
    "\n",
    "Recent changes in the way that the Mac OS handles the plotting of data means that you need to do certain things in a specific order at the start of any notebook in which you intend to show maps or graphs. Please make a copy of the following code for any notebook that you create and make it the _first_ code that you run in the notebook:\n",
    "\n",
    "```python\n",
    "# Needed on a Mac\n",
    "import matplotlib as mpl\n",
    "mpl.use('TkAgg')\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "```\n",
    "\n",
    "For non-Mac users it _should_ be:\n",
    "\n",
    "```python\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "```\n",
    "\n",
    "This _should_ enable you to create plots, including in the practical that we're about to start!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Web Maps\n",
    "\n",
    "Simple interactive web maps in Python are often made with a library called `folium`. These are not maps in the GIS sense because they can _only_ be viewed in a web browser, and sometimes even then only after disabling safety precautions in the browser itself (see notes at the end of this notebook). But in a Jupyter notebook context, `folium` works really well because it's just embedding more web content into a running web page. Simples!\n",
    "\n",
    "As always, if you get stuck you may want to RTM: [Latest folium documentation](https://media.readthedocs.org/pdf/folium/latest/folium.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting started\n",
    "\n",
    "The mapping that we'll do this week is covered by just three libraries. You would then make sure that you were running the right 'kernel' (gsa2018) by picking 'Change Kernel' from the Kernel drop-down above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import folium  \n",
    "import pandas as pd\n",
    "import os\n",
    "print (folium.__version__)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a map\n",
    "\n",
    "Creating a map with Folium is dead simple, that's why we're starting this term with web maps and not more 'GIS-style' or 'data science-style' mapping.\n",
    "\n",
    "Use the coding block below to create a map centered on 45<sup>o</sup> latitude and 0<sup>o</sup> longitude at zoom level 6. You may need to use `help(folium.Map)` to do this. You can do this with one line of code.\n",
    "\n",
    "What country is shown?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "folium.Map(location=(45,0), zoom_start=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer here for the country's name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a map with 'coordinate popup’\n",
    "\n",
    "Let's reposition the map: the location of Birmingham is (roughly) 52.4381<sup>o</sup>N and 1.8936<sup>o</sup>W. Replace the `???` below so that the map you create will be centred on Birmingham. Notice that now we are using a _parameter_ for the location instead of hard-coding it. That will allow us to make a more _general_ example that could work for _any_ location, or to update and re-run the code quickly and easily.\n",
    "\n",
    "Adding the 'coordinate popup' requires one more line of code, and notice that now we need to have a line that 'prints' the map (this the line with the simple `m`). This is because we now 'capture' the output of folium.Map so that we can add the popup functionality. In the example above, because we don't assign the output of `folium.Map` to a variable we didn't need to write `m` to 'print' it to the page.\n",
    "\n",
    "*Hint: To learn more about what you can do with children try using `help(folium.Map.add_children)` or searching online via Google.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAP_COORDINATES = (???,???) # Create a _tuple_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = folium.Map(location=MAP_COORDINATES, zoom_start=13) # Create the map\n",
    "m.add_children(folium.LatLngPopup()) # Add the popup functionality, try it!!!\n",
    "m.save(\"Map for Birmingham.html\") # Now the map for Birmingham has been saved in your directory as a html file, try to open it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see a map centred on the city of Birmingham. **If you see a map centred somewhere off of the East coast of the UK (near Lowestoft) then you need to think about what 1.8936°W means in terms of x and y coordinates on a graph.**\n",
    "\n",
    "Try clicking somewhere on the map, what happens?\n",
    "\n",
    "### Changing the Tile 'Style'\n",
    "\n",
    "Copy the folium code above into the coding cell below, and then change the map style to one of the other avilable types listed through the `help()` documentation or Google (e.g. the 'Stamen Toner' style). There are lots of examples [here](http://folium.readthedocs.io/en/latest/quickstart.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here!\n",
    "???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customising Maps\n",
    "\n",
    "We've seen how you can can manipulate the basic 'setup' of a map, now we're going to start adding features (i.e. 'stuff') to the map to customise it slightly.\n",
    "\n",
    "### Adding a Marker\n",
    "\n",
    "Drag the map that you created above until it is centred on Central London. Now click on the Strand Campus and make a note of the coordinates in the popup (they should be in the vicinity of 51.5<sup>o</sup> latitude and -0.1<sup>o</sup> longitude). We want to set this as the new map centre by creating a new variable containing those coordinates below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAP_COORDINATES = (???, ???)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to add a marker showing where the Geography Department is on our map. Note that, because we use the same coordinates for centering the map _and_ for the popup, that means that the popup should be in the middle of map!\n",
    "\n",
    "To do this, you will want to investigate how `folium.Marker` works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = folium.Map(location=???, zoom_start=18, tiles=\"Stamen Toner\") \n",
    "m.add_children(folium.LatLngPopup())\n",
    "# And now add a marker\n",
    "folium.Marker(???, \n",
    "              popup='Geography'\n",
    "             ).add_to(m)\n",
    "m.save(\"Marker Map for Strand.html\") # And now print the map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding More Markers\n",
    "\n",
    "Now add the rest of the main King’s buildings on the North side of the Thames (Virginia Woolf, Bush House, Maughan Library and Somerset House East Wing) to this map using [a few different marker styles](http://fontawesome.io/icons/)! *Note:* not all of these icons worked when I tested this, but many do. The coordinates I have are:\n",
    "\n",
    "| Location | Longitude | Latitude |\n",
    "| -------- | --------- | ------ |\n",
    "| Strand Campus   | -0.1160   | 51.5113  |\n",
    "| Virginia Woolf | -0.1180   | 51.5147  |\n",
    "| Maughan Library | -0.1104   | 51.5153  |\n",
    "| Somerset House | -0.1172   | 51.5127  |\n",
    "| Bush House  | -0.1166   | 51.5112  |\n",
    "\n",
    "As a hint, you'll need to use the `icon` parameter in your code. To see how this works you might want to `Insert` -> `Cell Below` so that you can type `help(folium.Icon)` and view the output... or you can also look at some examples at [read the docs](http://folium.readthedocs.io/en/latest/quickstart.html).\n",
    "\n",
    "I've created a simple data structure for you that has all of the location data you will need. Can you think how you might use a `for` loop and the dictionary-of-dictionaries data structure to make the process of adding the markers less tedious?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "locs = {\n",
    "    'Strand Campus':  {'lat': 51.5113, 'lon': -0.1160, 'icon': 'plane'},\n",
    "    'Virginia Woolf': {'lat': 51.5147, 'lon': -0.1180, 'icon': 'shopping-cart'},\n",
    "    'Maughan Library': {'lat': 51.5153, 'lon': -0.1104, 'icon': 'cog'},\n",
    "    'Somerset House': {'lat': 51.5127, 'lon': -0.1172, 'icon': 'cloud'},\n",
    "    'Bush House': {'lat': 51.5112, 'lon': -0.1166, 'icon': 'info-sign'}\n",
    "}\n",
    "\n",
    "m = folium.Map(location=MAP_COORDINATES, zoom_start=15, tiles=\"Stamen Toner\")\n",
    "\n",
    "for loc, vals in locs.items():\n",
    "    # And now add a marker\n",
    "    folium.Marker( (vals['lat'], ???), icon=folium.Icon(color='green', icon=???),  \n",
    "              popup=???\n",
    "             ).add_to(m)\n",
    "m.save(\"More Markers Map for Strand.html\") # And now print the map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Data with Folium\n",
    "\n",
    "OK, so that was fairly straightforward: \n",
    "1. Pick a lat/long location for the centre of the map.\n",
    "2. Pick a zoom level\n",
    "3. Add some popup info\n",
    "4. Show map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Choropleth Map\n",
    "\n",
    "For polygon data we can create what is called a _choropleth_ map: that's the technical term for a map that uses colour to represent data values. Perhaps you'd like to take a look at the help for this method or search Google for `Folium choropleth`)? \n",
    "\n",
    "If you search online you will need to make sure that you are looking at the documentation for how to show a choropleth for version 0.2 or higher since version 0.1 used a very different approach..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(folium.Map.choropleth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using GeoJSON files\n",
    "\n",
    "To create our choropleth we'll actually be overlaying data rather than just adding some markers. Folium can't read Shapefiles, but it _can_ read simple GeoJSON and show it on a map for you. If the GeoJSON file contains polygons, then we can turn our output into a basic _choropleth_ map.\n",
    "\n",
    "#### Using JSON data\n",
    "\n",
    "In theory, Folium actually has two ways to pull in the GeoJSON data that we want: 1) directly from the URL (using `geo_path`); 2) as a 'resource' (using `geo_str`) in which case you need to give Folium the raw data and not just a location. \n",
    "\n",
    "Unfortunately, in practice method \\#1 doesn't work quite the way I think it should (I've even submitted a bug to that effect) so we are going to use the second approach: although it's a little bit harder (_one_ extra line of code!) it gives us more flexibility later because we can also examine the JSON data directly rather than just wondering why our map isn't working. \n",
    "\n",
    "For the record, the second approach is the same as what we used in Term 1 with the MetOffice API. The key thing to remember is that your request to the JSON URL will have received a _response_ (saved in `r`) but that you need to tell the `response` object to treat the raw response as JSON data using a _method_. The way to do this will be the same as that time! \n",
    "\n",
    "#### Finding JSON data\n",
    "\n",
    "You can find JSON data all over the place, although GeoJSON is a little more rare. One good example that I could find is the GeoJSON world map available from https://raw.githubusercontent.com/datasets/geo-boundaries-world-110m/master/countries.geojson into your map! To find out more about this file have a look at: http://data.okfn.org/data/datasets/geo-boundaries-world-110m.\n",
    "\n",
    "Let's start be downloading the GeoJSON file I've just suggested..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "jsonURL = ???\n",
    "r = requests.get(???) \n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a map, you will need to tell Folium about the polygon data and you will also need to: \n",
    "1. Set the fill colour to none.\n",
    "2. Set the outline colour to red.\n",
    "\n",
    "I've filled in reasonable defaults for the rest of the parameters but feel free to play with them to see what effect this has!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(jsonURL)\n",
    "m = folium.Map(location=MAP_COORDINATES, zoom_start=5)\n",
    "m.choropleth (\n",
    "    geo_data= ???.json(), # This needs to be the JSON response to your request...\n",
    "    fill_color=???, line_color=???, fill_opacity=0.0, line_opacity=0.25, line_weight=2) \n",
    "m.save(\"Countries boundaries map.html\") # And now print the map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see an outline of the world with simplified borders -- this is a common 'problem' in geodata: if they were to provide us with high-resolution data then the JSON file would be much, much larger (Gigabytes, instead of Kilobytes). Remember this downsampling 'trick' when working with geodata!\n",
    "\n",
    "### Exploring the JSON file\n",
    "\n",
    "What _properties_ are available for each country in the GeoJSON file? There's a clue to how to find the answer in the question... I'll give you some starter code to help you skip over the most tedious stuff..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_country_by_index(json, cntry=0):\n",
    "    return json['features'][cntry]\n",
    "    \n",
    "print(get_country_by_index(r.json()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the long section of floating-point numbers? Why are they provided in pairs? What do you think they represent and what evidence to do you have for thinking this?\n",
    "\n",
    "And now, using what you've understood from the previous code:\n",
    "1. What country has index 20? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hint for you\n",
    "print(get_country_by_index(r.json(), 20)['properties'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then could you do it by yourself-\n",
    "\n",
    "2. How would you change the code to print only that country's abbreviated name?\n",
    "3. How about it's ISO-3 abbreviation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "???"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Joining Data to GeoJSON\n",
    "\n",
    "So now we've been able to add a world map on top of Folium, but that's not _hugely_ useful unless we can also link it to some data... \n",
    "\n",
    "Helpfully, the World Bank publishes a lot of data about countries here: http://data.worldbank.org/indicator/SP.POP.TOTL. Click on the 'Download' link and save the CSV file to the _data_ directory for the Spatial Analysis module (it will be a directory called 'data' one level _lower_ than _this_ Notebook).\n",
    "\n",
    "Here's a really useful trick for reading a CSV file out of a Zip archive that contains multiple files! We will start by just opening the Zip file and printing out what files are inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import library to read Zip files\n",
    "import zipfile \n",
    "import os\n",
    "\n",
    "# Load the Zip file\n",
    "z = zipfile.ZipFile(os.path.join('data','API_SP.POP.TOTL_DS2_en_csv_v2.zip'))\n",
    "\n",
    "# Show what files are contained inside\n",
    "print(z.namelist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you understand what's happening above. Add comments!\n",
    "\n",
    "When you want to read the CSV file from the Zip archive then you use `z.open(<filename>)`. So, you will need to pass `z.open...` to pandas' `read_csv` command and _then_ specify _both_ `skiprows` and `header` in order to parse the file properly -- while you try to work out what to do you might wants to unzip a copy of this file so that you can view the file directly (but you can then delete it once you've made sense of the headers so that it takes up less space).\n",
    "\n",
    "Although this process might seem a little convoluted compared to just unzipping the file and reading the CSV file directly into pandas, let's take note of what this allows us to do: we no longer need to extract files from archives in order to read them; we can save a lot of disk space (since tabular data tends to compress well); and we can now do this for _any_ Zip archive, including those that contain multiple files inside!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the CSV file extracted from the \n",
    "# Zip archive in the previous code block\n",
    "import pandas as pd\n",
    "df = pd.read_csv(z.open('API_SP.POP.TOTL_DS2_en_csv_v2.csv'), skiprows=[0], header=1)\n",
    "\n",
    "# Drop unneeded columns from the data frame\n",
    "df.drop(['Unnamed: 62','2017','Indicator Code','Indicator Name'], inplace=True, axis=1)\n",
    "\n",
    "# Change the names of the first two columns so that \n",
    "# they don't have spaces in them any more...\n",
    "new_cols = df.columns.values.tolist()\n",
    "new_cols[0] = 'Name'\n",
    "new_cols[1] = 'Code'\n",
    "df.columns = new_cols\n",
    "\n",
    "# Check it worked\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This sets a useful number format for all summaries\n",
    "pd.set_option('display.float_format', lambda x: '{:,.0f}'.format(x))\n",
    "\n",
    "# Summarise\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binning the Data\n",
    "\n",
    "I can tell simply from looking at the summary above that we're dealing with a pretty massively skewed data set... how can I tell that? What are the clues?\n",
    "\n",
    "Let's check that by making a plot of the data from 1960:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['2016'].plot.hist(bins=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning\n",
    "\n",
    "In fact, some of thsose counts are clearly impossible if _all_ this data file contains is country populations. Let's look a little more closely at the data... We know that the largest value in the data set is 7.44 _billion_. That seems a tad large for any one country and looks suspiciously like the world population. \n",
    "\n",
    "So how could we see if there's anything unusual included in the data beyond just country data? Let's think in terms of what we want from the data: the largest _legitimate_ value in the data should be the population of China, so let's start there by extracting the value of China's population in 2016 from the dataframe `df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c_pop = df.loc[df.Name=='China','2016'].values[0]\n",
    "print(\"China's population in 2016 was: \" + str(c_pop))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use this value to find out what rows have populations that are greater than or equal to what we _expected_ to be the maximum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[???][['Name','Code','2016']] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Interesting..._ So it looks like the World Bank has also created some aggregate values for us. If we were going to do some serious analysis then we wouldn't really want to keep these, and we'd also have to check if there were other aggregates that whose values _less_ than the population of China. For instance, perhaps there's a 'North America' row, or a 'Middle East' row? We'll not worry about the latter problem in this exercise, but let's get rid of the huge numbers that are pulling our distribution out of whack.\n",
    "\n",
    "For this we need to combine several steps in a complex way, but you can make sense of it by working from the inside out:\n",
    "\n",
    "1. `df['2016'] > c_pop]` selects the rows where the 2016 population is greater than China's population in 2016.\n",
    "2. `df.index[ <step 1> ]` tells pandas to use this boolean result set (the long True/False series) on the index, so it will only select those rows where this is `True`.\n",
    "3. `dr.drop( <step 2>, inplace=True)` then tells pandas to drop the rows selected by applying `True` values to the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop( df.index[df['2016'] > c_pop], inplace=True )\n",
    "df[???][['Name','Code','2016']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even after doing this, the data is still going to be skewed and it _might_ be useful to work with quantiles or rank. Why is that? What do those two approaches allow us to do when making a map?\n",
    "\n",
    "### Rank\n",
    "\n",
    "Let's start with rank. We want the population rank in 2016 as our legend for the map so let's calculate that first. This is so straightforward that I'm not going to help you: you can just search online for help to do with series rank and the only 'trick' is to remember that we probably want China to have rank 1 (because it's biggest) and Tuvalu to have rank 243 (because it's smallest)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['r2016'] = df['2016'].??? #check out the rank() function\n",
    "df[ (df.r2016==1) | (df.r2016==df.r2016.max()) ][['Name','2016','r2016']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantiles\n",
    "\n",
    "Do you remember how to get a list of quantiles from a pandas data frame? No? Well you’ll have to go look at your old Geocomputation code (or use `help(<data frame>.<column name>.quantile)`)! Let’s get the following quantiles: 0th, 20th, 40th, 60th, 80th, and 100th for 2016.\n",
    "\n",
    "*Hint*: you should get the following numbers...\n",
    "- Quantile: 11,097\n",
    "- Quantile: 586,261\n",
    "- Quantile: 4,964,693\n",
    "- Quantile: 15,622,068\n",
    "- Quantile: 59,662,245\n",
    "- Quantile: 1,378,665,000\n",
    "\n",
    "Let's calculate these first and then figure out how to create a quantiles column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantiles = df['2016'].???.values.tolist() # check out the quantile() function\n",
    "for q in quantiles:\n",
    "    print(\"Quantile: \" + '{:,.0f}'.format(q))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['q2016'] = pd.cut(df['2016'], bins=quantiles, labels=['Quantile ' + str(x) for x in range(1, len(quantiles))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you think that just did? Maybe Google search for the answer or output just some key fields?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[['Name','Code','2016','r2016','q2016']].sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepping the Data\n",
    "\n",
    "Unlike the tool we'll see next week, Folium requires that there be no rows in _one_ data set that have **no match** in the _other_ data set. So if we want to join World Bank population data to the GeoJSON world map data then we need to have the same number of rows in both data sets _and_ we need to have a match on every single row in the _join column_ (much more on this in the next two weeks). In plainer English: if we want to create our map then every country code in the GeoJSON data set must have a matching country code in the pandas data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to find all of the keys that are\n",
    "# 'in' the GeoJSON file but are 'missing'\n",
    "# from the pandas data frame. The best way\n",
    "# to do this is to extract a list from the \n",
    "# JSON file and then compare it the codes.\n",
    "def get_country_code(f):\n",
    "    return f['properties']['iso_a3']\n",
    "\n",
    "my_json_codes = [] # Get the country codes from the JSON file\n",
    "for c in r.json()['features']:\n",
    "    my_json_codes.append(get_country_code(c))\n",
    "    \n",
    "my_df_codes = df.Code.values.tolist() # Get the country codes from the data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can use this block to see what \n",
    "# these new lists contain...\n",
    "print(my_json_codes)\n",
    "\n",
    "print(\" \")\n",
    "\n",
    "print(my_df_codes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we're going to make use of a new type of Python class: the set. A set is like an _unordered_ list, which might seem less than useful but it has very specific benefits for what we are doing: we can ask Python to quickly tell us whether something is _in_ the set or not without needing to write a `for` loop and step through the list one item at a time. So using two sets we can quickly find out their _intersection_ or their _dijoint_ (what's in one set, but not the other)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the lists to sets...\n",
    "js = set(my_json_codes)\n",
    "ds = set(my_df_codes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rows Missing from World Bank Data\n",
    "\n",
    "To find out what's in the JSON data but missing from the World Bank data we use the 'difference' operation and then add these rows to the `df`..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(js.difference)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And find out what's in the JSON\n",
    "# file that is *not* in the data frame\n",
    "missing = js.difference(ds)\n",
    "print(\"Missing rows from data frame: \" + \", \".join(missing))\n",
    "print(type(missing))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a simple data frame from the missing country codes\n",
    "d = {'Name': list(missing), 'Code': list(missing)}\n",
    "dfm = pd.DataFrame.from_dict(d)\n",
    "dfm # dfm = data frame missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Append Missing Data Frame Records\n",
    "\n",
    "Now we can just append the missing values to the data frame and the years will be filled in with NaNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfn = df.append(dfm, ignore_index=True) # dfn = data frame new\n",
    "dfn.tail(8)[['Name','Code','2000','2016']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Removing 'Extra' Rows from World Bank Data\n",
    "\n",
    "We now need to do the 'reverse' operation: finding and removing the data that is in the World Bank data set but which has _no match_ in the GeoJSON file. So this is just the same `difference` operation, but with the positions reversed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extraneous = ds.difference(js)\n",
    "print(sorted(list(extraneous)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And delete using the index labels as we did \n",
    "# above when we removed the values larger than\n",
    "# China...\n",
    "dft = dfn.drop(dfn.index[dfn.Code.isin(extraneous)])\n",
    "\n",
    "# And check it worked by looking for the 'World' data set\n",
    "# dft = data frame tidy\n",
    "dft[dft.Code.isin(['ATF','CYM','OSS','TWN','WLD','CAN'])][['Code','Name','2016']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(dft[dft['2016'] <= 2931457])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rebuild the quantiles using tidied data frame (dft)\n",
    "quantiles = dft['2016'].quantile([0.0, 0.2, 0.4, 0.6, 0.8, 1.0]).???\n",
    "dft['q2016'] = pd.cut(dft['2016'], bins=quantiles, labels=[x for x in range(1, len(quantiles))])\n",
    "\n",
    "# For debugging purposes\n",
    "for q in range(1, len(quantiles)):\n",
    "    print(\"Quantile {} has threshold \".format(q) + '{:,.0f}'.format(quantiles[q]) + \" containing \" + str(len(dft[dft['q2016'] == q])) + \" rows\")\n",
    "\n",
    "# Rebuild the rank using the tidied data frame\n",
    "dft['r2016'] = dft['2016'].rank(ascending=False)\n",
    "\n",
    "dft.sample(7)[['Name','q2016','r2016']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finally!!!\n",
    "\n",
    "OK, we can now finally make our map by adding a few new parameters to our choropleth mapping command:\n",
    "\n",
    "1. `data=<data frame>` tells Folium to use data from the `dft` data frame (Folium 'speaks' pandas)\n",
    "2. `columns=[<column1>, <column2>]` tells Folium which columns to use from the data frame.\n",
    "3. `key_on` tells Folium how to join `column1` selected from the data frame with the GeoJSON data using a syntax of `<feature>.<properties>.<key>`.\n",
    "\n",
    "Since this is fairly tricky, I'm giving you code that _should_ work if everything up to this point has gone smoothly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = folium.Map(location=MAP_COORDINATES, zoom_start=3)\n",
    "m.choropleth (\n",
    "    geo_data =r.json(),\n",
    "    data=dft,\n",
    "    columns=['Code','q2016'],\n",
    "    key_on='feature.properties.iso_a3',\n",
    "    threshold_scale=range(1, len(quantiles)),\n",
    "    legend_name='2016 Population Quantiles',\n",
    "    fill_color='YlGn', line_color='grey', fill_opacity=0.85, line_opacity=0.75, line_weight=1)\n",
    "m.save('2016 Population Quantiles.html')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = folium.Map(location=MAP_COORDINATES, zoom_start=2)\n",
    "m.choropleth (\n",
    "    geo_data =r.json(),\n",
    "    data=dft,\n",
    "    columns=['Code','r2016'],\n",
    "    key_on='feature.properties.iso_a3',\n",
    "    threshold_scale=range(int(dft.r2016.min()), int(dft.r2016.max()), 30),\n",
    "    legend_name='2016 Population Rank',\n",
    "    fill_color='PuBu', line_color='grey', fill_opacity=0.85, line_opacity=0.75, line_weight=1)\n",
    "m.save('2016 Population Rank.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a minute to think about the results:\n",
    "2. How could we get a scale that _doesn't_ emphasise the countries with small populations?\n",
    "3. Is quantile or rank the best way to represent the population anyway?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing Your Understanding\n",
    "\n",
    "Let’s see how well you've understood everything so far... I want you to create a choropleth map using a [GeoJSON file of US states](http://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json) and the population data available from [the Census](https://www2.census.gov/programs-surveys/popest/tables/2010-2016/state/totals/nst-est2016-01.xlsx)\n",
    "\n",
    "Do this by pulling together _all_ of the code that you used above into one long code block below that you can run _separately_ from all of the other code on this page. Even the import statements at the top! Once you've got the code to run successfully once, the easiest way to check that you've got it all 'right' is to click on the `Kernel` drop-down and then select `Restart`: that will clear all of the existing variables and data so that when you run the code you've written below it won't be able to accidentally drawn upon any work you've already done. The result would then be the same as if you ran it as a standalone Python script.\n",
    "\n",
    "The structure for this code block should be something along the lines of:\n",
    "1. Import libraries\n",
    "2. Specify GeoJSON and Census URLs\n",
    "3. Download GeoJSON via a request object\n",
    "4. Download Census data via a pandas object.\n",
    "5. Do any cleaning required to match the state names (_hint_: there is a potential problem with the pandas column containing the state names -- which is probably your index -- and you will need to look up \"pandas lstrip\" in Google).\n",
    "5. Check the overlap between the State names in the GeoJSON data and the Geographic Area values in the pandas data frame.\n",
    "6. Remove any _unduplicated_ values.\n",
    "7. Map it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your Code Here...\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing the Output _Outside_ of a Notebook\n",
    "\n",
    "Note that for the interactive maps to work properly outside of the Jupyter Notebook environment you _may_ need to launch Google Chrome from the terminal using the following command:\n",
    "* Linux Terminal: `google-chrome --disable-web-security`\n",
    "* Windows: `start chrome --disable-web-security`\n",
    "* Mac: `open /Applications/Google\\ Chrome.app --args --disable-web-security`\n",
    "This is needed to get around a security restriction (normally a good thing) in Google Chrome to do with loading content from `file://...`"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:gsa2018]",
   "language": "python",
   "name": "conda-env-gsa2018-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}