Practical-03b-API Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Data\n",
    "\n",
    "## The importance of exploration\n",
    "\n",
    "One of the first things that we do when working with any new data set is to familiarise ourselves with it. There are a _huge_ number of ways to do this, but there are no shortcuts to:\n",
    "* Reading about the data (how it was collected, what the sample size was, etc.)\n",
    "* Reviewing any accompanying metadata (data about the data, column specs, etc.)\n",
    "* Looking at the data itself at the row- and column-levels\n",
    "* Producing descriptive statistics \n",
    "* Visualising the data using plots \n",
    "In fact, you should use _all_ of these together to really understand where the data came from, how it was handled, and whether there are gaps or other problems. If you're wondering which comes first, I've always liked this approach: _start with a chart_. We're _not_ going to do that here because, first, I want you to get a handle on pandas itself!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas with Weather API Data \n",
    "\n",
    "The UK's Met Office is a world-leading weather and climate research centre, and even if it doesn't always seem like their forecasts are very accurate that's because Britain's weather is inherently _unpredictable_. They've also done a lot of work to make their weather data widely available to people like us.\n",
    "\n",
    "I probably don't need to say a _lot_ about weather data because you've probably been making use of forecasts for much of your life! But it's _still_ worth understanding something about how weather data is gathered and reported: many organisations operate weather stations where data on wind speed, temperature, rain, and amount of sun are collected and then transmitted to a server to be integrated into a larger data set of weather _observations_ at a national or global scale. Of course, any _one_ station might be in the 'wrong' place (somewhere shady or protected from the rain) or it might even break down, but the idea is that if you have enough of them you can collect a pretty good range of data for the country and begin to look for patterns and, potentially, make predictions.\n",
    "\n",
    "We will be accessing data from the MetOffice from weather stations where observations, such as the ones below, are collected:\n",
    "* <Param name=\"F\" units=\"C\">Feels Like Temperature (units: degrees Celsius)\n",
    "* <Param name=\"G\" units=\"mph\">Wind Gust (units: mph)</Param>\n",
    "* <Param name=\"H\" units=\"%\">Screen Relative Humidity (units: percent)</Param> \n",
    "* <Param name=\"T\" units=\"C\">Temperature (units: degrees Celsius)</Param> \n",
    "* <Param name=\"V\" units=\"\">Visibility (units: km?)</Param> \n",
    "* <Param name=\"D\" units=\"compass\">Wind Direction (units: compass degrees)</Param>  \n",
    "* <Param name=\"S\" units=\"mph\">Wind Speed (units: mph)</Param> \n",
    "* <Param name=\"U\" units=\"\">Max UV Index (units: index value)</Param> \n",
    "* <Param name=\"W\" units=\"\">Weather Type (units: categorical)</Param> \n",
    "* <Param name=\"Pp\" units=\"%\">Precipitation Probability (units: percent)</Param>\n",
    "\n",
    "These observations are associated with a particular station (where did we see these values/where _will_ we see these values?), they will also be associated with _either_ a particular time in the past (when were they collected?) or, if they're forecasts, with a particular time in the future (when do we expect to see them?). \n",
    "\n",
    "So although weather data might seem more 'objective' than data on social class (though for obvious reasons it turns out that both are just attempts to capture data about reality, not reality itself), it may also turn out to be very complex to store and manage beccause of the temporal element _and_ the fact that it's not just a count of one thing, each of these observations uses a very different set of units.\n",
    "\n",
    "To really get to grips with the MetOffice API you will need to RTM (Read The Manual): http://www.metoffice.gov.uk/media/pdf/3/0/DataPoint_API_reference.pdf. The ruder version of that, which you will sometimes see on StackOverflow and elsewhere, is: RTFM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Weather Data via an API\n",
    "\n",
    "Because the weather is changing all the time, so is the data! And, 'worse', it's becoming obsolete: the forecast from 2 years ago isn't particularly useful to us now. *And* asking for \"yesterday's weather\" depends on the day that we're asking! When you have data that is always changing from minute to minute or day to day then you use an API (Application Programming Interface) to access it: the API knows that \"yesterday's weather\" means \"work out what day it is right now and then get the weather from the day before\", and it also knows that \"give me the current weather from station X\" means \"look up station X and find the latest weather report that I've received\". In other words, an API is  designed with programmatic, dynamic interaction in mind right from the start.\n",
    "\n",
    "### So What _is_ an API?\n",
    "\n",
    "There's a nice, friendly introduction to APIs over at [Free Code Camp](https://medium.freecodecamp.com/what-is-an-api-in-english-please-b880a3214a82#.rmjnli2nn). You can combine this with watching a video from MuleSoft that compares an API with a waiter in a restaurant:\n",
    "\n",
    "[![What is an API?](http://img.youtube.com/vi/s7wmiS2mSXY/0.jpg)](https://www.youtube.com/watch?v=s7wmiS2mSXY)\n",
    "\n",
    "Helpfully, the MetOffice provides a lot of documentation about their API (I'd suggest bookmarking it): http://www.metoffice.gov.uk/datapoint/support/api-reference\n",
    "\n",
    "This type of data requires a lot more research up front to work with, but it's very flexible once you know how to 'speak API' because you can _customise_ the API request so that the server responds with _only_ the data we're interested in instead of being 'stuck' with what the provider wants to give us.\n",
    "\n",
    "### Obtaining an API Key\n",
    "\n",
    "The first step to working with the API from the MetOffice is to obtain an API key: [do that here](http://www.metoffice.gov.uk/datapoint/API). \n",
    "\n",
    "Make a note of this key in your notebook. Right here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_key   = \"\" # your API key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That way your API key is saved somewhere easy to access.\n",
    "\n",
    "We _always_ have to use the key as part of an API request: the process by which we _ask_ for data. Think of the key as being _your_ unique identifier: no two people share the same key and that way the MetOffice can cut off people who abuse the system or look at which APIs are popular with lots of users... Twitter and Facebook do the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrub_api_key(text, key=api_key):\n",
    "    return text.replace(key, 'XXXXXX')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining a List of Sites from the API\n",
    "\n",
    "How to start? Well, the first thing that we need to know is: for what locations can I get weather data? For this to work, we need to know how to ask the API nicely for a list of available sites... We're going to show you the code and output _first_ and then we'll step through what's involved. You won't see any output from the code in the next block because we look at that later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, requests # Libraries we need\n",
    "\n",
    "api_url  = \"http://datapoint.metoffice.gov.uk/public/data/\" # base URL\n",
    "site_url = \"val/wxfcs/all/json/sitelist\" # sites API URL\n",
    "\n",
    "payload = {'key': api_key} # Dictionary to hold request parameters\n",
    "s = requests.get(api_url + site_url, params=payload) # Do the request\n",
    "\n",
    "sites = s.json() # Capture the output\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, now let's try making sense of the JSON code:\n",
    "```python\n",
    "import json, requests\n",
    "```\n",
    "So, first we import two new modules: one that makes requests to a web server, and one that will parse JSON\\* responses from the server in order to turn them into something that we can work with more easily.\n",
    "\n",
    "\\* We'll get to what JSON is in a second.\n",
    "\n",
    "```python\n",
    "api_url  = \"http://datapoint.metoffice.gov.uk/public/data/\" # base URL\n",
    "site_url = \"val/wxfcs/all/json/sitelist\" # sites API URL\n",
    "```\n",
    "\n",
    "Then we set up some default variables (`api_url`, `site_url`) that will help us to build our request to the MetOffice's server. The comments help us to remember what each of these variables holds.\n",
    "\n",
    "```python\n",
    "payload = {'key': api_key} # Dictionary to hold request parameters\n",
    "s = requests.get(api_url + site_url, params=payload) # Do the request\n",
    "```\n",
    "\n",
    "You'll notice that the `payload` is just a dictionary and that this dictionary is then passed to the `requests` library (the `get` function). All it does it convert this dictionary to a key-value pair in the format _expected by the API_. Think of it as a kind of translation between languages: from the language of Python to the language of the web (HTTP, to be precise).\n",
    "\n",
    "We issue our request and it returns a response that we store in `s` (short for `sites data`).\n",
    "\n",
    "```python\n",
    "sites = s.json() # Capture the output\n",
    "```\n",
    "\n",
    "Lastly, we ask the response object to convert the reply into a JSON data structure... more on JSON in a second, but first let's look at what we got from our request!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the requested URL\n",
    "print(s.url) # Click on the link below to see it nicely formatted automatically!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Capture the returned data\n",
    "print(str(sites[0:2500]) + \"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So, What's JSON?\n",
    "\n",
    "It's pretty hard to figure out what that 'JSON' reply means, but if you look closely you'll start to see a few things that you recognise:\n",
    "\n",
    "1. There is the '{' and '}' that we associate with dictionaries\n",
    "2. There is the ':' that we associate with key-value pairs in dictionaries\n",
    "3. The really hard thing is all the extra 'u's all over the place but that's something that's confusing to _us_, not the _computer_.\n",
    "\n",
    "So it's actually just a kind of dictionary containing lists and other dictionaries. That's it. The output looks like a mess, but it _is_ a dictionary and the only thing that is entirely new is the fact that every string has the letter 'u' in front of it. \n",
    "\n",
    "That 'u' means 'Unicode' and it just a special kind of string that supports accents, Chinese characters, emojis, and just about anything else that you can think of..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using XML to better understand JSON\n",
    "\n",
    "We want to work with JSON replies because they are easier for Python to read, but sometimes it is easier for _us_ to read the output of an API if we are able to ask for the answer in XML (eXtensible Markup Language). Try running the code block below and then copy+pasting the URL that it outputs into your browser window:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xml_url = \"val/wxfcs/all/xml/sitelist\"\n",
    "x = requests.Request('GET', api_url + xml_url, params=payload).prepare()\n",
    "print(x.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back to JSON...\n",
    "\n",
    "Now that you have a better idea of what data the reply contains, let's see if we can convert the JSON reply into something useful for Python; if you scroll back up to where we printed out the reply you'll notice that it all starts with a '{', meaning that it's a dictionary.\n",
    "\n",
    "Let's start by printing out the keys in the dictionary and the _type_ of data associated as a value to that key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in sites.keys():\n",
    "    print(\"Key: \" + str(k))\n",
    "    print(\"Value: \" + str(type(sites[k])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not the most exciting answer, but at least we know that the value is a dictionary. Let's try moving down a level:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in sites.keys():\n",
    "    print(\"Key: \" + str(k))\n",
    "    print(\"Value: \" + str(type(sites[k])))\n",
    "    for k2 in sites[k].keys():\n",
    "        print(\"\\tKey: \" + str(k2))\n",
    "        print(\"\\tValue: \" + str(type(sites[k][k2])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MetOffice is not making life easy for us here: there's a _lot_ of extra 'baggage' in this API response. But we at least know that the next level down is a list and _that_ suggests that things are about to get a bit more interesting... Let's simplify our code at the same time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "apiList = sites['Locations']['Location']\n",
    "print(\"List in API response is \" + str(len(apiList)) + \" long\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now _that_ is a rather more interesting response. What it means is that our JSON reply has this structure:\n",
    "```\n",
    "{\n",
    "    'Locations': {\n",
    "        'Location': [\n",
    "            ... lots of data here ...\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "If you scroll back up to the JSON reply you should now be able to read a little bit more of the response... and this should give you a clue as to how to print out the `name`, `id`, and `longitude` of the first five sites. I'll get you started:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(???):\n",
    "    location = apiList[???]\n",
    "    print(\"Location: \" + ??? + \" (id: \" + ??? + \") is at longitude: \" + location[???])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your answer should look like this:\n",
    "\n",
    "```python\n",
    "Location: Carlisle Airport (id: 14) is at longitude: -2.8092\n",
    "Location: Liverpool John Lennon Airport (id: 26) is at longitude: -2.85\n",
    "Location: Scatsta (id: 33) is at longitude: -1.2992\n",
    "Location: Kinloss (id: 3066) is at longitude: -3.5606\n",
    "Location: Lossiemouth (id: 3068) is at longitude: -3.322\n",
    "````\n",
    "\n",
    "And _now_ that we know where all the data was 'hidden', we can convert this to a proper data structure in which it is possible to _interact_ with it. To do that, we'll put the site data into a pandas data frame..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Turning API data into a Pandas DataFrame\n",
    "\n",
    "Pandas is remarkably intelligent and will _often_ -- though not always -- work out the sensible thing to do from many kinds of data structures (list-of-lists, dictionary-of-lists, list-of-dictionaries...). So let's see what happens when we simply pass `apiList` (a LoD) directly to the `DataFrame` 'constructor' instead of passing the data through, for instance, the `read_csv` function as we did above with a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df2 = pd.DataFrame(apiList)\n",
    "\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, that's... almost scarily easy. You can see that pandas worked out the structure of our LoD and then automatically converted that to columns in a data frame. So it got the hardest part of the process exactly right and has saved us a lot of work. _That_ is the point of functions and of code: to be constructively lazy.\n",
    "\n",
    "Of course, we could have predicted that pandas would cope since there is a whole section in the documentaiton [devoted to creating data frames from different structures](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe).\n",
    "\n",
    "But let's take a slightly closer look at the Elevation data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.elevation.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's that? The data-type (dtype) is an _object_? Shouldn't that be a float? And shouldn't the description give us the 7-figure summary for numeric data? The 'problem' (if you looked closely at the JSON response) is that everything came back to us as Unicode text (the little 'u's in front of each key and value) and not as numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fixing Column Data-types\n",
    "\n",
    "The problem is that pandas didn't know what we expected the columns to be, so it's treated them all as 'objects' (basically: strings) and not as numeric data types. To fix that you need to know that there's a function called `'astype'` that allows pandas to convert between different data types where it's fairly easy for pandas to figure out what we want to do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in ['region','unitaryAuthArea']:\n",
    "    df2[c] = df2[c].astype('category')\n",
    "for c in ['elevation','latitude','longitude']:\n",
    "    df2[c] = df2[c].astype('float')\n",
    "for c in ['id']:\n",
    "    df2[c] = df2[c].astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try that description again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.elevation.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "df2.elevation.plot.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigating Unknown Columns\n",
    "\n",
    "You'll note that I didn't change the column types for `name`, `nationalPark`, and `obsSource`. There's no reason to change the name since we rather _assume_ that it's pretty much unique (the output of `describe` confirms it). For the others, it's because we don't really know what we expect to find!\n",
    "\n",
    "There's no 'one' way to investigate the values in an 'unknown' column, but it's often worth thinking about _unique_ values as a starting point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Data type is: \" + df2.nationalPark.dtype.name)\n",
    "print(\" \")\n",
    "print(df2.nationalPark.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tells us that there are 618 rows (out of nearly 6,000) that have a value that is _not_ `NaN`, and that there are 16 unique non-`NaN` values. We can list them easily in pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.nationalPark.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the number of unique parks, what is the _best_ data-type for this column? Fix the code below to change the column type of the nationalPark data and assign the updated column it back to `nationalPark` in the block below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['???'] = df2.???.astype(???)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's check our data type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Data type is: \" + df2.nationalPark.dtype.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Data in Pandas\n",
    "\n",
    "We've got ourselves a pandas data frame containing a set of locations, how do we go about finding one or more _specific_ rows in the data set rather than just summarising the data via `describe`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for a Number\n",
    "\n",
    "This is the easiest type of search to do in pandas because it looks _most_ like code you've already seen.\n",
    "\n",
    "### Find All Sites East of 1.1 Degrees Longitude\n",
    "\n",
    "To translate this into code, we just need to remember that: a) East would be _greater than_; and b) longitude is already a float. So in that case it's..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfEast = df2[df2.longitude > 1.1]\n",
    "dfEast.head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's break this down:\n",
    "\n",
    "* `df2.longitude` is obviously the longitude column of our data frame `df2`\n",
    "* `df2.longitude > 1.1` is therefore a kind of _query_ (or _selection_) of rows where the longitude is greater than 1.1. What it _actually_ does is compare each row's longitude value to 1.1 and remember if the result is `True` or `False`.\n",
    "* `df2[ ... ]` is _like_ what we do with a list when we write: `myList[3:5]` to select the fourth through sixth elements of a list, but in pandas we can _select_ non-sequential rows because we are using a `boolean` array (a.k.a. list) that looks like this: `[False, False, False, True, True, True, ...]`.\n",
    "* `dfEast = ...` saves the _result_ of the selection into a new data frame called `dfEast` (data frame East).\n",
    "\n",
    "You can check what I'm saying about the boolean result using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.longitude > 1.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can check that `dfEast` and `df2` are not the same using `shape`, which gives us the dimensions of the data frame as `(<rows>, <columns>)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.shape)\n",
    "print(dfEast.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What this first example means is that _anything_ that can be evaluated to `True` or `False` can be used to select rows from a data frame... Let's try some more selections based on numbers..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding the Minimum & Maximum Elevation\n",
    "\n",
    "The lowest point is in the Fens, and the highest point is, of course, Ben Nevis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[df2.elevation == df2.elevation.min()] # Somewhere in Cambridgeshire"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[df2.elevation == df2.elevation.???] # Find Ben Nevis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding a Range Between Known Values\n",
    "\n",
    "Perhaps we aren't just looking for extremes... how about all of the areas between 55 and 55.2 degrees latitude? *[There were 91 the last time I checked.]*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfRange = df2.loc[ (df2.latitude > 55.0) & (df2.latitude < 55.2) ]\n",
    "dfRange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That example contains a few new things to which you need to pay attention:\n",
    "1. You'll see that, with mutiple selections, we had to put parentheses around each one -- this is to avoid confusing pandas as to what it should do _first_.\n",
    "2. We see an '&' (ampersand) which is completely new: it's a logical `AND` that asks pandas to \"Find all the rows where condition 1 _and_ condition 2 are both `True`\". So it calculates the `True`/`False` for the left side and the `True`/`False` for the right side of the `&`, and then combines them. Look at the appendix to this notebook for more examples and options.\n",
    "3. We had to a `.loc` on the end of the `df2` -- the best way to think of this is that it 'freezes' things so as to prepare the data frame to do a search based on the _location_ of some complex selection criteria. We'll see more of this next week.\n",
    "\n",
    "### Finding a Range Based on the Distribution\n",
    "\n",
    "Finally, let's try finding the stations whose elevation is _greater_ than the mean. *[There are 1,330 the last time I checked.]*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfMean = df2.loc[ df2.elevation > df2.???.mean() ]\n",
    "print(\"There are \" + str(dfMean.shape[0]) + \" sites above the mean elevation of \" + str(df2.elevation.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching on Text & Categories\n",
    "\n",
    "Numeric searching is all well and good, but what if I'm interested in finding stations in a particular area?\n",
    "\n",
    "### Searching for a Category\n",
    "\n",
    "Let's find the names of every station inside the Cairngorms National Park! *[There were 71 the last time I checked.]*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[ df2.??? == ??? ].name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching for Part of a String\n",
    "\n",
    "If you want to find a full match for a string then it's fairly easy and works like everything you've seen before with string matching:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[df2.name=='Cairn Gorm Summit']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And pandas also provides a lot of useful tools for searching _inside_ a string, as long as you remember to _tell_ pandas to use the string-methods (notice the format: `<data frame>.<data series>.str.<string method>()`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[df2.name.str.startswith('Beinn A\\'')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[df2.name.str.endswith('Summit')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Searching _inside_ a string is no harder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[df2.name.str.contains('Charn')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How would you ensure that you found _only_ the Geal Charn stations? \n",
    "\n",
    "Combine what you've learned above to create a complex query (two conditions on a single line) using a mix of search critera:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc[ (???) & (???'Highland') ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, I want you to find and print out **_only_ the ID of Heathrow the town _not_ the Airport** using a single line of code. There are _at least_ two ways to retrieve this...\n",
    "\n",
    "We are going to want that ID for the next step in working with the MetOffice API, but there is a last trick to learn here and that's how to extract an actual value as a string, int, or float from a data series. The thing to remember is that a Series is basically a list with a lot of value-added features. The contents of the list can be found in `<data series>.values`. So to get the 2nd through 5th values of the elevation column it would be:\n",
    "```python\n",
    "df2.elevation.values[1:5]\n",
    "```\n",
    "\n",
    "If you do the selection criteria for Heathrow Airport properly there should only be one item in the list of values that you retrieve, so the right code will include a `[0]` at the end:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heathrow = ???\n",
    "print(heathrow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Obtaining Weather Data\n",
    "\n",
    "The next step in this process is a bit more complicated because weather data is a bit more complicated than a list of locations...\n",
    "\n",
    "First, just in case you want to only run this section again (and not revisit the content above), I'd suggest saving a copy of your API key here as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_key = '???'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, requests # Libraries we need\n",
    "\n",
    "api_url  = \"http://datapoint.metoffice.gov.uk/public/data/\" # base URL\n",
    "obs_json= \"val/wxobs/all/json/\" # observations URL\n",
    "\n",
    "heathrow = str(352415)  # heathrow airport weather station\n",
    "\n",
    "payload = {'res': 'hourly', 'key': api_key} # Dictionary to hold request parameters\n",
    "\n",
    "r = requests.get(api_url + obs_json + heathrow, params=payload)\n",
    "\n",
    "print(r.url)\n",
    "\n",
    "weather = r.json() # Capture the reply\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(weather)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's yet another bunch of 'data' that's difficult for us to read, but by now this should be looking rather familiar to you... perhaps? Hang on a moment! It's a dictionary-of-lists-of-dictionaries-of..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a DataFrame from a Dictionary\n",
    "\n",
    "And that, of course is exactly the type of data structure that we can work with in pandas! \n",
    "\n",
    "So the _last_ step here is to figure out how to create a new data frame from this dictionary. Here, the MetOffice has _not_ made our lives very easy because the data is packaged in a way that doesn't allow us to easily load it into pandas. If you search online, you'll find plenty of people complaining about how the MetOffice API works. Or doesn't work, if you prefer.\n",
    "\n",
    "So we're not going to ask you to sort this out for yourselves. Instead, we're going to provide you with a function (!) to take the observation data and convert it into a data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime, timedelta\n",
    "def processMetOfficeObservations(loc): \n",
    "    \"\"\"\n",
    "    Process a series of 'reports' for a single\n",
    "    location using the datetime object as the \n",
    "    reference time against which to build the \n",
    "    timedelta (i.e. we start from midnight and \n",
    "    the timedelta is the number of minutes past \n",
    "    midnight)\n",
    "    \"\"\"\n",
    "    observations = {} # Stores results\n",
    "    \n",
    "    for d in loc['Period']: # d for day\n",
    "        dt = datetime.strptime(d['value'],'%Y-%m-%dZ') # Convert date to datetime object\n",
    "    \n",
    "        # Now deal with the actual observations (i.e. 'Reports')\n",
    "        for report in d['Rep']:\n",
    "            \n",
    "            # Find the timestampe and add it to the date\n",
    "            minutes_after_midnight = int(report['$'])\n",
    "            ts = dt + timedelta(minutes=minutes_after_midnight)\n",
    "            \n",
    "            # For each of the possible values, set a default value\n",
    "            # if the weather station doesn't actually collect that\n",
    "            # parameter... can you see a problem with our defaults?\n",
    "            if 'ts' not in observations:\n",
    "                observations['ts'] = []\n",
    "            observations['ts'].append( str(ts) )\n",
    "            for key in ['D','Pt']:\n",
    "                if key not in report:\n",
    "                    report[key] = u\"\"\n",
    "                if key not in observations:\n",
    "                    observations[key] = []\n",
    "                observations[key].append(report[key])\n",
    "            for key in ['W','V','S','G']:\n",
    "                if key not in report or report[key] == \"\":\n",
    "                    report[key] = 0\n",
    "                if key not in observations:\n",
    "                    observations[key] = []\n",
    "                observations[key].append(report[key])\n",
    "            for key in ['T','Dp','H']:\n",
    "                if key not in report or report[key] == \"\":\n",
    "                    report[key] = 0.0\n",
    "                if key not in observations:\n",
    "                    observations[key] = []\n",
    "                observations[key].append(report[key])\n",
    "    \n",
    "    return observations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = processMetOfficeObservations( weather['SiteRep']['DV']['Location'] )\n",
    "df3 = pd.DataFrame.from_dict( data )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well that was hard... but did it really do what we expected?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.head(3) # Check that it did what we expected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tidying Up\n",
    "\n",
    "Before we can get back to plotting (again) we have a few more steps to work through:\n",
    "\n",
    "1. To rename the columns to something a little more useful.\n",
    "2. To turn the 'ts' field into an _actual_ timeseries so that pandas understands what it is.\n",
    "3. To convert all of the other series to the right numerical/categorical format.\n",
    "\n",
    "Let's do this in several stages... \n",
    "\n",
    "### Changing Column Names\n",
    "\n",
    "You may remember that I indicated what the observations returned by each weather station might include:\n",
    "\n",
    "* D  = Wind Direction\n",
    "* Dp = Dew Point\n",
    "* G  = Wind Gust\n",
    "* H  = Humidity\n",
    "* Pt = Pressure Tendency\n",
    "* S  = Wind Speed\n",
    "* T  = Temperature\n",
    "* V  = Visibility\n",
    "* W  = [Weather Type](https://www.metoffice.gov.uk/services/data/datapoint/code-definitions)\n",
    "* ts = Time of Day\n",
    "\n",
    "Given this, and the fact that I've listed these in order, what needs to replace the '???' in the code below?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_map = {\n",
    "    'D':  'WindDirection',\n",
    "    'Dp': 'DewPoint',\n",
    "    'G':  'WindGust',\n",
    "    'H':  'Humidity',\n",
    "    'Pt': 'PressureTendency',\n",
    "    'S':  'WindSpeed',\n",
    "    'T':  'Temperature',\n",
    "    'V':  'Visibility',\n",
    "    'W':  'WeatherType',\n",
    "}\n",
    "df3.rename(columns=???, inplace=True)\n",
    "df3.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see the 'full' column names now.\n",
    "\n",
    "### Changing column types\n",
    "\n",
    "If you were exploring the data frame along the way, you might have already noticed that the description of numeric columns (like Temperature) doesn't seem much like what we had before -- shouldn't we get the 7-figure summary for numeric columns? The problem is that pandas didn't know what we expected the columns to be, so it's treated them all as 'objects' (basically: strings) and not as numeric data types.\n",
    "\n",
    "So we need to fix that now... as you saw before, there's a function called `'astype'` that allows us to convert between data types where it's fairly easy for pandas to figure out what we want to do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.Temperature.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in ['WindDirection','WeatherType','PressureTendency']:\n",
    "    df3[c] = df3[c].astype('category')\n",
    "for c in ['DewPoint','Humidity','Temperature']:\n",
    "    df3[c] = df3[c].astype('float')\n",
    "for c in ['WindGust','Visibility']:\n",
    "    df3[c] = df3[c].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.Temperature.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's more like it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with Timeseries Data\n",
    "\n",
    "So that's looking a lot more useful, but as a final step we need to make sure that the temporal data is actually treated as a time series... again, Google is your friend here: `\"pandas convert datetime to time series\"`. \n",
    "\n",
    "Given that we are creating a _new_ column called 'Time' from an _existing_ column called 'ts', what do you think needs to replace the '???'?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3['Time'] = pd.to_datetime(df3.???.values, infer_datetime_format=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.ts.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.Time.head() # A quick check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a Time Series in an Index\n",
    "\n",
    "We can tell that that type conversion succeeded because we've got a new `dtype`: `datetime64[ns]`. \n",
    "\n",
    "We can also now do some really neat things to 'resample' the data based on the fact that we have temporal data; however, to take advantage of this we have to let pandas know that the entire data set is organised by time. We do this by replacing the existing integer index with a datetime one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.index = pd.to_datetime(df3.ts.values, infer_datetime_format=True)\n",
    "df3.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the output above, you'll notice that the left-most column (the one without a name, because it's an _index_, not a column) is now a datetime object. Why is that useful? Well check _this_ out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.Temperature.resample('D').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, this is another geeky moment but how cool is that? By telling pandas that our data is temporal, we're now in a position to ask pandas to answer questions like \"What was the average daily ('D') temperature at Heathrow?\" And we can do this in _one_ line of code.\n",
    "\n",
    "If you had data at minutes-level resolution, then you could aggregate to Hourly or Daily. In principle, you can also do all sorts of datetime queries around things like \"What was the weekly average in the 3rd week of 2016?\" or \"What was last Friday's weather?\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting!\n",
    "\n",
    "This has been a long, slow build towards something more exciting: plotting! Well, plotting _again_. In a way, this has been a lot of effort just to make a graph, but let's recognise where we're at:\n",
    "\n",
    "* We can request data for _any_ lcoation in Britain by changing the location id.\n",
    "* We can get new data _any_ time we feel like it.\n",
    "* We can (in a minute) create a plot of that data.\n",
    "* We can update it continuously in the future!\n",
    "\n",
    "That's pretty awesome, right?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This command tells Jupyter that we want \n",
    "# the plots to be shown inline (on this \n",
    "# web page). You'll always need to do this\n",
    "# *once* on a notebook.\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas can do a _lot_ of different plots, [see for yourself](http://pandas.pydata.org/pandas-docs/stable/visualization.html#visualization-hexbin). Here's a sampling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.Humidity.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.Temperature.plot.bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.Temperature.plot.box() # Handy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.plot.scatter(x='Temperature', y='Humidity') # Spot the problem data point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.plot.scatter(x='Temperature', y='Humidity', s=(df3.DewPoint+1)*25);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.plot.hexbin(x='Temperature', y='Humidity', gridsize=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.WindDirection.plot() # Ooops."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I tend to think that that's quite enough to be coping with for one session... over to the script!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix\n",
    "\n",
    "The material below here is very helpful if you _really_ want to get to grips with both some powerful computing concepts (hello recursion!) and to understand how I pulled together the data frame through working with the data iteratively to get to grips with the MetOffice reply. However, it is not _necessary_ that you undertand these ideas now since they are relatively advanced and will be more directly useful in the _Spatial Analysis_ and _Applied Geocomputation_ modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logical Comparisons\n",
    "\n",
    "You have already seen a bit of Boolean logic using 'and', 'or', and 'not'. For reasons that aren't really worth getting into here, when you're dealing with the simple binary True/False data (a.k.a. [bit-wise](https://wiki.python.org/moin/BitwiseOperators)) comparisons then the same operations are written using a slightly different syntax:\n",
    "\n",
    "1. 'and' becomes '&'\n",
    "2. 'or' becomes '|'\n",
    "3. 'not' becomes '~'\n",
    "\n",
    "These are rather topical for complex queries in pandas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Recursion to Explore JSON Data\n",
    "\n",
    "We've seen dictionaries-of-lists and dictionaries-of-dictionaries before! We know how these work, but they've never been very easy to work with because we had to write lots and lots of nested loops:\n",
    "\n",
    "```python\n",
    "for key1 in bigDictionary:\n",
    "    for key2 in bigDictionary[key1]:\n",
    "        for key3 in bigDictionary[key1][key2]:\n",
    "            ... And so on ...\n",
    "```\n",
    "\n",
    "And if we have to add checks on each one of these `keys` to see if it is a list, a dictionary, or a simple float/int then this code would explode in complexity and become very, very hard to follow.\n",
    "\n",
    "But there is another way. It's a concept called _recursion_. \n",
    "\n",
    "Let's imagine that we have to deal with lists-of-lists (because those are a bit simpler to think about) but we don't know in advance how many lists there are inside of each list; e.g.:\n",
    "```python\n",
    "myList = [\n",
    "    ['Value 1',\n",
    "        ['Value 1.a.i', 'Value 1.a.ii'],\n",
    "        ['Value 1.b.i', 'Value 1.b.ii', \n",
    "            ['Value 1.b.ii.I', 'Value 1.b.ii.II'],\n",
    "        'Value 1.c'],\n",
    "    ['Value 2'],\n",
    "    'Value 3'\n",
    "]\n",
    "```\n",
    "What a nightmare! That's hard to even _read_, let alone know how to process! But recursion allows us reframe this problem as something that is _almost_ simple (it's certainly elegant): we need a function that steps through a list one element at a time and then: \n",
    "\n",
    "* if the element is a simple value (float, int or string) then it prints it out, \n",
    "* if the element is a list then the function _calls itself_ on the nested list! \n",
    "\n",
    "In other words, when our list-reading-function finds a new list _inside_ the list it is currently reading, then it calls itself and passes in the list-inside-the-list.\n",
    "\n",
    "That explanation probably _still_ doesn't take much sense, but take a look at the code below. \n",
    "\n",
    "**Really, really look**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def outputList(lst, depth=0): \n",
    "    for value in lst:\n",
    "        if type(value) is list:\n",
    "            outputList(value, depth+1)\n",
    "        elif type(value) is dict: \n",
    "            outputDict(value, depth+1)\n",
    "        else:\n",
    "            print \"\\t\" * depth + \"l-Value: \" + value\n",
    "    print \"\\n\"\n",
    "\n",
    "def outputDict(dct, depth=0):\n",
    "    for key, value in dct.iteritems():\n",
    "        print \"\\t\" * depth + \"d-Key: \" + key\n",
    "        if type(value) is list:\n",
    "            outputList(value, depth+1)\n",
    "        elif type(value) is dict:\n",
    "            outputDict(value, depth+1)\n",
    "        else:\n",
    "            print \"\\t\" * depth + \"  d-Value: \" + value\n",
    "    print \"\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, `outputList` takes a list `l` and then steps through each element of that list. If it encounters an element that is a list, it calls `outputList` and passes it the list that it just found. If it encounters an element that is a dictionary, it calls `outputDict` and passes it the dictionary that it just found. If it encounters a simple value (the `else`) then it just prints it out.\n",
    "\n",
    "`outputDict` works the same way.\n",
    "\n",
    "Now, what's going on with `depth`? That is what allows to demonstrate actual recursion, but it contains two _new_ ideas that you'll need to make sense of:\n",
    "\n",
    "1. It demonstrates that you can have _default values_ in a function. In this case, a call to the `outputList` function _has_ to include a list, but the parameter _depth_ is optional: if you don't specify a depth, then it defaults to 0.\n",
    "2. You can see that when `outputList` and `outputDict` call _themselves_ they increment the `depth` variable by one.\n",
    "\n",
    "Let's look at a simpler example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def showRecursion(depth=0):\n",
    "    print(\"\\t\" * depth + \"I'm now at depth \" + str(depth))\n",
    "    if depth < 3:\n",
    "        showRecursion(depth+1)\n",
    "    print(\"\\t\" * depth + \"I'm done with depth \" + str(depth))\n",
    "\n",
    "showRecursion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `\"\\t\" * depth` as part of our print statement to indent the output because it multiplies the tab character (`\\t`) by the depth (e.g. 1, 2, etc.) to print out 1, 2, 3 tabs. You'll also notice that every time we recurse we increment (increase) depth by 1. So:\n",
    "\n",
    "* We start at depth 0 with `showRecursion()`.\n",
    "* We print out \"I'm now at depth 0\".\n",
    "* We call `showRecursion(0+1)` (because that's what `depth+1` does)\n",
    "* We print out \"I'm now at depth 1\".\n",
    "* We call `showRecursion(1+1)` (because that's what `depth+1` does)\n",
    "* ...\n",
    "* We reach `depth == 3` and so _don't_ call `showRecursion(3+1)` because depth is no longer less than 3.\n",
    "* _Now_ we print out \"I'm done with depth 3\" (that's the end of `showRecursion(2+1)`)\n",
    "* _Now_ we print out \"I'm done with depth 2\" (that's the end of `showRecursion(1+1)`)\n",
    "* _Now_ we print out \"I'm done with depth 1\" (that's the end of `showRecursion(0+1)`)\n",
    "* _Now_ we print out \"I'm done with depth 0\" (that's the end of `showRecursion()`)\n",
    "\n",
    "That's a _lot_ to get your head around, but it's the same as we're doing with `outputList` or `outputDict`. The recursive function helps us to make the formatting legible, so it's probably better that we just see it action... In our case we know that we're starting with a dictionary so we would ask `outputDict` to start outputting the content of `pdesc` (the rePly DESCription). `outputDict` then takes each of the key/value pairs in turn, looks at the value to see if _it_ is a dictionary or list or (by default) string and takes appropriate action. Don't get too stressed out if it doesn't make sense just yet, but it's such a powerful concept that it's definitely worth getting to grips with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, requests # Libraries we need\n",
    "\n",
    "api_url  = \"http://datapoint.metoffice.gov.uk/public/data/\" # base URL\n",
    "obs_json= \"val/wxobs/all/json/\" # observations URL\n",
    "\n",
    "heathrow = str(heathrow)  # heathrow airport weather station\n",
    "\n",
    "payload = {'res': 'hourly', 'key': api_key} # Dictionary to hold request parameters\n",
    "\n",
    "r = requests.get(api_url + obs_json + heathrow, params=payload)\n",
    "\n",
    "print(scrub_api_key(r.url))\n",
    "\n",
    "weather = r.json() # Capture the reply\n",
    "pdesc   = weather['SiteRep']['Wx'] # Weather Metadata\n",
    "pdat    = weather['SiteRep']['DV'] # Weather Data Values\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputDict(pdesc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so what we have is:\n",
    "\n",
    "* A dictionary saved in the variable `pdesc` (parameter-description)\n",
    "* It contains one key only: `Param`\n",
    "* `pdesc['Param']` is a list of dictionaries\n",
    "\n",
    "How do I know this? I investigated..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Type for pdesc['Param']: \" + str(type(pdesc['Param'])))\n",
    "\n",
    "print(\"Type for pdesc['Param'][0]: \" + str(type(pdesc['Param'][0])))\n",
    "\n",
    "print(\"Contents of pdesc['Param'][0]: \" + str(pdesc['Param'][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the point here is that we know have little bundles of information about the data the MetOffice is giving back to us: the parameter description dictionary tells us, for instance, that the name 'G' in the data-part of the reply is data about 'wind gusts' given in miles per hour ('mph'). We can do the same for every other parameter.\n",
    "\n",
    "Now, let's see what we get when we look at the reply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputDict(pdat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right, so that's a lot more complex isn't it? But we can make sense of it in the same incremental way...\n",
    "\n",
    "We can start off by noticing that there are some useful _generic_ fields:\n",
    "\n",
    "* `pdat['dataDate']` will give us the date and time of the data in the reply.\n",
    "* `pdat['type']` tells us that we're looking at _Obs_-ervations\n",
    "\n",
    "And so on. The really interesting one in there is the 'Location'... let's investigate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pdat['Location']['name'])\n",
    "print(pdat['Location']['elevation'])\n",
    "print(pdat['Location']['lat'])\n",
    "print(pdat['Location']['lon'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That leaves us with the rather nasty-looking `pdat['Location']['Period']`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdat['Location']['Period']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, however, if we don't panic then we can make sense of it! First, let's look at the big pieces:\n",
    "\n",
    "* It's pretty obvious that there's a set of dictionaries in there -- we can see the '{...}'!\n",
    "* We can also see things that look like readings: 'D', 'Dp', 'H', 'P'...\n",
    "* We can also see two rather useful-looking bits of information: something that says 'Day' and something that looks like a timestamp (e.g. '2016-10-14Z')\n",
    "\n",
    "Let's work on this some more by trial-and-error, starting from the point that the data structure must be a list because of the '[...]':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputDict(pdat['Location']['Period'][0], 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputDict(pdat['Location']['Period'][1], 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, now we know that `pdat['Location']['Period']` is a list of daily reports. How do I know that? Because when I asked for the first item in the list I got an answer with yesterday's date, and when I asked for the second item in the list I got something containing today's date! And _within_ each of those is _another_ list that contains a set of reports about the weather at Heathrow!\n",
    "\n",
    "The _last_ clue in there is that one of the parameters is changing in an unusal way: we can guess what H (Humidity), P (Pressure) and most of the rest are from having output `pdesc` above, but the '$' is always in multiples of 60. Can you guess why?\n",
    "\n",
    "Let's see if we can turn this into something useful... Fix the '???' so that prints out the temperature reading at Heathrow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for d in pdat['Location']['Period']: # d is short for day\n",
    "    print(\"Date: \" + d['value'])\n",
    "    for i in d['Rep']: # i is short for time interval\n",
    "        print(\"\\tTime: \" + str(i['$']))\n",
    "        print(\"\\t\\tTemperature is: \" + str(i[???]))\n",
    "        print(\"\\t\\tHumidity is:    \" + str(i[???]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you explain why there are two days in there and why the '$' values don't overal?\n",
    "\n",
    "Use the coding area below to print out the other values over the same period of time..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Time Data\n",
    "\n",
    "To really deal with the Weather data we need to know how to work with dates and times. Naturally, Python has a library that can help with that so let's first have a look at the obvious date in there: `dataDate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime, timedelta \n",
    "\n",
    "# Ignore the time part of 'dataDate' as we're \n",
    "# getting API data with values in minutes after\n",
    "# midnight. Given that, we want to set this \n",
    "# starting point to 00:00:00Z\n",
    "obsDate = datetime.strptime(pdat['dataDate'].split(\"T\")[0],'%Y-%m-%d')\n",
    "\n",
    "print(obsDate)\n",
    "print(type(obsDate)) # It's a new type of object... beyond float, int, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, that's a start, now let's see if we can do two more things:\n",
    "\n",
    "1. Do 'date math' so that we can add the number of minutes since midnight associated with the observation to the datetime object extracted from `dataDate`.\n",
    "2. Print out _one_ observation from the full range.\n",
    "\n",
    "We do this as a 'nested' loop: we know that we have one 'period' for each day contained in the data. And we know that within each day we have one 'report' for each hour. We can't (and don't want to) go through these in a random order so we use two `for` loops:\n",
    "\n",
    "```python\n",
    "for d in <day>:\n",
    "    for h in <hour>:\n",
    "        ...do something...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for d in pdat['Location']['Period']: # d is for day of observations\n",
    "    \n",
    "    dataDate = datetime.strptime(d['value'],'%Y-%m-%dZ') # Convert date to datetime object\n",
    "    print(\"Observation date: \" + str(dataDate)) # Print for debugging\n",
    "    \n",
    "    for h in d['Rep']: # h is for hourly reports\n",
    "        obsDate = dataDate + timedelta(minutes = int(h['$'])) # Add the time in minutes to the datetime\n",
    "        # print(obsDate) # Debug!\n",
    "        \n",
    "        print(\"Temperature at \" + str(obsDate) + \" is \" + str(h['T']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how that worked? We did the following:\n",
    "\n",
    "1. For each day `d` in the data...\n",
    "2. We used the `value` parameter to initialise (i.e. set) a variable called `dataDate`\n",
    "3. We then retrieved the 'hours since midnight' from the \\$ parameter and added it to dataDate using the `timedelta` function.\n",
    "4. And then we found the temperature using the `T` key."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "GSA2019",
   "language": "python",
   "name": "gsa2019"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}