Practical-02-Functions and Packages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center\">\n",
    "<h1>Week 2: Functions & Packages</h1>\n",
    "<h2>5SSG2059 Geocomputation 2019/20</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This Week’s Overview\n",
    "\n",
    "This week we're going to apply a number of the concepts covered in class in order to read a remote CSV file, turn it into data, and then perform some simple analyses on it. This weel we'll do it 'by hand' (building the tools we need); next week we'll see how using the 'pandas' package can facilitate and make things easier for us.\n",
    "\n",
    "## Learning Outcomes\n",
    "\n",
    "By the end of this practical you should:\n",
    "- Have checked your understanding of how lists, dictionaries and loops 'work'\n",
    "- Have imported a package and used its functions to read a remote data file\n",
    "- Written a function to derive some statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists, Dictionaries and Loops\n",
    "\n",
    "We're going to start off using lists and dictionaries that _we_ define right at the start of the 'program', but the _real_ value of these data structures comes when we build a list or dictionary _from_ data such as a file or a web page... and that's what we're going to do below!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lists\n",
    "First, here's a reminder of some useful methods (_i.e._ functions) that apply to lists:\n",
    "\n",
    "| Method                 | Action |\n",
    "|------------------------|------------------------------------------------------------------|\n",
    "| list.count(`x`)        | Return the number of times x appears in the list                 |\n",
    "| list.insert(`i`, `x`)  | Insert value `x` at a given position `i`                         |\n",
    "| list.pop([`i`])        | Remove and return the value at position `i` (`i` is optional)    |\n",
    "| list.remove(`x`)       | Remove the first element from the list whose value is `x`        |\n",
    "| list.reverse()         | Reverse the elements of the list in place                        |\n",
    "| list.sort()            | Sort the items of the list in place                              |\n",
    "| list.index(`x`)        | Find the first occurence of `x` in the list                      |\n",
    "| list[x:y]              | Subset the list from index `x` to `y-1`                          |\n",
    "\n",
    "This should all be revision... if you've done [Code Camp 6](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/). If not, then you've got some catching up to do!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 Interacting with Lists\n",
    "\n",
    "Replace `???` in the following code blocks to make the code work as instructed in the comments. All of the methods that you need are listed above, so this is about testing yourself on your understanding *both* of how to read the help *and* how to index elements in a list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** The next line creates a list of city names (each element is a string) - run the code and check you understand what it is doing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cities = [\"Bristol\", \"London\", \"Manchester\", \"Edinburgh\", \"Belfast\", \"York\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Replace the `???` so that it prints the position of Manchester in the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The position of Manchester in the list is: \" + str(cities.???('Manchester')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Replace the `???` so that it prints Belfast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cities[2 + ???])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** Use a **negative** index to print *Belfast*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cities[???])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e)** Force Python to generate a 'list index out of range' error. NB: This error happens you provide an index for which a list element does not exist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cities[???])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f)** Think about what the next line creates, then run the code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temperatures = [15.6, 16.5, 13.4, 14.0, 15.2, 14.8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g)** What would you change `???` to, to return `[16.5, 13.4, 14.0]`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(temperatures[???])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**h)** What are two different ways of getting `[15.2, 14.8]` from the `temperatures` list?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(temperatures[???])\n",
    "print(temperatures[???])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**i)** Notice that the list of `temperatures` is the same length as the list of `cities`, that's because these are (roughly) average temperatures for each city! Given this, how do you print: \"The average temperature in Manchester is 13.4 degrees.\" _without_ doing any of the following:\n",
    "1. Using a list index directly (*i.e.* `cities[2]` and `temperatures[2]`) or \n",
    "2. Hard-coding the name of the city? \n",
    "\n",
    "**To put it another way, neither of these solutions is the answer**:\n",
    "```python\n",
    "print(\"The average temperature in Manchester is \" + str(temperatures[2]) + \" degrees.\")\n",
    "```\n",
    "or \n",
    "```python\n",
    "city=2\n",
    "print(\"The average temperature in \" + cities[city] + \" is \" + str(temperatures[city]) + \" degrees.\")\n",
    "```\n",
    "\n",
    "_Hint:_ you need to combine some of the ideas we've used above!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "city=\"Manchester\" # Use this to get the solution..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now copy+paste your code and change only **one** thing in order to print out: \"The average temperature in Belfast is 15.2 degrees\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Manipulating Multiple Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create two lists for the next set of questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list1 = [1, 2, 3]\n",
    "list2 = [4, 5, 6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**j)** How do you get Python to print: `[1, 2, 3, 4, 5, 6]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( ??? )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**k)** How to you get Python to print: `[1, 2, 3, [4, 5, 6]]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "???\n",
    "print( list1 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's re-set the lists (run the next code block)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list1 = [1, 2, 3]\n",
    "list2 = [4, 5, 6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**l)** How would you print out: `[6, 5, 4, 3, 2, 1]` ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list3 = ???\n",
    "list3.???\n",
    "print(list3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**m)** How would you print out: `[3, 2, 1, 6, 5, 4]` ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list1.???\n",
    "list2.???\n",
    "print( list1+list2 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**n)** How would you print out `[3, 2, 6, 5]` with a permanent change to the list (not slicing)? NB: this follows on from the previous question, so note that the order is still 'reversed'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list1.???\n",
    "list2.???\n",
    "print( list1+list2 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dictionaries\n",
    "\n",
    "Remember that dictionaries (a.k.a. dicts) are like lists in that they are [data structures](https://docs.python.org/2/tutorial/datastructures.html) containing multiple elements. A key difference between [dictionaries](https://docs.python.org/2/tutorial/datastructures.html#dictionaries) and [lists](https://docs.python.org/2/tutorial/introduction.html#lists) is that while elements in lists are ordered, dicts are unordered. This means that whereas for lists we use integers as indexes to access elements, in dictonaries we use 'keys' (which can multiple different types; strings, integers, etc.). Consequently, an important concept for dicts is that of key-value pairs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Creating an Atlas\n",
    "\n",
    "Replace `???` in the following code block to make the code work as instructed in the comments. If you need some hints and reminders, revisit [Code Camp Lesson 7](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code and check you understand what the data structure that is being created (the data for each city are latitude, longitude and airport code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cities = {\n",
    "    'San Francisco': [37.77, -122.43, 'SFO'],\n",
    "    'London': [51.51, -0.08, 'LDN'],\n",
    "    'Paris': [48.86,2.29, 'PAR'],\n",
    "    'Beijing': [39.92,116.40 ,'BEI'],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Add a record to the dictionary for Chennai ([data here](https://en.wikipedia.org/wiki/Chennai))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cities???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** In *one* line of code, print out the airport code for Chennai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(???)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Check you understand the difference between the following two blocks of code by running them, checking the output and editing them (e.g. try the code again, but replacing *Berlin* with *London*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cities['Berlin'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cities.get('Berlin'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** Adapting the code below, print out the city name and airport code for every city in our Atlas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k, v in cities.items():\n",
    "    print(k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loops\n",
    "\n",
    "Recall from [Code Camp Lesson 8](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/) that loops are a way to *iterate* (or repeat) chunks of code. The two most common ways to iterate a set of commands are the `while` loop and the `for` loop. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Working with Loops\n",
    "The questions below use `for` loops. Replace `???` in the following code block to make the code work as instructed in the comments. If you need some hints and reminders, revisit [Code Camp Lesson 8](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Print out the name and latitude of every city in the cities dictionary using a for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ??? in cities.???:\n",
    "    print(??? + \" is at latitude \" + str(???))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Print out every city on a separate line using a for loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in ???:\n",
    "    print(???)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Now print using a loop this new data structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "citiesB = [\n",
    "    {'name':     'San Francisco',\n",
    "     'position': [37.77, -122.43],\n",
    "     'airport':  'SFO'},\n",
    "    {'name':     'London',\n",
    "     'position': [51.51, -0.08],\n",
    "     'airport':  'LDN'},\n",
    "    {'name':     'Paris',\n",
    "     'position': [48.86, 2.29],\n",
    "     'airport':  'PAR'},\n",
    "    {'name':     'Beijing',\n",
    "     'position': [39.92, 116.40],\n",
    "     'airport':  'BEI'}\n",
    "]\n",
    "\n",
    "for ??? in citiesB.???:\n",
    "    print(??? + \" is at latitude \" + str(???))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tackling Programming Problems\n",
    "\n",
    "Let's now think about how to approach problems in programming (with code) and how that might differ from other ways of thinking. \n",
    "\n",
    "The problem we will use here as an example is: _download a data file that we know is hosted on a web site and then do some analysis of those data_.  \n",
    "\n",
    "In this example, we are going to access and analyse a data set that we created: [Cities from Wikipedia Data](https://raw.githubusercontent.com/kingsgeocomp/geocomputation/master/CitiesWithWikipediaData-simple.csv). Have a look at those data in your browser now by pasting the URL into a new tab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyse the Problem\n",
    "\n",
    "The first step to writing a program is thinking about your goal and the steps required to achieve that. We _**don't**_ write programs like we write essays: all at once by writing a whole lot of code and then hoping for the best when we hit 'submit'. \n",
    "\n",
    "When you're tackling a programming problem you break it down into separate, simpler steps, and then tick them off one by one. Doing this gets easier as you become more familiar with programming, but it remains crucial and, in many cases, good programmers in large companies spend more time on _design_ than they do on actual _coding_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions & Packages\n",
    "\n",
    "We saw in [Code Camp Lesson 9](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/) how _functions_ are a useful programming tool to enable us to re-use chunks of code. Basically, a function is a way to do something to something in a portable, easy-to-use little bundle of code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some steps in a program are done so many times by so many people that, eventually, someone writes a _package_ that bundles up those operations into something easy to use that saves _you_ having to figure out the gory details. Reading a file (even one on a computer halfway round the world) is one of those things. Analysing a data file for you is probably not.\n",
    "\n",
    "![xkcd: Easy vs. Hard](https://imgs.xkcd.com/comics/tasks.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To a computer, reading data from a remote location (e.g. a web site halfway around the world) is not really any different from reading one that's sitting on your your local hard drive (e.g. on your desktop). To simplify things a great deal: the computer really just needs to know the location of the file and an appropriate _protocol_ for accessing that file (_e.g._ http, https, ftp, local...) and then a clever programming language like Python will typically have packages that can kind of take of the rest. \n",
    "\n",
    "In all cases -- local and remote -- you use the package to handle the hard bit of knowing how to actually 'read' data (because all files are just `1`s and `0`s of data) at the _device_ level and then Python gives you back a 'file handle' that helps you to achieve things like 'read a line' or 'close an open file'. You can think of a filehandle as something that gives you a 'grip' on a file-like object no matter where or what it is, and the package is the way that this magic is achieved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q.How is a package different from a function?\n",
    "\n",
    "**A**: A package is just a bundle of useful functions. \n",
    "\n",
    "There's a little more to it than this (as we'll see later in the term), but simplest way to think of this is that when we type `import foo` then we are including functions from the `foo` package in our code. You'll see how this works below.\n",
    "\n",
    "The point of the bundle-of-functions is that they can help us to achieve quite a lot very quickly since we don't need to reinvent the wheel and can just make use of someone else's code. In the same way that we won't mark you down for Googling the answer to a coding question, we won't mark you down for using someone else's package to help you get going with your programming. _**That's the whole point!**_\n",
    "\n",
    "Often, if you're not sure where to start Google (or StackOverflow) is the place to go:\n",
    "\n",
    "[`how to read text file on web server python`](https://www.google.co.uk/search?q=how+to+read+text+file+on+web+server+python&oq=how+to+read+text+file+on+web+server+python&aqs=chrome..69i57.629j0j7&sourceid=chrome&ie=UTF-8)\n",
    "\n",
    "Boom!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q: But why have packages at all, why not just copy and paste all the functions that we need into our notebook directly? After all, that way it is simplest, right?\n",
    "\n",
    "**A**: Because it keeps our code 'clean' and allows us to have functions with the *same name* that do different things. \n",
    "\n",
    "Imagine the following:\n",
    "\n",
    "- You want to write a program to read data from two different web sites (_e.g._ [NOMIS](https://www.nomisweb.co.uk/default.asp) and the [London Data Store](https://data.london.gov.uk/)) \n",
    "- It turns out that both already have handy Python packages that help you to read the data if you give them the URL of the page to read\n",
    "- Even better, *both* packages have a simple function called `read_data(...)` that takes a URL and extracts the data that you want from that page as a CSV file, except...\n",
    "- Because the web sites are _completely_ different the `read_data(...)` functions are _specific_ to that one web site!\n",
    "\n",
    "So, if both packages have functions callend `read_data()`, then after you have done:\n",
    "```python\n",
    "import nomis_reader\n",
    "import ldn_data_store_reader\n",
    "```\n",
    "How does Python know which `read_data` you want to use? \n",
    "\n",
    "The answer is the _namespace_: you can _only_ access the NOMIS `read_data` function by typing `nomis.read_data(...)` and you can _only_ access the Data Store `read_data` function by typing `ldn_data_store_reader.read_data(...)`. \n",
    "\n",
    "#### Import... As...\n",
    "\n",
    "The length and awkwardness of that last line is why you can write this:\n",
    "```python\n",
    "import ldn_data_store_reader as ldn\n",
    "```\n",
    "Because the _alias_ (`\"...as ldn\"`) lets you type `ldn.read_data(...)` and Python will know exactly which function you mean!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back to our Regularly Scheduled Programme\n",
    "\n",
    "OK, so back to our problem. We need to break this seemingly hard problem down into something simpler and can do this by thinking about it as three separate steps:\n",
    "\n",
    "1. First we want to read a remote file (i.e. a text file somewhere the planet), \n",
    "2. Then we want to turn it into a local data structure (i.e a list or a dictionary), \n",
    "3. Finally we want to perform some calculations on the data (e.g. calculate the mean, find the easternmost city, etc.).\n",
    "\n",
    "We can tackle each of those in turn, getting the first bit working, then adding the second bit, etc. It's just like using lego to build something: you take the same pieces and assemble them in different ways to produce different things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Reading a Remote File\n",
    "\n",
    "So, as I said, in Step \\#1 we are going to download a file hosted on a remote web site at [Cities from Wikipedia Data](https://raw.githubusercontent.com/kingsgeocomp/geocomputation/master/CitiesWithWikipediaData-simple.csv) (this can also be stored as a bit-link to make it easier to copy+paste and avoid _really_ long lines: http://bit.ly/2vrUFKi)\n",
    "\n",
    "We aren't going to to try to turn it into data or otherwise make 'sense' of it yet, we just want to **get** it. We are then going to build from this first step towards more substantial exercises and, eventually, you could easily request Megabytes of data in real-time according to flexibly-specified parameters!\n",
    "\n",
    "Because we're accessing data from a URL we will use the `urlopen` [function](https://docs.python.org/3.0/library/urllib.request.html?highlight=urlopen#urllib.request.urlopen) from the `urllib.request` [package](https://docs.python.org/3.0/library/urllib.request.html). \n",
    "\n",
    "If you're wondering how we know to use this function and package, you might google something like: _read remote csv file python 3_ which in turn might get you to a StackOverflow question and anwer like [this](https://stackoverflow.com/questions/36965864/opening-a-url-with-urllib-in-python-3). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting help with packages\n",
    "\n",
    "Of course, just knowing that you need `urlopen` doesn't necessarily help you to actually _use_ it. In addition to finding example code on StackOverflow, you can also ask the package itself for help with `dir` and `help`.\n",
    "\n",
    "##### dir\n",
    "\n",
    "The 'Dive into Python' web site will tell you that \"dir returns a list of the attributes and methods of any object\". That introduces yet another term ('modules') that we don't want to get into right now, but _everything_ in Python is an object and so `dir` will give you help with packages, variables, functions... you name it.\n",
    "\n",
    "```python\n",
    "import math\n",
    "dir(math)\n",
    "```\n",
    "\n",
    "What **`dir`** gives you is information about things you can potentially do: it's like navigating the menu of a web site -- you aren't yet looking at the information you need, you're trying to figure out if the site even _has_ what you need. So `dir` on a package will give you a list of the functions (and any variables) that the person who created the package has provided.\n",
    "\n",
    "Typically, the information given by `dir` is highly abbreviated and is really just a prelude to using `help`.\n",
    "\n",
    "##### help\n",
    "\n",
    "The `help` function gives you the actual detail you need about how to use a particular function: what are the inputs, what are the outputs, and what will the function actually _do_?\n",
    "\n",
    "Both of these can be used on a package, a function, or a variable that you've created; for example: \n",
    "```python\n",
    "import math\n",
    "help(math.acos)\n",
    "```\n",
    "\n",
    "Let's see this in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "print(\"dir(urlopen) returns:\\n\")\n",
    "print(dir(urlopen))\n",
    "print(\"\\n\\n\")\n",
    "print(\"help(urlopen) returns:\\n\")\n",
    "print(help(urlopen)) # Notice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: you can also get help in Jupyter by typing `?urlopen` in a code block and then hitting 'run'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "?urlopen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "\n",
    "# Given the info you were given above, what do you \n",
    "# think the value of 'url' should be? What\n",
    "# type of variable is it? int or string? \n",
    "url = ???\n",
    "\n",
    "# Read the URL stream into variable called 'response'\n",
    "# using the function that we imported above\n",
    "response = ???(url)\n",
    "\n",
    "#now read from the stream, decoding so that we get actual text\n",
    "datafile = response.read().decode('utf-8')\n",
    "\n",
    "print(\"datafile variable is of type: '\" + datafile.__class__.__name__ + \"'.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the _datafile_ variable is of type `string` (because we decoded it as such).  If we hadn't decoded it, the result would have been of type `bytes` which wouldn't be as easy for us (humans) to work with. \n",
    "\n",
    "Now that we've read our data as text, we can print it to check. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(datafile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this is definitely text, but it doesn't look like the [data we see at the URL itself](http://bit.ly/2vrUFKi) which has individual data records on each line. To split the text into individual lines, we can use the handily named `.splitlines()` method (more on methods below):  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://bit.ly/2vrUFKi\"\n",
    "\n",
    "response = urlopen(url)\n",
    "datafile = response.read().decode('utf-8').splitlines()\n",
    "\n",
    "print(\"datafile variable is of type: '\" + datafile.__class__.__name__ + \"'.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note now, how the _data_ variable has type `list`. So to view the data as we see them in the original online file, we can now use a `for` loop to print out each element of the `list` (each element being a row of the original online file):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(datafile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a little hard to read, though something has clearly changed. Let's try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(datafile[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last row should be `10,Sheffield,10,-163545.3257,7055177.403,685368`.\n",
    "\n",
    "If you've managed to get the code above to run and have received 11 rows of text in response to your `urlopen` query then, congratulations, you've now read a text file sitting on a server in, I think, Alberta, Canada and Python _didn't care_. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Turning Text into Data\n",
    "\n",
    "We now need to work on turning the response we got to our `urlopen` request into useful data. You'll notice that we are dealing with a _CSV_ (Comma-Separated Value) file and that the format is quite simple since none of the rows have fields that *themselves* contain commas. So to turn this into data we just need to _split_ the row into separate fields using the commas.\n",
    "\n",
    "In the code below, `dir('string')` lists the available function for strings (because `'string'` is itself a _String_; we could just as easily written `dir('foo')` or `dir('supercalifragilisticexpialidocious')` because 'foo' and 'supercalifragilisticexpialidocious' are also strings and so have the _same_ functions available. \n",
    "\n",
    "In the output below, the functions that start and end with `__` are generally considered private, so you can skip over these and focus on the ones further down that are designed to be useful to programmers. Can you spot the method that is most likely to be useful?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Brief Musical Interlude\n",
    "\n",
    "Just in case you need help pronouncing supercalifragilisticexpialidocious:\n",
    "\n",
    "[![Mary Poppins](http://img.youtube.com/vi/uZNRzc3hWvE/0.jpg)](https://youtu.be/uZNRzc3hWvE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that you can find out what _methods_ are supported by a string using `dir(<string>)`:\n",
    "```python\n",
    "dir('supercalifragilisticexpialidocious')\n",
    "```\n",
    "I'm going to save you some time (_this_ time!) and tell you that we're interested in the `split` method. Why not use the `help` function to figure out how to make use of it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help('supercalifragilisticexpialidocious'.split)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, using the output of the `help` command, how would you use `split` to turn that word into a list like this: \n",
    "```python\n",
    "['sup','rcalifragilisticexpialidocious']\n",
    "```\n",
    "If you replace the `???` with the right bits of code then running the block below will print out \"You got it!\". You *only* need to change the `???` and nothing else!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if ['sup','rcalifragilisticexpialidocious']=='supercalifragilisticexpialidocious'.split(???, maxsplit=???):\n",
    "    print(\"You got it!\")\n",
    "else:\n",
    "    print(\"Not yet!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some other string methods\n",
    "print('supercalifragilisticexpialidocious'.upper())\n",
    "print('supercalifragilisticexpialidocious'.title())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so you've tracked down the way to split a `string` using a delimiter and _even_ how to limit the number of 'words' that come out of the split operation. And you already saw another of these methods above (i.e. `splitlines`). We work a lot with `string`s, so it's handy to get to know the readily-available methods well.\n",
    "\n",
    "Let's test `string` splitting using our sample data (the last line of the 'simple' CSV file) to make sure it works the way we think it does... We want to turn the `string` below into a list like this:\n",
    "```python\n",
    "['10', 'Sheffield', '10', '-163545.3257', '7055177.403', '685368']\n",
    "```\n",
    "Again, we only need to change the `???`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = datafile[-1].split(???)\n",
    "print(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Consider\n",
    "\n",
    "A question: why do you think that I consider \n",
    "```python\n",
    "['10', 'Sheffield', '10', '-163545.3257', '7055177.403', '685368']\n",
    "```\n",
    "to be data and not\n",
    "```python\n",
    "\"10,Sheffield,10,-163545.3257,7055177.403,685368\"\n",
    "```\n",
    "\n",
    "Here's a clue:\n",
    "```python\n",
    "print(\"The population of \" + myList[1] + \" is \" + myList[5])\n",
    "```\n",
    "\n",
    "Now that you've figured out how to make use of the appropriate method using `help` and a simple test, it's time to revise the code above so that it turns the remote file into data. You can hopefully see how we're breaking a complex problem down into a set of _increments_, each of which is a bit easier to write and understand. \n",
    "\n",
    "Now, remember that ultimately we want to make use of the data in this file, so simply printing it back out isn't particularly helpful. What we really want to do is stash the data we've read in some kind of _data structure_ that resembles the CSV file but is easier and faster for the computer to navigate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The CSV Package\n",
    "\n",
    "Notice something subtle: we are implicitly _assuming_ that commas only appear at the boundaries between elements of a row. In other words, *none* of the fields in the data can containg a comma. Why is that? Before you continue reading, take a moment to think about what `split(',')` does and why it won't work well with a line of data from Wales that looks like this:\n",
    "```python\n",
    "11,\"Cardiff,Caerdydd\",11,51.483333,-3.183333,447287\n",
    "```\n",
    "Let's try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('11,\"Cardiff,Caerdydd\",11,51.483333,-3.183333,447287'.split(','))\n",
    "print(\"\")\n",
    "print('10,\"Sheffield\",10,-163545.3257,7055177.403,685368'.split(','))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you see the problem? Look at the data type of the elements in the list. Do you think they are appropriate if we wanted to work with numbers?\n",
    "\n",
    "Will this code still work:\n",
    "\n",
    "```python\n",
    "for c in cityData:\n",
    "    print(\"The population of \" + c[1] + \" is \" + c[5])\n",
    "```\n",
    "\n",
    "This is where using code that someone _else_ has written and contributed is helpful: we don't need to think through how to deal with this sort of thing ourselves, we can just import the library that we need and make use of its functionality. In the _simple_ file there are no examples of this issue, but there are in the _full_ data set: we always try to start simple and build from there...\n",
    "\n",
    "I've given you the skeleton of the answer below, but you'll need to do a little Googling to find out how to `\"read csv python\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thinking About Data Structures\n",
    "\n",
    "Instinctively, it seems like we need:\n",
    "\n",
    "1. To keep the rows in order\n",
    "2. To keep the columns in order\n",
    "\n",
    "So if we wanted to do that then the 'right' data structure would clearly be a list-of-lists (LoLs!):\n",
    "```python\n",
    "data = [\n",
    "    ['id', 'Name',           'Rank', 'Longitude',    'Latitude',    'Population'],\n",
    "    ['1',  'Greater London', '1',    '-18162.92767', '6711153.709', '9787426'],\n",
    "    ['10', 'Sheffield',      '10',   '-163545.3257', '7055177.403', '685368']\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "import csv\n",
    "\n",
    "url = \"http://bit.ly/2vrUFKi\" # A bit-link to save space\n",
    "\n",
    "urlData = [] # Somewhere to store the data\n",
    "\n",
    "response = urlopen(url)        # Get the data using the urlopen function\n",
    "csvfile  = csv.reader(response.read().decode('utf-8').splitlines()) # Pass it over to the reader function\n",
    "\n",
    "for row in csvfile:              \n",
    "    urlData.append( row )\n",
    "    \n",
    "print(urlData) # Check it worked!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If it worked, then you should have this output:\n",
    "```python\n",
    "[['id', 'Name', 'Rank', 'Longitude', 'Latitude', 'Population'], ['1', 'Greater London', '1', '-18162.92767', '6711153.709', '9787426'], ['2', 'Greater Manchester', '2', '-251761.802', '7073067.458', '2553379'], ['3', 'West Midlands', '3', '-210635.2396', '6878950.083', '2440986'], ['4', 'West Yorkshire', '4', '-185959.3022', '7145450.207', '1777934'], ['5', 'Glasgow', '5', '-473845.2389', '7538620.144', '1209143'], ['6', 'Liverpool', '6', '-340595.1768', '7063197.083', '864122'], ['7', 'South Hampshire', '7', '-174443.8647', '6589419.084', '855569'], ['8', 'Tyneside', '8', '-187604.3647', '7356018.207', '774891'], ['9', 'Nottingham', '9', '-131672.2399', '6979298.895', '729977'], ['10', 'Sheffield', '10', '-163545.3257', '7055177.403', '685368']]\n",
    "```\n",
    "To you that might look a lot _worse_ that the data that you originally had, but to a computer that list-of-lists is something it can work with; check it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in urlData:                                      # For each row in the list\n",
    "    print(\"The population of \" + c[1] + \" is \" + c[5]) # Print out the name and population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The advantage of using the `csv` library over plain old `string.split` is that the csv library knows how to deal with fields that contain commas (_e.g._ `\"Cardiff,Caerdydd\"`) or even newlines and so is much more flexible and consistent that our naive `split` approach. The vast majority of _common_ tasks (reading certain types of files, getting remote files, etc.) have libraries that do exactly what you want without you needing to write much code yourself to take advantage of it. You should always have a look around online to see if a library exists before thinking that you need to write everything/anything from scratch. The tricky part is knowing what words to use for your search and how to read the answers that you find...\n",
    "\n",
    "Let's try this with the 'full' data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "import csv\n",
    "\n",
    "url = \"http://bit.ly/2iIK9bA\" # A bit-link to save space\n",
    "\n",
    "urlData = [] # Somewhere to store the data\n",
    "\n",
    "response = urlopen(url)\n",
    "csvfile = csv.reader(response.read().decode('utf-8').splitlines())\n",
    "\n",
    "for row in csvfile:              \n",
    "    urlData.append( row )\n",
    "    \n",
    "for c in urlData:                                      # For each row in the list\n",
    "    print(\"The population of \" + c[1] + \" is \" + c[5]) # Print out the name and population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hmmmm, is that right???\n",
    "\n",
    "Have a think about why doing the same thing that worked on one data set might not work on a different one. If you understand what happened here then the next section will make a _lot_ more sense!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why 'Obvious' is Not Always 'Right'\n",
    "\n",
    "But you need to be careful assuming that, just because something is hard for you to read, it's also hard for a computer to read! I've said before that the way a computer 'thinks' and the way that we think doesn't always line up naturally. Experienced programmers can think their way _around_ a problem by working _with_ the computer, rather than against it.\n",
    "\n",
    "Some issues to consider:\n",
    "\n",
    "- Is the first row of 'data' actually data?\n",
    "- Do we really care about column _order_, or do we just care about being able to pick the *correct* column?\n",
    "- Does the LoL approach deal with data efficiently? \n",
    "\n",
    "Let's apply this approach to the parsing of our data..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What's an _Appropriate_ Data Structure?\n",
    "\n",
    "If you look closely, then using either of the 'Cities' data files to do anything _useful_ presents something of a 'problem': our list-of-lists isn't very easy to navigate. Notice that not only _might_ the location of the Population column be different in the two files (as it turns out it was, deliberately), but when we want to work out the answer to a simple question such 'what is the 3rd largest city' we need to step through a lot of irrelevant data as well: we'd need to write a for loop and make sure that we skip past the name, latitude, longitude, etc. \n",
    "\n",
    "That doesn't make much sense since this should all be easier and faster in code than in Excel, but right now it's _harder_, and quite possibly _slower_ as well! When you get into situations like this (having to write a lot of code to do something that should be fast and easy) it is often the case that you've got the wrong _data structure_. \n",
    "\n",
    "So how does the experienced programmer get around this? 'Simple' (i.e. neither simple, nor obvious, until you know the answer): she realises that the data is organised the wrong way! We humans naturally tend to think in rows of data: London has the following _attributes_ (population, location, etc.), and York has a different set of attributes. Se we read across the row because that's the easiest way for us to read it. But, in short, a list-of-lists does _not_ seem to be the right way to store this data!\n",
    "\n",
    "Crucially, a computer doesn't have to work that way. For a computer, it's as easy to read _down_ a column as it is to read _across_ a row. In fact, it's easier, because each column has a consistent _type_ of data: one column contains names (strings), another column contains populations (integers), and other columns contain other types of data (floats, etc.). \n",
    "\n",
    "Better still, the order of the columns often doesn't matter as long as we know what they are called: it's easier to ask for the 'population column' than it is to ask for the 6th column since, for all we know, the population column might be in a different place for different files but they are all (relatively) likely to use the 'population' label for the column itself.\n",
    "\n",
    "#### A Dictionary of Lists\n",
    "\n",
    "So, if we don't care about column order, only row order, then a dictionary of lists would be a nice way to handle things. And why should we care about column order? With our two CSV files above we already saw what a pain it was to fix things when the layout of the columns changed from one data set to the next. If, instead, we can just reference 'population' column then it doesn't matter where that column actually is. Why is that? \n",
    "\n",
    "Well, here are the first four rows of data from the simple city file as a list-of-lists:\n",
    "\n",
    "```python\n",
    "['id', 'Name', 'Rank', 'Longitude', 'Latitude', 'Population'], \n",
    "['1', 'Greater London', '1', '-18162.92767', '6711153.709', '9787426'], \n",
    "['2', 'Greater Manchester', '2', '-251761.802', '7073067.458', '2553379'], \n",
    "['3', 'West Midlands', '3', '-210635.2396', '6878950.083', '2440986']\n",
    "```\n",
    "\n",
    "Now, here's how it would look as a dictionary of lists organised by _column_, not by row:\n",
    "\n",
    "```python\n",
    "myData = {\n",
    "    'id'         : [1, 2, 3],\n",
    "    'Name'       : ['London', 'Manchester', 'West Midlands'],\n",
    "    'Rank'       : [1, 2, 3],\n",
    "    'Longitude'  : [-18162.92767, -251761.802, -210635.2396],\n",
    "    'Latitude'   : [6711153.709, 7073067.458, 6878950.083],\n",
    "    'Population' : [9787426, 2553379, 2440986],\n",
    "}\n",
    "\n",
    "```\n",
    "\n",
    "What does this do better? Well, for starters, we know that everything in the 'Name' column will be a string, and that everything in the 'Longitude' column is a float, while the 'Population' column contains integers. So that's made life easier already. But let's test this out and see how it works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Do some calculations\n",
    "\n",
    "Now we have our final data structure (which next we will see is already available via the Pandas package), we can do some calculations.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myData = {\n",
    "    'Name'       : ['London','Manchester','West Midlands'],\n",
    "    'Rank'       : [1, 2, 3, 4],\n",
    "    'Longitude'  : [-18162.92767, -251761.802, -210635.2396],\n",
    "    'Latitude'   : [6711153.709, 7073067.458, 6878950.083],\n",
    "    'Population' : [9787426, 2553379, 2440986],\n",
    "}\n",
    "\n",
    "# Find the population of Manchester\n",
    "pop = myData['Population'][myData['Name'].index('Manchester')]\n",
    "print(\"The population of Manchester is: \" + str(pop))\n",
    "\n",
    "# Find the easternmost city\n",
    "city = myData['Name'][myData['Longitude'].index(max(myData['Longitude']))]\n",
    "print(\"The easternmost city is: \" + str(city))\n",
    "\n",
    "# Find the mean population of the cities\n",
    "import numpy as np # Need to import a useful package\n",
    "mean = np.mean(myData['Population'])\n",
    "print(\"The mean population is: \" + str(mean))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a _lot_ of content to process in the code above, so do _not_ rush blindly on if this is confusing. \n",
    "\n",
    "**Stop. Think it through. Talk it out with your neighbour and lecturers.** \n",
    "\n",
    "We'll go through each one in turn, but they nearly all work in the same way and the really key thing is that you'll notice that we no longer have any loops (which are slow) just `index` (which is very fast). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Population of Manchester\n",
    "\n",
    "The code can look pretty daunting, so let's break it down into two parts.\n",
    "\n",
    "What would you get if you ran just this code?\n",
    "```python\n",
    "myData['Population'][0]\n",
    "```\n",
    "Remember that this is a dictionary-of-lists (DoL). So, Python first looks for a key named `Population` in the myData dictionary. It finds out that the value associated with this key is a _list_ (`[9787426, 2553379, 2440986]`). In this example, it just pulls out the first value (index 0), which is `9787426`. Does that make sense?\n",
    "\n",
    "Now, to the second part:\n",
    "```python\n",
    "myData['Name'].index('Manchester')\n",
    "```\n",
    "\n",
    "This is very similar: we look in the dictionary for the key `Name` and find that that's _also_ a list (`['London','Manchester','West Midlands']`, since you asked). If you don't remember what `index` does, don't worry, here's the output from Python's `help()` function:\n",
    "```\n",
    "Help on built-in function index:\n",
    "\n",
    "index(...)\n",
    "    L.index(value, [start, [stop]]) -> integer -- return first index of value.\n",
    "    Raises ValueError if the value is not present.\n",
    "```\n",
    "So all we're doing is asking Python to find out the index of 'Manchester' in the list associated with the dictionary key 'Name' _instead_ of just sticking in a `0` to get the first index value. Putting these two things back together what we're doing is:\n",
    "\n",
    "* Finding the index (i.e. **row**) of 'Manchester' in the Name column,\n",
    "* Using that index to read a value out of the Population column.\n",
    "\n",
    "Notice the complete _absence_ of a for loop?\n",
    "\n",
    "Does that make sense? If it does then you should be having a kind of an Alice-through-the-Looking-Glass moment because what we've done by taking a column view, rather than a row view is to make Python's ``index()`` command do the work for us. Instead of having to look through each row for a field that matches 'Name' and then check to see if it's 'Manchester', we've pointed Python at the right column immediately and asked it to find the match (which it can do very quickly). Once we have a match then we _also_ have the row number to go and do the lookup in the 'Population' column because the index _is_ the row number!\n",
    "\n",
    "### The Easternmost City\n",
    "\n",
    "Where this approach really comes into its own is on problems that involve maths. To figure out the easternmost city in this list we need to find the _maximum_ Longitude and then use _that_ value to look up the city name. So let's do the same process of pulling this apart into two steps:\n",
    "\n",
    "It should be _pretty_ obvious what this does:\n",
    "```python\n",
    "myData['Name'][0]\n",
    "```\n",
    "\n",
    "But we don't just want the first city in the list, we want the one with the highest longitude. So to achieve that we need to replace the `0` with an index that we found by looking in the `Longitude` list.\n",
    "```python\n",
    "myData['Longitude'].index(max(myData['Longitude']))\n",
    "```\n",
    "\n",
    "Ugh, that's still a little hard to read, isn't it? Let's write it down another way to make it easier to read:\n",
    "\n",
    "```python\n",
    "myData['Longitude'].index(\n",
    "    max(myData['Longitude'])\n",
    ")\n",
    "```\n",
    "\n",
    "There's the same `.index` which tells us that Python is going to look for something in the list associated with the `Longitude` key. All we've done is change what's _inside_ that index function to `max(myData['Longitude'])`. This is telling Python to find the _maximum_ value in the `myData['Longitude']` list. So to explain this in three steps, what we're doing is:\n",
    "* Finding the maximum value in the Longitude column (we know there must be one, but we don't know what it is!),\n",
    "* Finding the index (position) of that maximum value in the Longitude column (now that we know what the value is!),\n",
    "* Using that index to read a value out of the Name column.\n",
    "\n",
    "I _am_ a geek, but that's pretty cool, right? In one line of code we managed to quickly find out where the data we needed was even though it involved three discrete steps. Remember how much work it was to find the mean when you were still thinking in _rows_, not _columns_?\n",
    "\n",
    "### The Average City Size\n",
    "\n",
    "Yeah, let's try that too.\n",
    "\n",
    "Here we're going to 'cheat' a little bit: rather than writing our own function, we're going to import a package and use someone _else's_ function. The `numpy` package contains a _lot_ of useful functions that we can call on (if you don't believe me, add \"`dir(np)`\" on a new line after the `import` statement), and one of them calculates the average of a list or array of data.\n",
    "```python\n",
    "import numpy as np # Need to import a useful package\n",
    "mean = np.mean(myData['Population'])\n",
    "```\n",
    "This is where our new approach really comes into its own: because all of the population data is in one place (a.k.a. a _series_ or column), we can just throw the whole list into the `np.mean` function rather than having to use all of those convoluted loops and counters. Simples, right?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review!\n",
    "\n",
    "So the _really_ clever bit in all of this isn't switching from a list-of-lists to a dictionary-of-lists, it's recognising that the latter is a _better_ way to work _with_ the data that we're trying to analyse and that that there are useful functions that we can exploit to do the heavy lifting for us. Simply by changing the way that we stored the data in a 'data structure' (i.e. complex arrangement of lists, dictionaries, and variables) we were able to do away with lots of for loops and counters and conditions, and reduce many difficult operations to something that could be done on one line! \n",
    "\n",
    "So now, finally, we can bundle up our code into our own function to read data from a URL and report some calculations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "import csv\n",
    "import numpy as np\n",
    "\n",
    "urlData = [] # Somewhere to store the data\n",
    "\n",
    "# define our function\n",
    "def read_city_data(url):\n",
    "    \"\"\"\n",
    "    Reads a remote CSV file of city and returns\n",
    "    a dictionary-of-lists containing the data.\n",
    "    \"\"\"\n",
    "    urlData = {} # Somewhere to store the data\n",
    "\n",
    "    response = urlopen(url)\n",
    "    csvfile = csv.reader(response.read().decode('utf-8').splitlines())\n",
    "    \n",
    "    headers = next(csvfile)\n",
    "    \n",
    "    for h in headers:\n",
    "        urlData[h] = []\n",
    "    \n",
    "    for row in csvfile: \n",
    "        for c in range(0,len(row)):\n",
    "            urlData[headers[c]].append(row[c])\n",
    "    \n",
    "    for col in ['id','Rank','Population']:\n",
    "        urlData[col] = list(map(int, urlData[col])) # list(map(int, results)) in Python 3\n",
    "        \n",
    "    for col in ['Latitude','Longitude']:\n",
    "        urlData[col] = list(map(float, urlData[col]))\n",
    "    \n",
    "    return urlData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the function we have defined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"URL 1:\")\n",
    "data1 = read_city_data(\"http://bit.ly/2vrUFKi\")\n",
    "print(\"\\tThe average population is data1 is \" + str(np.mean(data1['Population'])))\n",
    "\n",
    "print(\"URL 2:\")\n",
    "data2 = read_city_data(\"http://bit.ly/2iIK9bA\")\n",
    "print(\"\\tThe average population is data2 is \" + str(np.mean(data2['Population'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the code to read the population is the _same_ for both data sets (unlike our old LoL with for loops)? See if you can use the code that we saw above to work out the Westernmost city in Britain, the 25th largest city... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bringing it all together...\n",
    "\n",
    "The function we have created at the end here is the key to the entire practical since it brings together all of the ideas that we've covered this session:\n",
    "\n",
    "- We've used for loops, dictionaries, and lists to generate a data structure\n",
    "- We'be used no fewer that three packages to do complicated things for us\n",
    "- We've wrapped some complex operations up in a function that we can re-use\n",
    "- We've seen how clever ways to do things can make our lives much, much easier\n",
    "\n",
    "If this didn't make sense to you on the _first_ runthrough, I'd suggest going back through the latter half of the practical _again_ in a couple of days' time -- that will give your brain a little time to wrap itself around the basics before you throw the hard stuff at it. _Don't_ panic if it doesn't all make sense on the _second_ runthrough either -- this is like a language, you need to practice! With luck, the second time you went through this code a little bit _more_ made sense. If you need to do it a third time you'll find that even _more_ makes sense... and so on. \n",
    "\n",
    "Conceptually, this is one of the hardest practicals in the entire term because it joins up so many of the seemingly simple ideas that you covered in Code Camp into a very complex 'stew' --our basic ingredients (lists, dictionaries, etc.) simmered for a bit and became this heady mix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Brain Teaser\n",
    "\n",
    "If you want to have a stab at writing the code to print out the 4th most populous city-region then knock yourself out! This can _still_ be done on one line and is _very_ fast even though we are now dealing with quite a bit more data.\n",
    "\n",
    "Here's a clue: you don't want to use `<list>.sort()` because that will sort your data _in place_ and break the link between the indexes across the columns; you want to use the function `sorted(<list>)` where `<list>` is the variable that holds your data and `sorted(...)` just returns whatever you pass it in a sorted order _without_ changing the original. You'll see why this matters if you get the answer... otherwise, wait a few days for the answers to post."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the name of the 4th most populous city-region\n",
    "city = data2???\n",
    "print(\"The fourth most populous city is: \" + str(city))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus Material\n",
    "\n",
    "Here is a simple demonstration of how classes work and why they're useful in programming. Building on last week's 'volume of a sphere' question here's how we'd create and work with a 'shape' class in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sphere\n",
      "\tVolume is: 4188.79\n",
      "\tDiameter is: 20.00\n",
      "\n",
      "Cube\n",
      "\tVolume is: 1000.00\n",
      "\tDiameter is: 14.14\n",
      "\n",
      "Triangular Pyramid\n",
      "\tVolume is: 117.85\n",
      "\tPhew, no mummies!\n"
     ]
    },
    {
     "ename": "Exception",
     "evalue": "Unimplmented method error.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-2566ecf3262f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     90\u001b[0m \u001b[0;31m# is not implemented in either triangular pyramid, pyramid,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m \u001b[0;31m# or shape, only for spheres.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 92\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\\tDiameter is: {0:5.2f}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiameter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     93\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-1-2566ecf3262f>\u001b[0m in \u001b[0;36mdiameter\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdiameter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Unimplmented method error.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mException\u001b[0m: Unimplmented method error."
     ]
    }
   ],
   "source": [
    "from math import pi\n",
    "\n",
    "class shape(object): # Inherit from base class \n",
    "    def __init__(self): \n",
    "        return \n",
    "    \n",
    "    def volume(self):\n",
    "        raise Exception(\"Unimplmented method error.\")\n",
    "    \n",
    "    def diameter(self):\n",
    "        raise Exception(\"Unimplmented method error.\")\n",
    "        \n",
    "    def type(self):\n",
    "        return(self.shape_type)\n",
    "    \n",
    "class cube(shape): # Inherit from shape \n",
    "    def __init__(self, e):\n",
    "        self.shape_type = 'Cube'\n",
    "        self.edge = e\n",
    "        return\n",
    "    \n",
    "    def volume(self):\n",
    "        return self.edge**3\n",
    "    \n",
    "    def diameter(self):\n",
    "        return (self.edge**2 + self.edge**2)**(1/2)\n",
    "\n",
    "class sphere(shape): # Inherit from shape \n",
    "    def __init__(self, r):\n",
    "        self.shape_type = 'Sphere'\n",
    "        self.radius = r\n",
    "        return\n",
    "    \n",
    "    def volume(self):\n",
    "        return (4/3) * pi * self.radius**3\n",
    "    \n",
    "    def diameter(self):\n",
    "        return self.radius*2\n",
    "\n",
    "class pyramid(shape): # Inherit from shape\n",
    "    \n",
    "    has_mummies = True # This is for *all* regular pyramids\n",
    "    \n",
    "    def __init__(self, e):\n",
    "        self.shape_type = 'Regular Pyramid'\n",
    "        self.edge   = e\n",
    "        return  \n",
    "\n",
    "class t_pyramid(pyramid): # Inherit from pyramid (this is a triangular pyramid)\n",
    "    \n",
    "    has_mummies = False # This is for all triangular pyramids\n",
    "    \n",
    "    def __init__(self, e):\n",
    "        self.shape_type = 'Triangular Pyramid'\n",
    "        self.edge   = e\n",
    "        return \n",
    "    \n",
    "    def area(self):\n",
    "        return (3**(1/2)/4) * self.edge**2\n",
    "    \n",
    "    def height(self):\n",
    "        # https://www.youtube.com/watch?v=ivF3ndmkMsE\n",
    "        return (6**(1/2) * self.edge/3)\n",
    "    \n",
    "    def volume(self):\n",
    "        #  V = 1/3 * A * h\n",
    "        return (1/3) * self.area() * self.height()\n",
    "\n",
    "s = sphere(10)\n",
    "print(s.type())\n",
    "print(\"\\tVolume is: {0:5.2f}\".format(s.volume()))\n",
    "print(\"\\tDiameter is: {0:5.2f}\".format(s.diameter()))\n",
    "print(\"\")\n",
    "\n",
    "c = cube(10)\n",
    "print(c.type())\n",
    "print(\"\\tVolume is: {0:5.2f}\".format(c.volume()))\n",
    "print(\"\\tDiameter is: {0:5.2f}\".format(c.diameter()))\n",
    "print(\"\")\n",
    "\n",
    "p = t_pyramid(10)\n",
    "print(p.type())\n",
    "print(\"\\tVolume is: {0:5.2f}\".format(p.volume()))\n",
    "if p.has_mummies is True:\n",
    "    print(\"\\tMummies? Aaaaaaaaagh!\")\n",
    "else:\n",
    "    print(\"\\tPhew, no mummies!\")\n",
    "\n",
    "# The error here is deliberate -- note that diameter\n",
    "# is not implemented in either triangular pyramid, pyramid, \n",
    "# or shape, only for spheres.\n",
    "print(\"\\tDiameter is: {0:5.2f}\".format(p.diameter()))\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Have a really good think about how this kind of behaviour is useful!**\n",
    "\n",
    "### Test Your Understanding \n",
    "\n",
    "Based on the above examples of classes and methods, I have two challenges for you to implement in the code below:\n",
    "1. Try fully-implementing the *regular* pyramid class with a square base that we skipped over above!\n",
    "2. Try adding an `area` method that returns the surface area of each shape and then add that information to the output below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import pi\n",
    "\n",
    "class shape(object): # Inherit from base class \n",
    "    def __init__(self): \n",
    "        return \n",
    "    \n",
    "    def volume(self):\n",
    "        raise Exception(\"Unimplmented method error.\")\n",
    "    \n",
    "    def diameter(self):\n",
    "        raise Exception(\"Unimplmented method error.\")\n",
    "        \n",
    "    def type(self):\n",
    "        return(self.shape_type)\n",
    "    \n",
    "class cube(shape): # Inherit from shape \n",
    "    def __init__(self, e):\n",
    "        self.shape_type = 'Cube'\n",
    "        self.edge = e\n",
    "        return\n",
    "    \n",
    "    def volume(self):\n",
    "        return self.edge**3\n",
    "    \n",
    "    def diameter(self):\n",
    "        return (self.edge**2 + self.edge**2)**(1/2)\n",
    "\n",
    "class sphere(shape): # Inherit from shape \n",
    "    def __init__(self, r):\n",
    "        self.shape_type = 'Sphere'\n",
    "        self.radius = r\n",
    "        return\n",
    "    \n",
    "    def volume(self):\n",
    "        return (4/3) * pi * self.radius**3\n",
    "    \n",
    "    def diameter(self):\n",
    "        return self.radius*2\n",
    "\n",
    "class pyramid(shape): # Inherit from shape\n",
    "    \n",
    "    has_mummies = True # This is for *all* regular pyramids\n",
    "    \n",
    "    def __init__(self, e):\n",
    "        self.shape_type = 'Regular Pyramid'\n",
    "        self.edge   = e\n",
    "        return  \n",
    "\n",
    "class t_pyramid(pyramid): # Inherit from pyramid (this is a triangular pyramid)\n",
    "    \n",
    "    has_mummies = False # This is for all triangular pyramids\n",
    "    \n",
    "    def __init__(self, e):\n",
    "        self.shape_type = 'Triangular Pyramid'\n",
    "        self.edge   = e\n",
    "        return \n",
    "    \n",
    "    def area(self):\n",
    "        return (3**(1/2)/4) * self.edge**2\n",
    "    \n",
    "    def height(self):\n",
    "        # https://www.youtube.com/watch?v=ivF3ndmkMsE\n",
    "        return (6**(1/2) * self.edge/3)\n",
    "    \n",
    "    def volume(self):\n",
    "        #  V = 1/3 * A * h\n",
    "        return (1/3) * self.area() * self.height()\n",
    "\n",
    "# How would you test these changes?"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "GSA2019",
   "language": "python",
   "name": "gsa2019"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}