notebook-12-packages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook-12: Introduction to Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions vs. Packages\n",
    "\n",
    "We saw in a previous Code Camp Lesson how **functions** are a useful programming tool to enable us to re-use chunks of code. Basically, a function is a way to do something to something in a portable, easy-to-use little bundle of code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, some steps in a program are done so many times by so many people that, eventually, someone writes a **package** that bundles up those operations into something easy to use that saves _you_ having to figure out the gory details. \n",
    "\n",
    "In this lesson we'll see how packages are useful in the context of tacking problems programmatically. The problem we will use here as an example is: _download a data file that we know is hosted on a web site and then do some analysis of those data_.  \n",
    "\n",
    "In this example, we are going to access and analyse a data set that we created: [Cities from Wikipedia Data](https://raw.githubusercontent.com/kingsgeocomp/geocomputation/master/CitiesWithWikipediaData-simple.csv). Have a look at those data in your browser now by pasting the URL into a new tab. We'll use the `urlopen` package to help us access the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q: How is a package different from a function?\n",
    "\n",
    "**A**: A package is just a bundle of useful functions. \n",
    "\n",
    "There's a little more to it than this, but simplest way to think of this is that when we type `import foo` then we are including functions from the `foo` package in our code. You'll see how this works below.\n",
    "\n",
    "The point of the bundle-of-functions is that they can help us to achieve quite a lot very quickly since we don't need to reinvent the wheel and can just make use of someone else's code. In the same way that we won't mark you down for Googling the answer to a coding question, we won't mark you down for using someone else's package to help you get going with your programming. _**That's the whole point!**_\n",
    "\n",
    "Often, if you're not sure where to start Google (or StackOverflow) is the place to go:\n",
    "\n",
    "[`how to read text file on web server python`](https://www.google.co.uk/search?q=how+to+read+text+file+on+web+server+python&oq=how+to+read+text+file+on+web+server+python&aqs=chrome..69i57.629j0j7&sourceid=chrome&ie=UTF-8)\n",
    "\n",
    "Boom!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q: But why have packages at all, why not just copy and paste all the functions that we need into our notebook directly? \n",
    "\n",
    "**A**: Because it keeps our code 'clean' and allows us to have functions with the *same name* that do different things. \n",
    "\n",
    "Imagine the following:\n",
    "\n",
    "- You want to write a program to read data from two different web sites (_e.g._ [NOMIS](https://www.nomisweb.co.uk/default.asp) and the [London Data Store](https://data.london.gov.uk/)) \n",
    "- It turns out that both already have handy Python packages that help you to read the data if you give them the URL of the page to read\n",
    "- Even better, *both* packages have a simple function called `read_data(...)` that takes a URL and extracts the data that you want from that page as a CSV file, except...\n",
    "- Because the web sites are _completely_ different the `read_data(...)` functions are _specific_ to that one web site!\n",
    "\n",
    "So, if both packages have functions callend `read_data()`, then after you have done:\n",
    "```python\n",
    "import nomis_reader\n",
    "import ldn_data_store_reader\n",
    "```\n",
    "How does Python know which `read_data` you want to use? \n",
    "\n",
    "The answer is the _namespace_: you can _only_ access the NOMIS `read_data` function by typing `nomis.read_data(...)` and you can _only_ access the Data Store `read_data` function by typing `ldn_data_store_reader.read_data(...)`. \n",
    "\n",
    "### Import... As...\n",
    "\n",
    "The length and awkwardness of that last line is why you can write this:\n",
    "```python\n",
    "import ldn_data_store_reader as ldn\n",
    "```\n",
    "Because the _alias_ (`\"...as ldn\"`) lets you type `ldn.read_data(...)` and Python will know exactly which function you mean!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages in Programming Context\n",
    "\n",
    "The first step to writing a program is thinking about your goal and the steps required to achieve that. We _**don't**_ write programs like we write essays: all at once by writing a whole lot of code and then hoping for the best when we hit 'submit'. \n",
    "\n",
    "When you're tackling a programming problem you break it down into separate, simpler steps, and then tick them off one by one. Doing this gets easier as you become more familiar with programming, but it remains crucial and, in many cases, good programmers in large companies spend more time on _design_ than they do on actual _coding_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To a computer, reading data from a remote location (e.g. a web site halfway around the world) is not really any different from reading one that's sitting on your your local hard drive (e.g. on your desktop). To simplify things a great deal: the computer really just needs to know the location of the file and an appropriate _protocol_ for accessing that file (_e.g._ http, https, ftp, local...) and then a clever programming language like Python will typically have packages that can kind of take of the rest. \n",
    "\n",
    "In all cases -- local and remote -- you use the package to handle the hard bit of knowing how to actually 'read' data (because all files are just `1`s and `0`s of data) at the _device_ level and then Python gives you back a 'file handle' that helps you to achieve things like 'read a line' or 'close an open file'. You can think of a filehandle as something that gives you a 'grip' on a file-like object no matter where or what it is, and the package is the way that this magic is achieved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's recall our problem: _download a data file that we know is hosted on a web site and then do some analysis of those data_.\n",
    "\n",
    "We need to break this seemingly hard problem down into something simpler and can do this by thinking about it as three separate steps:\n",
    "\n",
    "1. First we want to read a remote file (i.e. a text file somewhere the planet), \n",
    "2. Then we want to turn it into a local data structure (i.e a list or a dictionary), \n",
    "3. Finally we want to perform some calculations on the data (e.g. calculate the mean).\n",
    "\n",
    "We can tackle each of those in turn, getting the first bit working, then adding the second bit, etc. It's just like using lego to build something: you take the same pieces and assemble them in different ways to produce different things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Reading a Remote File\n",
    "\n",
    "So, as I said, in Step \\#1 we are going to download a file hosted on a remote web site at [Cities from Wikipedia Data](https://raw.githubusercontent.com/kingsgeocomp/geocomputation/master/CitiesWithWikipediaData-simple.csv) (this can also be stored as a bit-link to make it easier to copy+paste and avoid _really_ long lines: http://bit.ly/2vrUFKi)\n",
    "\n",
    "We aren't going to to try to turn it into data or otherwise make 'sense' of it yet, we just want to **get** it. We are then going to build from this first step towards more substantial exercises and, eventually, you could easily request Megabytes of data in real-time according to flexibly-specified parameters!\n",
    "\n",
    "Because we're accessing data from a URL we will use the `urlopen` [function](https://docs.python.org/3.0/library/urllib.request.html?highlight=urlopen#urllib.request.urlopen) from the `urllib.request` [package](https://docs.python.org/3.0/library/urllib.request.html). \n",
    "\n",
    "If you're wondering how we know to use this function and package, you might google something like: _read remote csv file python 3_ which in turn might get you to a StackOverflow question and anwer like [this](https://stackoverflow.com/questions/36965864/opening-a-url-with-urllib-in-python-3). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting help with packages\n",
    "\n",
    "Of course, just knowing that you need `urlopen` doesn't necessarily help you to actually _use_ it. In addition to finding example code on StackOverflow, you can also ask the package itself for help with `dir` and `help`.\n",
    "\n",
    "##### dir\n",
    "\n",
    "The 'Dive into Python' web site will tell you that \"dir returns a list of the attributes and methods of any object\". That introduces yet another term ('modules') that we don't want to get into right now, but _everything_ in Python is an object and so `dir` will give you help with packages, variables, functions... you name it.\n",
    "\n",
    "```python\n",
    "import math\n",
    "dir(math)\n",
    "```\n",
    "\n",
    "What **`dir`** gives you is information about things you can potentially do: it's like navigating the menu of a web site -- you aren't yet looking at the information you need, you're trying to figure out if the site even _has_ what you need. So `dir` on a package will give you a list of the functions (and any variables) that the person who created the package has provided.\n",
    "\n",
    "Typically, the information given by `dir` is highly abbreviated and is really just a prelude to using `help`.\n",
    "\n",
    "##### help\n",
    "\n",
    "The `help` function gives you the actual detail you need about how to use a particular function: what are the inputs, what are the outputs, and what will the function actually _do_ ?\n",
    "\n",
    "Both of these can be used on a package, a function, or a variable that you've created; for example: \n",
    "```python\n",
    "import math\n",
    "help(math.acos)\n",
    "```\n",
    "\n",
    "Let's see this in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "print(\"dir(urlopen) returns:\\n\")\n",
    "print(dir(urlopen))\n",
    "print(\"\\n\\n\")\n",
    "print(\"help(urlopen) returns:\\n\")\n",
    "print(help(urlopen)) # Notice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: you can also get help in Jupyter by typing `?urlopen` in a code block and then hitting 'run'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "?urlopen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A Challenge for you!\n",
    "\n",
    "Fix the code to set the `url` variable, then use it with the `urlopen` function to open it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "shown",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "\n",
    "# Given the info you were given above, what do you \n",
    "# think the value of 'url' should be? What\n",
    "# type of variable is it? int or string? \n",
    "url = ???\n",
    "\n",
    "# Read the URL stream into variable called 'response'\n",
    "# using the function that we imported above\n",
    "response = ???(url)\n",
    "\n",
    "#now read from the stream, decoding so that we get actual text\n",
    "datafile = response.read().decode('utf-8')\n",
    "\n",
    "print(\"datafile variable is of type: '\" + datafile.__class__.__name__ + \"'.\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "shown"
   },
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "\n",
    "url = \"http://bit.ly/2vrUFKi\"\n",
    "response = urlopen(url)\n",
    "datafile = response.read().decode('utf-8')\n",
    "\n",
    "print(\"datafile variable is of type: '\" + datafile.__class__.__name__ + \"'.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the _datafile_ variable is of type `string` (because we decoded it as such).  If we hadn't decoded it, the result would have been of type `bytes` which wouldn't be as easy for us (humans) to work with. \n",
    "\n",
    "Now that we've read our data as text, we can print it to check. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(datafile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this is definitely text, but even though it has been nicely formatted _visually_ in the notebook, right now it's actually not in a very convenient format to work with in our _code_. The `datafile` `str` object is actually a single string of text that the notebook is interpreting as having line breaks. We can see this by printing the 'raw' object (without jupyter notebook formatting for us) using the `repr` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(repr(datafile))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the `\\n` character that jupyter notebook is using to determine when to print a [new line](https://en.wikipedia.org/wiki/Newline) for us to read nicely.\n",
    "\n",
    "To split the text into individual lines ourselves (ready to work with in our code), we can use the handily named `.splitlines()` method (more on methods below):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://bit.ly/2vrUFKi\"\n",
    "\n",
    "response = urlopen(url)\n",
    "datafile = response.read().decode('utf-8').splitlines()\n",
    "\n",
    "print(\"datafile variable is of type: '\" + datafile.__class__.__name__ + \"'.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note now, that the data variable has type `list`. When we print this `datafile` `list` object (without `repr`) the `\\n` characters have gone and the list elements are split where those new line characters were previously (look carefully at where the `'` are): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(datafile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see this more clearly if we use a for loop to print out each element of the list (each element being a row of the original online file):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for row in datafile:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last row should be `10,Sheffield,10,-163545.3257,7055177.403,685368`.\n",
    "\n",
    "If you've managed to get the code above to run and have received 11 rows of text in response to your `urlopen` query then, congratulations, you've now read a text file sitting on a server in, I think, Alberta, Canada and Python _didn't care_. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Turning Text into Data\n",
    "\n",
    "We now need to work on turning the response we got to our `urlopen` request into useful data. You'll notice that we are dealing with a _CSV_ (Comma-Separated Value) file and that the format is quite simple since none of the rows have fields that *themselves* contain commas. So to turn this into data we just need to _split_ the row into separate fields using the commas.\n",
    "\n",
    "In the code below, `dir('string')` lists the available function for strings (because `'string'` is itself a _String_; we could just as easily written `dir('foo')` or `dir('supercalifragilisticexpialidocious')` because 'foo' and 'supercalifragilisticexpialidocious' are also strings and so have the _same_ functions available. \n",
    "\n",
    "In the output below, the functions that start and end with `__` are generally considered private, so you can skip over these and focus on the ones further down that are designed to be useful to programmers. Can you spot the method that is most likely to be useful?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Brief Musical Interlude\n",
    "\n",
    "Just in case you need help pronouncing supercalifragilisticexpialidocious:\n",
    "\n",
    "[![Mary Poppins](http://img.youtube.com/vi/uZNRzc3hWvE/0.jpg)](https://youtu.be/uZNRzc3hWvE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that you can find out what _methods_ are supported by a string using `dir(<string>)`:\n",
    "```python\n",
    "dir('supercalifragilisticexpialidocious')\n",
    "```\n",
    "I'm going to save you some time (_this_ time!) and tell you that we're interested in the `split` method. Why not use the `help` function to figure out how to make use of it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help('supercalifragilisticexpialidocious'.split)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, using the output of the `help` command, how would you use `split` to turn that word into a list like this: \n",
    "```python\n",
    "['sup','rcalifragilisticexpialidocious']\n",
    "```\n",
    "If you replace the `???` with the right bits of code then running the block below will print out \"You got it!\". You *only* need to change the `???` and nothing else!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "shown",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "if ['sup','rcalifragilisticexpialidocious']=='supercalifragilisticexpialidocious'.split(???, maxsplit=???):\n",
    "    print(\"You got it!\")\n",
    "else:\n",
    "    print(\"Not yet!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "solution2": "shown"
   },
   "outputs": [],
   "source": [
    "if ['sup','rcalifragilisticexpialidocious']=='supercalifragilisticexpialidocious'.split('e', maxsplit=1):\n",
    "    print(\"You got it!\")\n",
    "else:\n",
    "    print(\"Not yet!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some other string methods\n",
    "print('supercalifragilisticexpialidocious'.upper())\n",
    "print('supercalifragilisticexpialidocious'.title())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so you've tracked down the way to split a `string` using a delimiter and _even_ how to limit the number of 'words' that come out of the split operation. And you already saw another of these methods above (i.e. `splitlines`). We work a lot with `string`s, so it's handy to get to know the readily-available methods well.\n",
    "\n",
    "Let's test `string` splitting using our sample data (the last line of the 'simple' CSV file) to make sure it works the way we think it does... We want to turn the `string` below into a list like this:\n",
    "```python\n",
    "['10', 'Sheffield', '10', '-163545.3257', '7055177.403', '685368']\n",
    "```\n",
    "Again, we only need to change the `???`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = datafile[-1].split(???)\n",
    "print(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = datafile[-1].split(',')\n",
    "print(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Consider\n",
    "\n",
    "A question: why do you think that I consider a) to be data and not b)? \n",
    "\n",
    "a)\n",
    "```python\n",
    "['10', 'Sheffield', '10', '-163545.3257', '7055177.403', '685368']\n",
    "```\n",
    "\n",
    "b)\n",
    "```python\n",
    "\"10,Sheffield,10,-163545.3257,7055177.403,685368\"\n",
    "```\n",
    "\n",
    "**Decide what you think the answer is _before_ clicking for the solution**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "Hopefully you can see that a) is a `list` and b) is a `str` (string). Because a) is a list we can easily access each element. For example try the following code yourself: \n",
    "\n",
    "Here's a clue:\n",
    "```python\n",
    "print(\"The population of \" + myList[1] + \" is \" + myList[5])\n",
    "```\n",
    "\n",
    "It is much more difficult to access the individual pieces of information from the string...  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can hopefully see how we're breaking a complex problem down into a set of _increments_ , each of which is a bit easier to write and understand. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3 Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total = 0\n",
    "count = 0\n",
    "\n",
    "for idx,row in enumerate(datafile):\n",
    "    \n",
    "    if(idx > 0):\n",
    "        total = total + int(row[-1])\n",
    "        count += 1\n",
    "    #print(row)\n",
    "mean = total / count \n",
    "\n",
    "print(mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Credits!\n",
    "\n",
    "#### Contributors:\n",
    "The following individuals have contributed to these teaching materials: \n",
    "- [James Millington](https://github.com/jamesdamillington)\n",
    "- [Jon Reades](https://github.com/jreades)\n",
    "\n",
    "#### License\n",
    "The content and structure of this teaching project itself is licensed under the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/), and the contributing source code is licensed under [The MIT License](https://opensource.org/licenses/mit-license.php).\n",
    "\n",
    "#### Acknowledgements:\n",
    "Supported by the [Royal Geographical Society](https://www.rgs.org/HomePage.htm) (with the Institute of British Geographers) with a Ray Y Gildea Jr Award.\n",
    "\n",
    "#### Potential Dependencies:\n",
    "This notebook may depend on the following libraries: None"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}