Practical-03-Introducing Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center\">\n",
    "<h1>Introducing Pandas</h1>\n",
    "<h2>7SSG2059 Geocomputation 2018</h2>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Importance of Data Exploration\n",
    "\n",
    "After a couple of weeks getting to grips with Python itself, we're now going to start working with some real data. \n",
    "\n",
    "\n",
    "One of the first things that we do when working with any new data set is to familiarise ourselves with it. There are a _huge_ number of ways to do this, but there are no shortcuts to:\n",
    "\n",
    "<ul>\n",
    "  <li>a. Reading about the data (how it was collected, what the sample size was, etc.)</li>\n",
    "  <li>b. Reviewing any accompanying metadata (data about the data, column specs, etc.)</li>\n",
    "  <li>c. Looking at the data itself at the row- and column-levels</li>\n",
    "  <li>d. Producing descriptive statistics </li>\n",
    "  <li>e. Visualising the data using plots </li>\n",
    "</ul>\n",
    "\n",
    "In fact, you should use _all_ of these together to really understand where the data came from, how it was handled, and whether there are gaps or other problems. If you're wondering which comes first, the concept of  _start with a chart_ is always good.  But this week we want you to get a handle on pandas itself so although we will do some plotting of charts, we'll focus on a-d. There will be much more on plotting charts next week."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. What is Pandas?\n",
    "\n",
    "Pandas stands for 'Python Data Analysis Library'; it is designed to provide data scientists working in Python with a set of powerful tools to load, transform, and process large-ish data sets. As a result, it has become something of a *de facto* standard for online tutorials and many of the lessons that you can find online will make use of pandas at some point.\n",
    "\n",
    "Beyond what we provide below there are [numerous](http://lmgtfy.com/?q=introduction+to+pandas+python) useful introductions; [one of our favourites](http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/) is from Greg Reda, and there are some [good videos](https://youtu.be/TSsSWuhBpmY) on [our YouTube channel](https://www.youtube.com/playlist?list=PLJ5Y5hxm-0W7rOOYBHf6KC6QNnWOi09kh). And of course, there’s [TONS of stuff](http://stackoverflow.com/questions/tagged/pandas) on StackOverflow. If you want an actual physical book, you might try [McKinney (2017)](http://shop.oreilly.com/product/0636920050896.do).\n",
    "\n",
    "However, one thing you will really want to bookmark is [the official documentation](http://pandas.pydata.org/pandas-docs/stable/) since you will undoubtedly need to refer to it fairly regularly. _Note_: this link is to the most recent release. Over time there will be updates published and you _may_ find that you no longer have the most up-to-date version. If you find that you are now using an older version of pandas then you'll need to track down the _specific_ version of the documentation that you need from the [home page](http://pandas.pydata.org).\n",
    "\n",
    "You can always check what version you have installed like this:\n",
    "```python\n",
    "import pandas as pd\n",
    "print(pd.__version__)\n",
    "```\n",
    "*Note*: this approach isn't guaranteed to work with _every_ package, but it will work with most of them. Remember that variables and methods starting and ending with '`__`' are **private** and any interaction with them should be approached very, very carefully.\n",
    "\n",
    "First we need to import pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "print(pd.__version__) # Not necessary every notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we have done here is `import` pandas with an alias, _pd_. So now we can call pandas using `pd` instead of having to type out pandas in full each time (remember, programmers are lazy). You will need to run the line `import pandas as pd` _once_ in every notebook where you want to use pandas; you can use whatever alias you like but we'll assume you always use `pd` in GSA modules and that's what most examples online use as well..  \n",
    "\n",
    "If pandas imported properly, we can now look at the help file for a pandas `DataFrame` using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "help(pd.DataFrame)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! The help documentation for the DataFrame is not just a bit longer than anything we've seen before, it's massively longer! That's because pandas is much more sophisticated than anything we've looked at before. There's probably quite a lot of intimidating terminology in there too... Right from the start we get things like \"Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns).\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You've already invented pandas!\n",
    "\n",
    "Here's the thing: in the [last notebook](https://render.githubusercontent.com/view/ipynb?commit=bdf5a7b1388d8c22db7ca9ff34cb71f61131dc35&enc_url=68747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f6b696e677367656f636f6d702f67656f636f6d7075746174696f6e2f626466356137623133383864386332326462376361396666333463623731663631313331646333352f50726163746963616c2d30322d46756e6374696f6e73253230616e642532305061636b616765732e6970796e62&nwo=kingsgeocomp%2Fgeocomputation&path=Practical-02-Functions+and+Packages.ipynb&repository_id=67207954&repository_type=Repository#Why-'Obvious'-is-Not-Always-'Right') ([Download](https://raw.githubusercontent.com/kingsgeocomp/geocomputation/master/Practical-02-Functions%20and%20Packages.ipynb)) we came close to writing something like pandas from scratch. That's because pandas takes a column-view of data in the same way that our Dictionary-of-Lists did, it's just that it's got a lot more features than our 'simple' tool does. That's why the documentation is so much more forbidding and why pandas is so much more powerful.\n",
    "\n",
    "But at its heart, a pandas `DataFrame` (`df` for short) is a collection of data `Series` (i.e. columns) with an index. Each `Series` is like one of our column-lists from the last notebook. And the `df` is like the dictionary-of-lists that held the data together. You've seen this before, so you already _know_ what's going on... or at least you now have an _analogy_ that you can use to make sense of pandas:\n",
    "```python\n",
    "myDataFrame = {\n",
    "    '<column name 1>': <Series 1>,\n",
    "    '<column name 2>': <Series 2>,\n",
    "    '<column name 3>': <Series 3>\n",
    "}\n",
    "``` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with `Series` before loading the data as a `df`\n",
    "\n",
    "## 2. Pandas `Series`\n",
    "The official documentation for `Series` is [here](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#series). But as [Greg Reda](http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/) puts it so clearly; \n",
    "\n",
    "> _“a Series is a one-dimensional object similar to an array, list, or column in a table. It will assign a labelled index to each item in the Series. By default, each item will receive an index label from 0 to N, where N is the length of the Series minus one.”_ \n",
    "\n",
    "### Working with a Series\n",
    "\n",
    "This indexing is very similar to what we have used in the past for lists. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = pd.Series([7, 'Bristol', 3.14, -1789, 'Happy Birthday!'])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, to get to item 1 in the `Series`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(s[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks pretty similar to how we would index (access) a list, right? **Except...** notice that printing out the series showed a number (0..4) ndext to every element in the list; that should be a _clue_ that we're not using a normal list any more. That number is the *index*: this type of index is a lot more complicated than what we did when using indexes to access elements of a list, but they've used the same name because that describes its function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Pandas `Index`\n",
    "\n",
    "By default, pandas uses a 0-based integer sequence for its indexes (*i.e.* if you don't tell it do anything else, your index will be the numbers 0..n). But one of neat things about pandas `Series` is that you can set the index to anything you like. \n",
    "\n",
    "For example, let's imagine that you've calculated some summary statistics from a much larger data set and you want the name of the statistic to be the index value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mySummary = pd.Series([7, 1, 3.23, 0.88],  index=['max', 'min', 'mean', 'std'])\n",
    "print(mySummary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So *now* this looks a little like creating a dictionary with keys instead of a list with integer indexes since we can access values in the `Series` using the `index` we just specified, but it's still a _one-dimensional list_ (see definition above) since we only have one observation (_i.e._ data point) for each row.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(mySummary['min'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, we could do something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(mySummary.min())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the difference between the output for the last two `print()` statements? \n",
    "\n",
    "This is because we used two different ways to access the contensts of the `Series`:\n",
    "1. the first uses the index value we specified (_min_) to get the second value in the `Series`, \n",
    "2. the second uses the `.min()` [method](http://pandas.pydata.org/pandas-docs/stable/api.html#computations-descriptive-stats) to find the minimum value in the Series. \n",
    "\n",
    "That's actually pretty confusing, so we'd really recommend trying to _avoid_ using words like these as part of your index if you possibly can -- fortunately, that shouldn't happen too often: the only time you'll encounter it as *normally* part of *using* pandas is when you summarise a dataframe! However, the difference between an **index** value and a **data** value is crucial, so make sure you understand how these two lines of code differ and why their output is different -- ask if you’re not sure.\n",
    "\n",
    "#### Questions!\n",
    "\n",
    "**Also** notice one other thing: right at the bottom of `print(mySummary)` the `dtype` has changed from the first example: it's changed from `object` (the default) to `float64`. Try thinking this out for yourself:\n",
    "\n",
    "1. What does `dtype` mean?\n",
    "\n",
    "2. What is the difference between `object` and `float64` and why might it matter to pandas?\n",
    "\n",
    "Add your answer to this cell by just double-clicking on any of the next here and then writing your answer next to questions 1 and 2. You can _always_ add your own annotations, explanations, comments, complaints directly into a notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boolean Indexes\n",
    "\n",
    "We can also use a _boolean index_ (remember this from [Code Camp notebook 4](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/)?) to select or filter particular elements of the `Series` that meet certain conditions using logical operators like `==` or `>` (so this is like a conditional statement). For example, let's create a new `Series`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mySeries = pd.Series([100, 200, 300, 400], index = ['one', 'two', 'three', 'four']) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can pandas to find all elements of the `Series` with values greater than 250 and this will be returned as a Boolean index (**hint**: look at the `dtype`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mySeries > 250"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Think about it***: thats's already useful since we didn't have to write a `for` loop in order to find every element of `mySeries` that was greater than 250! But we can use that boolean index to retrieve *only* those values of the series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mySeries[mySeries > 250]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See if you can work out what  the last line of code is doing. It’s quite tricky:\n",
    "1.\tFirst it runs the conditional for *all* elements of `mySeries` (i.e. checking which values are greater than 250), \n",
    "2.\tThis produces a Boolean Series of `True`/`False` values the same length as the original series\n",
    "3.\tThen this returns all the elements in `mySeries` for which the partner values in the Boolean series are `True`. \n",
    "\n",
    "Think about it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s another example but, first, just by reading this code can you work out what it is doing?\n",
    "\n",
    "**Answer**: ???"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myEastings  = pd.Series([7063197, 6708480, 6703134, 7538620], index = ['Liverpool', 'Bristol', 'Reading', 'Glasgow'])\n",
    "\n",
    "myEastings[myEastings < 7000000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Pandas `DataFrames`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll often with `Series` as we go along, but usually we have more than just a few observations in one column of data (i.e. a one-dimensional data set. Let's step this up slowly by working with two-dimensional data in pandas; that means we now need to use the data structure called a `DataFrame`. We saw the help documentation for this above, but hopefully the rest of this notebook will be a little more comprehensible... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Official documentation for DataFrame is [here](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe). But again, [Greg Reda](http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/) puts it a little more intuitively:\n",
    "\n",
    "> _“A DataFrame is a tabular data structure comprised of rows and columns, akin to a spreadsheet, database table ... **You can also think of a `DataFrame` as a group of `Series` objects that share an index (the column names).**”_ \n",
    "\n",
    "Or, in terms of Dictionaries-of-Lists (what you did last week), a `DataFrame` could be thought of as a `dict` of `list` objects that share a `key`. \n",
    "\n",
    "Indeed, we could actually create our own `df` from a Dict-of-Lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = {'col_one' : [1., 2., 3., 4.],\n",
    "     'col_two' : [4., 3., 2., 1.]}\n",
    "\n",
    "df = pd.DataFrame(d)\n",
    "\n",
    "print(type(df))\n",
    "print(\"\")\n",
    "print(df)\n",
    "print(\"\")\n",
    "print(df.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bet the nerds found that one cool! (we did). \n",
    "\n",
    "Just to clarify what we did there:\n",
    "1. We created an object `d`, which is a dict-of-lists, using basic python \n",
    "2. We created an object `df`, which is a pandas `DataFrame`\n",
    "3. We printed the type of the `df` object (to check it really is a pandas `DataFrame`)\n",
    "4. We printed the pandas `DataFrame` itself\n",
    "5. We printed out a summary of the `DataFrame` that gave us some useful summary statistics about each (numerical) column\n",
    "\n",
    "Check you understand this and ask if you're not sure. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Reading and Writing Data \n",
    "\n",
    "It’s rare that we would actually want to write code to create a `DataFrame`. Usually, we **read** existing data into the `df`; this could be a data stored on a remote computer (*i.e.* on the Internet) or from a local file (on your hard drive). After manipulation and analysis we may also want to **save** a `df` for later use. To faciliate reading (and writing) files, pandas has a [variety of functions](http://pandas.pydata.org/pandas-docs/stable/io.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll look at how we read and write existing data using pandas functions by taking three steps:\n",
    "1. read the LSOA data file from a remote location\n",
    "2. write the LSOA data to a file on your local HDD\n",
    "3. (re)read the LSOA data from your local HDD.\n",
    "\n",
    "You do _not_ need to run this entire process every time you start this or any other notebook. We are just demonstrating the flexibility of pandas. In future you will likely want to just work with your local file, or if you need to download the data then you do this _once_ when you first start up the notebook and then never run that code again! \n",
    "\n",
    "We have stored the initial LSOA data online in a file named `LSOA Data.csv.gz`. This file is in [csv format](https://en.wikipedia.org/wiki/Comma-separated_values), but compressed (using the [gzip algorithm](https://en.wikipedia.org/wiki/Gzip)) to reduce data storage and transfer demands. \n",
    "\n",
    "The pandas `read_csv` function can read data saved in csv format _and_ which has been compressed directly into a `df`. Similarly, the `to_csv` function can write a pandas `df` in csv format _and_ compress the file using gzip.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read a remote data file in pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming we're connected to the Internet, we can read data from a remote server into the memory of our computer so that we can work with it. Just to drive home how pandas expands on what we did last week, let's read the large `CitiesWithWikipediaData.csv` file that we worked with last week into a pandas data frame called `my_df`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df = pd.read_csv(\n",
    "    'http://bit.ly/2iIK9bA',\n",
    "    low_memory=False) # The 'low memory' option means pandas doesn't guess data types\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All being well, the code above should not have produced an error when running and printed `Done!` when the pandas data frame was loaded. To see the result of running the code, let's check what type of object `my_df` is: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(my_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you see `<class 'pandas.core.frame.DataFrame'>` that's great! This is telling that you that you have created a pandas `df`. Let's have a quick look at the `DataFrame` we've created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice** that there is one column (the first one on the left next to `id`) has no label in bold type? *That* is the index. It is created automatically for us when we load the data frame *unless* we tell pandas to use one of the columns in the data.\n",
    "\n",
    "For now, think about how useful to pandas `read_csv` function is: instead of having to write some kind of _readRemoteCSV_ function ourselves, and then manually create a Dictionary-of-Lists from that remote file, we just told pandas to read it for us and it automagically converted it to a data structure that we we will be able use lots of functions to analyse (as we'll see below). You'll notice that it even figured out where the column names were. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a `DataFrame` to a local file \n",
    "\n",
    "Writing a file to disc (i.e. saving it for later use), is just as easy as reading the deata. The following code writes the `my_df` `df` to a csv file using the `to_csv` pandas method: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.to_csv('cities.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two things to note here: \n",
    "1. we have passed a value of `False` to the `index` argument - read what that does [in the documentation](http://pandas.pydata.org/pandas-docs/stable/io.html#io-store-in-csv)\n",
    "2. we have passed a value of `lsoa.csv` as a filename to use for the csv file created\n",
    "\n",
    "As the filename does not specify the full path, the data will be written to whatever your current working directory is. For a Jupyter notebook like this, that means the directory where the notebook itself is saved. Use Finder or File Explorer to go view the file you have just saved to HDD. \n",
    "\n",
    "Make sure you can find the file! It is important you understand where and how pandas and python write data. Ask if you need help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have noticed that the csv file is reasonablly large in terms of the space used on the HDD (~7 MB). If we want to save some space we could use the gzip compression option:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save this locally to avoid having to re-download it every time we start this notebook\n",
    "my_df.to_csv('cities.csv.gz', compression='gzip', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check you can find the file just created on your HD: if you're using JupyterLab you should have just seen the file appear in the navigation area on the left side of your screen. \n",
    "\n",
    "It is up to you to decide how you want to store your data, so think about these two different methods what they entail for reading data back into a `DataFrame` later (Google or ask for advice on [best practice](https://library.stanford.edu/research/data-management-services/data-best-practices) if you want).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we read the data back in, to ensure the reading works properly we'll delete the`my_df` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "del(my_df)\n",
    "\n",
    "print(type(my_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's one bit of code you actually _do_ want an error from! The error shows we have successfully removed the `df` object from memory (so python can't find it when we ask what type it is).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading data from a local file to a pandas `DataFrame`\n",
    "\n",
    "Above, we wrote the data to two types of file - one compressed and one uncompressed. Reading these files back into memory from a local HDD is just as easy as reading from a remote location. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, reading the compressed file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "gz_df = pd.read_csv('cities.csv.gz', compression='gzip') \n",
    "print(type(gz_df))\n",
    "del(gz_df) # And tidy up -- we don't need this again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly for the the uncompressed csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df = pd.read_csv('cities.csv')\n",
    "print(type(my_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all pretty straightforward _assuming python knows where your data are stored on the HDD_. Here, because we do not provide a full path the file is read from the current directory. In this case that should be oay (because we only just wrote the data to disk, without specifying a location), and we can see the read must have been at least partially successful as there was no error message and the object created is a pandas `df`. \n",
    "\n",
    "However, if your data are saved in a different location you will need to provide the full path to the location of a file on your HDD. For example, assuming you have a file named _lsoa2.csv_ in your downloads, you might do something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df = pd.read_csv('C:\\Users\\K1111111\\Downloads\\lsoa2.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unless my some miraculous coincidence your username is _k1111111_ and you really _do_ have a file named _lsoa2.csv_ in your Downloads folder, you should receive an error when you run the code above. Read the error and check you can see where python is trying to tell you it can't find the file you're trying to read. You may get such an error in future if you mis-specify a path or mis-spell a file name (so get used to this - don't be scared of error messages, they are _trying_ to help you!). \n",
    "\n",
    "Also remember that how you specify the path will differ between Windows (using \\\\) and Max (using \\/) - read more [here](https://www.howtogeek.com/181774/why-windows-uses-backslashes-and-everything-else-uses-forward-slashes/) if you're interested."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quickly viewing the contents of a `DataFrame`\n",
    "\n",
    "So we think we've read our data in correctly, but really need to check by looking at it. Later in the practical we'll see some of the useful plotting functionlity provided by pandas. \n",
    "\n",
    "But how can we quickly check the contents of what has been created without plotting? This is where the pandas `head()` method is useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All being well, you should have just printed the first five lines of the `DataFrame`. (If not, you might need to go back to code above to read the data into the `my_df` object - ask if you need help.)\n",
    "\n",
    "We can explore some of the [attributes](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) of the data frame. For example the `shape` attribute is a tuple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_df.shape) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This should show you that `my_df` has 72 rows and 20 columns. To print this more nicely we could do something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of columns is \" + str(my_df.shape[1]))\n",
    "print(\"Number of rows is \" + str(my_df.shape[???]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check you understand what's going on there; how did we access the column and row values individually?\n",
    "\n",
    "Note that we use `shape` not `shape()` - this is because `shape` is an attribute of the `DataFrame`, not a pandas method (function). You can compare many of the attributes and methods of pandas `DataFrames` [here](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html). For some [unknown](https://stackoverflow.com/a/19483025) reason, some attributes are not shown in the full documentation. For example, the `columns` _attribute_ is also useful: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that prints out all of the column names that were shown in **bold** up above. But notice that it's an _Index_! That's (to simplify in a useful way) because it isn't a `Series` containing data; so Indexes hold information _about_ rows and columns, while a Series holds _data_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It was want to get at the columns names as `list` of `values` (which will be more useful for writing code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_df.columns.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the subtle difference between the outputs of the last two lines of code (e.g. one starts, `Index`, the other does not; one has only [ ], the other ( ) and [ ]). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list of `column.values` points us to a way that we can read in only a subset of the columns from our original data file by specifying a subset to read in: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colnames = my_df.columns.values[1:4]\n",
    "print(colnames)\n",
    "print(\"\")\n",
    "\n",
    "sub_df = pd.read_csv('cities.csv', usecols=colnames)\n",
    "print(\"Shape of sub-setted data frame is: {0}\".format(list(sub_df.shape)))\n",
    "\n",
    "sub_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how we have only loaded the columns listed in the `colnames` list? What we have done here is pass a list of the column names that we want to load (called `colnames`) to the `read_csv` method using the `usecols` argument. Also note that there are **many** other arguments we could pass to `read_csv`, as shown [in the documentation](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html#pandas.read_csv). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also read in only a subset of the rows, which can be useful when working with Really Big datasets. For example, while the LSOA data at ~7MB has only around 40 columns and 4,800 rows, larger data sets like the London [InsideAirBnB data](http://insideairbnb.com/get-the-data.html) come in at nearly 50MB, and the OS' OpenRoads data set is more than 500MB! Reading and writing that amount of data will slow things down quite a bit! Sometimes it's easier to work with only a portion of the data while we are doing our coding and then, once we know that we've written things correctly, we do our analysis on the whole data set. \n",
    "\n",
    "One way to achieve this right at the start is to specify the number of rows (abbreviated to `nrows`) that you want to load in `pandas`. Using our cities data set, for instance, do we really need to start out with all 70+ rows? Or could we work with 'just' 50 to get things started? If we set up our analysis then the answer is: it doesn't matter! We should be able to just run the analysis on the subset of data and then re-run it on the _entire_ data set later (by changing the number of lines we read into memory right at the start of the analysis process or just removing `nrows...` from the `read_csv` method).\n",
    "\n",
    "So to read only 50 rows for specific columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_df = pd.read_csv('cities.csv', usecols=colnames, nrows=50)\n",
    "print(sub_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll see a lot more later in term how to more effectively subset and sample data, but this should speed things up quite a bit until we're ready to deal with the whole data set!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's right, pretty much exactly the same as for the uncompressed file. That's because pandas recognizes the compression tye and uncompresses for us!\n",
    "\n",
    "So a quick recap of what we've just done before moving on. We:\n",
    "1. read a data file from a remote location;\n",
    "2. wrote the data to a file on your local hard drive in two different formats (compressed and uncompressed);\n",
    "3. (re)read the data from your local hard drive (in two different formats);\n",
    "4. explored some different ways to quickly view the shape and content of data that we have read. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting the Index from a Series\n",
    "\n",
    "Let's have a nother look at `my_df` and see if we can use a more meaningful index than the numbers `0..n`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It probably makes the most sense to set the `Name` to be the index because that is naturally how we want to look up cities. Let's go:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.set_index('Name', inplace=True, drop=True)\n",
    "my_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right now, it's enough to know that you can (and usually *should*) set the index to one of the columns (data series) that you've loaded. The example below will make a *lot* more sense in a few weeks' time, but basically we are taking a sub-set of the observations in `my_df` using *both* row *and* column values: we select both using a list that refers to either the row or the column index!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And now...\n",
    "selection = ['Stoke-on-Trent','Coventry','Sunderland','Birkenhead','Reading','Kingston upon Hull','Preston']\n",
    "cols = ['Longitude','Latitude']\n",
    "my_df.loc[selection, cols]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Writing and Reading Data\n",
    "1. Create a directory called `data` in the _same_ directory that you are currently working in (_i.e._ where the practical is running) and write `my_df` as a compressed CSV to that directory.\n",
    "2. Delete the `DataFrame` object from memory.\n",
    "3. Read the data back in from the directory to an object named `new_df` and check it has loaded into memory correctly. \n",
    "4. Print the top few lines of `new_df` \n",
    "\n",
    "_**Note**: in Windows you may need to put an 'r' in front of the string that you use for your path (e.g. `r'path\\to\\mdata\\cities.csv'`)._\n",
    "\n",
    "_**Also Note**: on a Mac it's `path/to/data` but on Windows it's (often) `path\\to\\data`!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resetting the DataFrame\n",
    "\n",
    "Let's re-load the data now just to make sure we're working with the full data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df = pd.read_csv('cities.csv.gz', compression='gzip', low_memory=False) # Load the data frame\n",
    "idx   = list(my_df.columns.values).index('Unnamed: 11') # Find the first 'unnamed' column of garbage data\n",
    "my_df = my_df.iloc[:,:idx] # We'll see what iloc does later, but we are selecting all rows (':') and all columns between 0 and Unnamed: 11 (:idx)\n",
    "my_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Onwards, to data description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Describing Numerical Data in Pandas\n",
    "\n",
    "Let’s calculate some descriptive statistics for our data. Handily, there is a method we can apply to `DataFrame` objects that returns the standard descriptive statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a bit faster than trying to calculate all these summary stats in Excel, right? \n",
    "\n",
    "Just by calling `describe()`...\n",
    "1. We've asked Python to describe the `DataFrame` and it has returned a set of columns (actually, it's `DataFrame` itself as we'll see below) with descriptive metrics for each.\n",
    "2. Note what is _missing_ from this list: where is `Name`? Can you think why it wasn't reported in the descriptives?\n",
    "\n",
    "As you work through the description code below, you may want to refer back to the shape and contents of the `DataFrame` above..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we don’t just have to settle for the standard descriptive statistics the `describe()` method [gives us](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html). We can also specify particular percentiles of the data that we would like to view. This is done using the `percentiles` argument. For example to get the 1st and 99th percentiles we would use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.describe(percentiles = [0.20, 0.80])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, maybe you don't want the report for all columns. Maybe you're just interested in one column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.Population.describe(percentiles = [0.20, 0.80])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `dtype` at the end: that tells us the _data type_ is a 64-bit float. Remember, you can have strings, floats, integers, booleans, etc. in a `DataFrame`.\n",
    "\n",
    "But the really crucial thing is that this introduces _one_ of the two ways that we access a `Series` in pandas: `<data frame>.<series name>.method`. So we could get similar information on the `HHOLDS` column with:\n",
    "```python\n",
    "df.Population.describe()\n",
    "```\n",
    "And so forth.\n",
    "\n",
    "Also note that the output of the above returns a `Series` which we can save in memory and query later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdesc = my_df.Population.describe()\n",
    "type(pdesc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `info()` method is also useful to find the type of values in each field:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that describing a text column gives us an `object` data type because a `String` is a complex object, not a simple `float` or `int`. This now shows how `describe()` (above) automatically ignores non-numeric fields and only gives summaries for numeric fields (sensible!). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to `describe()` there are lots of other methods we can use to get descriptive statistics (and other derived variables) from our `DataFrame`. In general, we can ask the pandas to quickly work out a derived statistic in a `DataFrame` (such as the mean) by operating on a `Series`: `<data frame>.<series>.method()`. \n",
    "\n",
    "For example, for the mean of the `HHOLDS` `Series`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.Population.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a pretty easy way to calculate the mean of a `Series`, but look at how many decimal places the value has been reported to. More than 10 decimal places is too many - we just can't be that precise with these data. And if we created a table of values with this many decimal places we wouldn't be able to read anything very easily. So let's look briefly at how to present numeric values as `strings` with a specified level of precision:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The mean of Population is {0:.1f}\".format(my_df.Population.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice here that we have used a new `string` method named `format()`, to which we passed the number we want to output (in this case result of `my_df.Population.mean()`). The bit that might look particularly unfamiliar is the `{0:.2f}` at the end of the `string`. \n",
    "\n",
    "In order to understand how this works for formatting the results in a nice, systematic way you could have a read of [this](http://www.python.org/dev/peps/pep-3101/) 'pep' which tells us that:\n",
    "1. `{0}` would just grab the value being passed and stick it into the `string` at this point, but the `:` tells python that we want to give it some more information about how to format the string;\n",
    "2. the `f` tells python to treat anything it receives as a `float` (even if the variable is an `int`);\n",
    "3. the `.1` indicates we want only only one decimal place. \n",
    "\n",
    "So if we wanted four decimal places and up to 20 spaces for digits to the left of the decimal place:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The mean of Population is {0:20.4f}\".format(my_df.Population.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the `20` before the `.` indicates that we want the entire number to have 20 digits (but here 13 of those digits are actually white space). \n",
    "\n",
    "Read [the documentation](https://www.python.org/dev/peps/pep-3101/) and see if you can work out why even though we asked for 40(!) digits, the `<` means the value is still left aligned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The mean of Population is {0:<40.3f}\".format(my_df.Population.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might want to have a [look at the documentation](http://pandas.pydata.org/pandas-docs/stable/api.html#series): it's rather a long list, but most of your descriptive stats are on that page in the [Cumulative / Descriptive Stats](http://pandas.pydata.org/pandas-docs/stable/api.html#computations-descriptive-stats) section, and there's also lots of information about methods for [strings](http://pandas.pydata.org/pandas-docs/stable/api.html#string-handling) and [categorical data](http://pandas.pydata.org/pandas-docs/stable/api.html#categorical)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Selecting `DataFrame` Contents "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about if we wanted to select or access certain elements (i.e. numbers in) a `DataFrame`? That is, how do we access the contents of specific parts of the data frame, like accessing elements of a list?\n",
    "\n",
    "Rather than see how this works with the huge LSOA `DataFrame`, let's work with a smaller one. The describe method returns a `DataFrame` so we could create a smaller `DataFrame` by assigning the output of the `describe()` method to an object. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "descr = my_df.describe() # We are assigning the data frame of summary results to a new data frame called descr(iption)\n",
    "print(type(descr))\n",
    "print(descr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check you understand the structure of this `DataFrame`: the columns are `Series` and indexes are specified by what look like 'row names' above.\n",
    "\n",
    "We can select what elements of the `DataFrame` we want using the `Series` (column) names and/or the indexes. As proof, we can check these using the `columns` and `index` attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(descr.columns.values)\n",
    "print(descr.index.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Brackets vs Dot Notation\n",
    "\n",
    "To get at rows and columns of the data frame `descr` that we just created we can use their names directly. Python offers two ways to use the names, and these are usually referred to a Bracket and Dot notation:\n",
    "\n",
    "```python\n",
    "descr.Latitude     # Dot notation\n",
    "descr['Latitutde'] # Bracket notation\n",
    "```\n",
    "\n",
    "There is *no* difference between what these *do*, but there are some basic reasons why both are available to you:\n",
    "1. Dot notation is normally considered 'cleaner' and more elegant since it's easier to read.\n",
    "2. Bracket notation is needed if your column name has spaces; for example:\n",
    "\n",
    "```python\n",
    "desc.Rank of Size    # Syntax error!\n",
    "desc['Rank of Size'] # No problem!\n",
    "```\n",
    "\n",
    "Let's see this in action with the `Latitude` series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lat = descr.Latitude \n",
    "print(\"Descriptive stats of Latitude column: \\n\" + str(lat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above uses dot notation, which is possible because `Series` names are like attributes of the `DataFrame`. But we can also use square brackets `[]` which allows more direct access to contents of the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lat = descr['Latitude']\n",
    "print(\"Descriptive stats of Latitude column: \\n\" + str(lat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how the above produces exactly the same same output as dot notation?\n",
    "\n",
    "Let's experiment with the notation again to find the maximum Latitude value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd_max = descr['Latitude']['max']\n",
    "\n",
    "print(\"Maximum of Latitude column: {0:.2f}\".format(pd_max))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, think about how the next lines of code are different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd_max = descr.Latitude.max()\n",
    "\n",
    "print(\"Maximum of Latitude column: {0:.2f}\".format(pd_max))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key to the differences here is that in the second example (`descr['POPDEN'].max()`) pandas is trying to apply the `max()` method to the `Latitude` series, whereas in the former (`descr['Latitude']['max']`) it's clear that we want the `max` index of the `Latitude` series. Think about this and if you're unsure what's going on ask and discuss. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuing with bracket notation, which seems a little safer given the possible confusion between methods and indexes, we can also access two columns (`Series`) together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "descr_list = descr[['Latitude', 'Longitude']]\n",
    "print(\"Descriptive stats of Lat and Long columns: \\n\" + str(descr_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And because this is a `DataFrame` we can access the `max` `index` for both together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_list = descr[['Latitude', 'Longitude']].loc['max']\n",
    "print(\"Maximum of Lat and Long columns: \\n\" + str(max_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how in this final example we needed to use the `.loc` method; this highlights how there are [different choices for indexing](http://pandas-docs.github.io/pandas-docs-travis/indexing.html#different-choices-for-indexing) (and more [here](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#indexing-selection)) depending on what we’re working with. Check you understand how the examples above are working, ask if you're not sure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, because of the way `DataFrames` are structured `.loc[]` is useful for getting to the rows of the `DataFrames` (which are actually the indexes of the `Series`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_max = descr.loc['max'] \n",
    "print(\"Maximum values of all columns: \\n\" + str(all_max))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can access multiple indexes together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_mult = descr.loc[['max','min']] #get multiple together\n",
    "print(\"Max and Min values of all columns: \\n\" + str(all_mult))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Important Note\n",
    "\n",
    "The output of `describe` can be useful when you first load a data set, but it's not *normally* something we look at when performing data *analysis*... Not when we can access individual columns directly in order to retrieve important values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Maximum of Latitude column: {0:.2f}\".format(my_df.Latitude.max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selection using integer indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets quickly see how it’s also possible to use integer indexing in a very similar way to how we would access elements of a basic python list. We do this for `DataFrames` using via `.iloc[]` (where iloc is for integer location):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_max = descr.iloc[7,]  #row 7, all columns\n",
    "print(all_max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td_max = descr.iloc[7,2]   #row 7, column 2\n",
    "print(td_max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_mult = descr.iloc[[7,3],]  #rows 7 and 3 for all columns\n",
    "print(all_mult)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how the above are like `list` indexing in basic python. Think about this and ask if you're unsure what's going on.  \n",
    "\n",
    "This integer indexing approach also allows 'slicing', which we'll get to later in term (but some more advanced reading is [here](http://pandas-docs.github.io/pandas-docs-travis/indexing.html#selection-by-position) if you want). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selection using Criteria\n",
    "\n",
    "Accessing entire columns (`Series`) or rows (indexes) of data (or even individual cells of data) is one thing, but what if we don't know where the information we want is? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selection values greater/less than... \n",
    "\n",
    "For example, what if we wanted to find LSOAs with more than 1000 households? \n",
    "\n",
    "To do this we use a combination of the selection approaches above in combination with conditionals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smallCities = my_df[my_df.Population < 750000.0]\n",
    "smallCities.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's break this down:\n",
    "\n",
    "* `my_df.Population` is the Population column of our data frame `my_df`\n",
    "* `my_df.Population < 750000.0` is therefore a kind of _query_ (or _selection_) of rows where the population is less than 750,000. What it _actually_ does is compare each row's Population value to 750,000 and 'remember' if the result is `True` or `False` (a boolean index!).\n",
    "* `my_df[ ... ]` is _like_ what we do with a list when we write: `myList[3:5]` to select the fourth through sixth elements of a list, but in pandas we can _select_ non-sequential rows because we are using a `boolean` array (a.k.a. list) that looks like this: `[False, False, False, True, True, True, ...]`.\n",
    "* `smallCities = ...` saves the _result_ of the selection into a new data frame called `smallCities`.\n",
    "\n",
    "You can check what I'm saying about the boolean result using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.Population < 750000.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can check that `smallCities` and `my_df` are not the same using `shape`, which gives us the dimensions of the data frame as `(<rows>, <columns>)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_df.shape)\n",
    "print(smallCities.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What this first example means is that _anything_ that can be evaluated to `True` or `False` can be used to select rows from a data frame. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding maxima and minima\n",
    "\n",
    "Let's find the smallest and largest cities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df[my_df.Population == my_df.Population.min()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df[my_df.Population == my_df.Population.max()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subsetting Columns on a Selection\n",
    "\n",
    "And to get just the name of the city:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df[my_df.Population == my_df.Population.min()].Name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df[my_df.Population == my_df.Population.max()].Name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check you understand what is happening in this code. First, the row with the smallest or largest population is identified, then we tell pandas we want only the `Name` column for that row. \n",
    "\n",
    "We could also have done that using bracket notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df[my_df.Population == my_df.Population.min()][\"Name\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df[my_df.Population == my_df.Population.max()][\"Name\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whether you use dot notation of brackets is a matter of preference; dot notation is generally deemed more elegant but sometimes things are easier (or only possible) with brackets. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding a Range Between Known Values\n",
    "\n",
    "Perhaps we aren't just looking for extremes... how about all of the households with more than 750 households but less than 1000? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfRange = my_df.loc[ (my_df.Population > 750000.0) & (my_df.Population < 1500000.0) ]\n",
    "print(dfRange.shape)\n",
    "dfRange.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That example contains a few new things to which you need to pay attention:\n",
    "1. You'll see that, with mutiple selections, we had to put parentheses around each one -- this is to avoid confusing pandas as to what it should do _first_.\n",
    "2. We see an `&` (ampersand) which is completely new: it's a logical AND (like the `and` that we used when working with conditions) that asks pandas to \"Find all the rows where condition 1 _and_ condition 2 are both `True`\". So it calculates the `True`/`False` for the left side and the `True`/`False` for the right side of the `&`, and then combines them. Look at the appendix to this notebook for more examples and options.\n",
    "3. We had to a `.loc` on the end of the `my_df` -- the best way to think of this is that it 'freezes' things so as to prepare the data frame to do a search based on the _location_ of some complex selection criteria. We'll see more of this next week."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, you're not limited to using hard-coded numbers as filters on, for example, the `Population` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "thresh = 750000.0\n",
    "smallCities = my_df[my_df.Population < thresh]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case the threshold is still relative to the variable, although this code is now more transferable. But often we want to subset data based on the distribution of the data itself. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding a Range Based on the Distribution\n",
    "\n",
    "For example, to find the number of LSOAs whose count of households is _greater_ than the mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnPop = my_df.Population.mean()\n",
    "dfMean = my_df.loc[ my_df.Population > mnPop ]\n",
    "print(\"There are {0} cities above the mean population count of {1:.0f}\".format(dfMean.shape[0], mnPop))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice what we've done with the string formatting there? As the [documentation](https://www.python.org/dev/peps/pep-3101/) indicates, we can pass more than one variable to the `string.format()` method. Here, the first variable (`dfMean.shape[0]`) is placed in the first position (`{0}`) and the second variable (`mnHHolds`) goes in the second position (indicated by `{1...}`). \n",
    "\n",
    "Check you understand what is going on here:\n",
    "1. What is `dfMean.shape[0]`?\n",
    "2. What is `mnPop`?\n",
    "3. How is the `string.format()` method working?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Quick (and Dirty) Plotting\n",
    "\n",
    "One of the first things we should do when exploring a new dataset is plot (aka graph) the data. We've left plotting until a little later in this practical so that we could see some other basic attributes of how pandas stores data. We'll look at plotting and exploratory data analyses in much more detail next week, including using packages other than pandas. \n",
    "\n",
    "For now, let's look at the basic plotting functionality pandas provides - in conjunctions with the online documentation for both [DataFrames](http://pandas.pydata.org/pandas-docs/stable/api.html#plotting) and [Series](http://pandas.pydata.org/pandas-docs/stable/api.html#api-dataframe-plotting). There are also examples of all [the different types of plots pandas can produce](http://pandas.pydata.org/pandas-docs/stable/visualization.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important Note\n",
    "\n",
    "<div style=\"color:red; font-weight:bold\">This applies mainly to Mac users who are <i>not</i> using Docker.</div>\n",
    "\n",
    "Recent changes in the way that the Mac OS handles the plotting of data means that you need to do certain things in a specific order at the start of any notebook in which you intend to show maps or graphs. Please make a copy of the following code for any notebook that you create and make it the _first_ code that you run in the notebook:\n",
    "\n",
    "```python\n",
    "# Needed on a Mac\n",
    "import matplotlib as mpl\n",
    "mpl.use('TkAgg')\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "```\n",
    "\n",
    "For non-Mac users it _shouldn't_ hurt if you use the above, but you _should_ be able to get away with:\n",
    "\n",
    "```python\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "```\n",
    "\n",
    "This _should_ enable you to create plots, including in the remainder of this practical!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Needed on a Mac\n",
    "import matplotlib as mpl\n",
    "mpl.use('TkAgg')\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important for both Mac _and_ Windows\n",
    "For _both_ Mac **and** Windows, you will need to run the next line of code in a notebook before trying to plot anything. The code ensures the plots will appear properly in the notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This command tells Jupyter that we want \n",
    "# the plots to be shown inline (on this \n",
    "# web page). You'll always need to do this\n",
    "# *once* on a notebook.\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our first plots\n",
    "\n",
    "First, let's see some of the ways we could visualise the distribution of the `Series` in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.Density.plot.hist() # histogram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the code worked properly you should have just created a standard [histogram](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.plot.hist.html#pandas.Series.plot.hist) plot (if you can't see one, ask for help). \n",
    "\n",
    "Similarly, we can produce a [Kernel Density Estimate plot](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.plot.kde.html#pandas.Series.plot.kde) (modifying the limits of the x-axis to match the histogram above):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.Density.plot.kde(xlim = (3200,5500)) #kernel density estimate plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or a standard [boxplot](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.plot.box.html#pandas.Series.plot.box):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.Density.plot.box() # boxplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kind of handy, no? These aren't the _best_ looking plots, but they are all being generated on-the-fly for you by pandas with no more than a cheery `DataFrame.Series.plot.<plot type>`! Since those plots are all just method calls, many of them take optional parameters to change the colour, the notation (scientific or not), and other options. For example, many of the documentation pages linked to above are rather brief, but include a link to [the general options that can be applied to all `Series.plot`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.plot.html#pandas.Series.plot) calls.\n",
    "\n",
    "This is why we like pandas: it allows us to be _constructively lazy_. We don't need to know _how_ a draw a KDE plot (though it always helps if you don't see what you expected), we just need to know that pandas provides a method that will do it for you. And _that_ is why it's always worth having a [look at the documentation](http://pandas.pydata.org/pandas-docs/stable/api.html#plotting). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot two variables in a [scatter plot](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.scatter.html#pandas.DataFrame.plot.scatter) by applying a plot method to the `DataFrame` (not an individual `Series`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.plot.scatter(x='Longitude', y='Latitude')  #scatter plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the code above has the form `DataFrame.plot.<plot type>`, not `DataFrame.Series.plot.<plot type>` as in the prior plots. Think about why this then means we need the `x` and `y` arguments. \n",
    "\n",
    "Looking at the plot produced, we can see a couple of LSOAs stand out, having much greater numbers of residents relative to the number of households than for other LSOAs. Without producing a simple plot like this, we wouldn't have been able to see this from analysing the numbers.  \n",
    "\n",
    "We can also vary the size of the point in a plot by some variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We divide the population by 50,000 to get a useful point size for each cite\n",
    "my_df.plot.scatter(x='Longitude', y='Latitude', s=(my_df.Population/50000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can plot subsets of our data without creating a new object. See if you can work out what the following code is doing that is different from the last plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df[my_df.Population > my_df.Population.mean()].plot.scatter(x='Longitude', y='Latitude', s=(my_df.Population/50000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And pandas allows us to create 'less standard' plots, like a [hex bin plot](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.hexbin.html#pandas.DataFrame.plot.hexbin):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df.plot.hexbin(x='Longitude', y='Latitude', gridsize=10)  #hex bin plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's just a taste of what the basic plotting functionality of pandas can do. Feel free to explore more yourself but we'll go into more detail, and see [the seaborn package](http://seaborn.pydata.org/index.html), next week. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. The *Real* Data\n",
    "\n",
    "That's it for the Cities data. Now that we've progressed (rather rapidly) from using our own code to using pandas, we're ready for the big leagues! For much of the remainder of the module, and **for your final report**, we will be working with data for London that are aggregated spatially into geographical units known as Lower Super Output Area (**LSOAs**). Hence, we will refer to these data as the ***LSOA data***. \n",
    "\n",
    "The boundaries of LSOAs are adminstered by the [Office for National Statistics](https://www.ons.gov.uk/) (ONS) and used to analyse [UK Census](https://census.ukdataservice.ac.uk) and other data. You can read more about [the ONS coding system](https://en.wikipedia.org/wiki/ONS_coding_system), find out about the [UK census geography](https://www.ons.gov.uk/methodology/geography/ukgeographies/censusgeography), and explore [census data at LSOA level for London](http://londondatastore-upload.s3.amazonaws.com/instant-atlas/lsoa-atlas1/atlas.html).   \n",
    "\n",
    "The data set we have created and which you can use and build on is designed to provide a diverse range of data types so that there is 'something for everyone' -- physical and human geographer alike (and anything in between!) -- to work with. The sources and types of data in the initial LSOA dataset we will examine are: \n",
    "* [London Data Store](https://data.london.gov.uk/dataset/statistical-gis-boundary-files-london) provides the geography and some basic census data \n",
    "* [OS Greenspace](https://www.ordnancesurvey.co.uk/business-and-government/products/os-open-greenspace.html) provides data about the amount of greenspace\n",
    "* [OS VectorMap](https://www.ordnancesurvey.co.uk/business-and-government/products/vectormap-district.html) (District) provides data about distances to major roads\n",
    "* [NOMIS](https://www.nomisweb.co.uk/census/2011) provides more detailed census data\n",
    "* [InsideAirbnb](http://insideairbnb.com/get-the-data.html) scrapes information from the AirBnB website\n",
    "\n",
    "Later in term we will see how we can 'add' additional (air pollution) data to this initial dataset, but this is more than enough to be going on with for now! Don't spend too long looking at these links now, as we'll get on with exploring the data themselves. \n",
    "\n",
    "We have created a metadate file in Excel that provides information about the variables contained in the LSOA data. Look at that [metadata file](https://github.com/kingsgeocomp/geocomputation/raw/master/Data/LSOA_metadata.xlsx) now (file should auto-download; also available on KEATS). \n",
    "\n",
    "Come back to this section later to look at the links make sure you understand how what the data represent, how they were collected, etc. as this is important understanding that will support your data analyses.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_df = pd.read_csv(\n",
    "    'https://github.com/kingsgeocomp/geocomputation/blob/master/data/LSOA%20Data.csv.gz?raw=true',\n",
    "    compression='gzip', low_memory=False) # The 'low memory' option means pandas doesn't guess data types\n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the head to see what we have...\n",
    "my_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting Rows Using a Category\n",
    "\n",
    "One thing we couldn't show above with the Cities data was how to select rows using a value in *another* column! \n",
    "\n",
    "Let's select all of the LSOAs in the borough (which is a Local Authority District \\[LAD\\]) of Ealing. LAD names are indicated in the `LAD11NM` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ealingLSOAS = my_df[ my_df.LAD11NM == \"Ealing\" ]\n",
    "print(\"There are \" + str(ealingLSOAS.shape[0]) + \" LSOAs in Ealing\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could make our code a little more flexible for that last example, by allowing passing the search criteria (the borough name) as a variable itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = \"City of London\"\n",
    "bLSOAs = my_df[ my_df.LAD11NM == ??? ]\n",
    "print(\"There are \" + str(bLSOAs.shape[0]) + \" LSOAs in \" + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Think about how passing the borough name as a variable makes this code more flexible and requires fewer changes in the code to switch between boroughs.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises to Test Your Understanding \n",
    "\n",
    "#### Descriptive Stats\n",
    "\n",
    "Calculate the following descriptive statistics for the given variable for all London LSOAs:\n",
    "1. The maximum value of Median Income \n",
    "2. The number households in the LSOA with smallest area\n",
    "3. The total number of Asian/Asian British people in London LSOAs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#1\n",
    "#your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#2\n",
    "#your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#3\n",
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting Data\n",
    "Write code to select and print: \n",
    "1. LSOAs with HHOLDS more than 1 SD greater than the mean \n",
    "2. LSOAs in Hackney and Tower Hamlets\n",
    "3. The 'Social rented' column of LSOAs with population density of greater than 300 people per hectare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#1\n",
    "print(\"LSOAs with HHOLDS more than 1 SD greater than the mean: \\n\")\n",
    "#your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#2\n",
    "print(\"LSOAs in Hackney and Tower Hamlets: \\n\")\n",
    "#your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#3\n",
    "print(\"The 'Social rented' column of LSOAs with population density of greater than 300 people per hectare: \\n\")\n",
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting Data\n",
    "Create plots of the following:\n",
    "1. A histogram of the distribution of White people across LSOAs\n",
    "2. A boxplot of Asian people in LSOAs rotated so the box is horizontal (you will need to google how to rotate!)\n",
    "3. A scatter plot to compare populations of Asian (horizontal axis) vs White (vertical axis) people in LSOAs "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Credits!\n",
    "\n",
    "#### Contributors:\n",
    "The following individuals have contributed to these teaching materials: James Millington (james.millington@kcl.ac.uk), Jon Reades (jonathan.reades@kcl.ac.uk)\n",
    "\n",
    "#### License\n",
    "These teaching materials are licensed under a mix of [The MIT License](https://opensource.org/licenses/mit-license.php) and the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/).\n",
    "\n",
    "#### Acknowledgements:\n",
    "Supported by the [Royal Geographical Society](https://www.rgs.org/HomePage.htm) (with the Institute of British Geographers) with a Ray Y Gildea Jr Award.\n",
    "\n",
    "#### Potential Dependencies:\n",
    "This notebook may depend on the following libraries: pandas, matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "GSA2019",
   "language": "python",
   "name": "gsa2019"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}