Chapter1_Python_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Python\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Python is a popular general purpose, free language. As opposed to other systems that are focused towards a particular task (e.g. R for statistics), Python has a strong following on the web, on systems operations and in data analysis. For scientific computing, a large number of useful add-ons (\"libraries\") are available to help you analyse and process data. This is an invaluable resource.\n",
    "\n",
    "In addition to being free, Python is also very portable, and easy to pick up. \n",
    "\n",
    "\n",
    "The aim of this Chapter is to introduce you to some of the fundamental concepts in Python. Mainly, this is based around fundamental data types in Python (`int`, `float`, `str`, `bool` etc.) and ways to group them (`tuple`, `list`, array, string and `dict`).\n",
    "\n",
    "Although some of the examples we use are very simple to explain a concept, the more developed ones should be directly applicable to the sort of programming you are likely to need to do.\n",
    "\n",
    "Further, a more advanced section of the chapter is available, that goes into some more detail and complkications. This too has a set of exercises with worked examples.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this session, you will be introduced to some of the basic concepts in Python.\n",
    "\n",
    "**The session should last 4 hours (one week).**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input cells\n",
    "\n",
    "In a notebook, a code cell (i.e. one we can write computer code in) is labelled \n",
    "\n",
    "    In [N]:\n",
    "    \n",
    "where `N` is a number indicating the index of when the cell was run. So, when we run our first input cell, with will be labelled \n",
    "\n",
    "    In [1]:\n",
    "    \n",
    "then\n",
    "\n",
    "    In [2]:\n",
    " \n",
    "etc.\n",
    "\n",
    "You can 'run' the code in a code cell by placing the cursor in the ciode block to be run, then use the `Run` widget (above), use the `Cell->Run Cells` menu selection, or hit the keys ('typing') `<SHIFT>` and `<RETURN>` at the same time. \n",
    "\n",
    "<span class=\"girk\">Do this in the cells below:</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "# hello world"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hello world again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class=\"girk\">Now, run those cells (above) again and note how the index increases.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can re-start running cells ('restart the kernel') in a notebook (and clear the output if you like) using options from the `Kernel` menu.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "<span class=\"girk\">**Exercise**\n",
    "\n",
    "* Restart the kernel now and re-run the cells above\n",
    "* What happens to the indices now?</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "<span class=\"burk\">**Answer**\n",
    "\n",
    "When you restart the kernel with `Kernel->Restart & Clear Output`, the indices in the cells above should become blank:\n",
    "\n",
    "    In [ ]:\n",
    "    \n",
    "When you re-run these, they should be numbered:\n",
    "\n",
    "    In [1]:\n",
    "    In [2]:\n",
    "\n",
    "If you restart the kernel without clearing, the index is not blanked first.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------\n",
    "\n",
    "\n",
    "### # Comments\n",
    "\n",
    "Comments are statements ignored by the language interpreter.\n",
    "\n",
    "Any text after a `#` in a *code cell* (input cell) is a comment.\n",
    "\n",
    "\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class=\"girk\">**Exercise**\n",
    "\n",
    "* Run the code block below\n",
    "* Explain what happened ('what the computer did')</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "code_folding": [],
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "# Hello world"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden"
   },
   "source": [
    "<span class=\"burk\">**Answer**\n",
    "\n",
    "Run the code block with the cursor in the relevant cell:\n",
    "* with `<SHIFT-RETURN>`, i.e. `<SHIFT>` and `<RETURN>` keys at the same time.\n",
    "* or `Cell->Run Cells` from the notebook menu\n",
    "* or click on the `Run` button\n",
    "\n",
    "Explain what happened:\n",
    "In the case, the code in the cell was:\n",
    "\n",
    "    # Hello world\n",
    "    \n",
    "When we run this, 'nothing' seems to happen, because anything after `#` in a code line is a comment (i.e. not interpreted).</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------\n",
    "\n",
    "### String and text blocks as comments\n",
    "\n",
    "A `string` is a type of variable used to contain some text. A single line of text is defined within quotes (either single quotes `'` or double quotes `\"`), and a multi-line string between `'''` triple quotes.\n",
    "\n",
    "We use strings for a range of purposes, but you will often see them used as 'document strings', which yiou can think of as a form of comment.\n",
    "\n",
    "In the code block below, we demonstrate the `#` comment again, and also show two types of string text. \n",
    "\n",
    "Because the final string:\n",
    "\n",
    "        '''\n",
    "        Hello \n",
    "        world\n",
    "        is printed\n",
    "        '''\n",
    "        \n",
    "is at the end of the cell, this string is 'printed' to the output cell `Out [3]:` (assuming the index is at `3` here).\n",
    "\n",
    "<span class=\"girk\">Run this cell to confirm this happens.</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nHello \\nworld\\nis printed\\n'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# single line text\n",
    "'hello world is not printed'\n",
    "\n",
    "# multi-line text\n",
    "'''\n",
    "Hello \n",
    "world\n",
    "is printed\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " \n",
    "<span class=\"girk\">**Exercise**\n",
    " \n",
    "* Copy the text from above in the window below\n",
    "\n",
    "* Then put a new text block at the end and note down what happens\n",
    "        \n",
    "* What does the `\\n` mean/do?</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nThis is new text\\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Answer:\n",
    "\n",
    "# ---------------------\n",
    "# copy the previous text \n",
    "# single line text\n",
    "'hello world is not printed'\n",
    "\n",
    "# multi-line text\n",
    "'''\n",
    "Hello \n",
    "world\n",
    "is printed\n",
    "'''\n",
    "\n",
    "# ---------------------\n",
    "# a new text block\n",
    "# A newline character \\n is placed in the \n",
    "# string for each new line in multi-line text\n",
    "# The final entry in the input cell is output to \n",
    "# the correspinding output cell\n",
    "'''\n",
    "This is new text\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------\n",
    "\n",
    "### Print function\n",
    "\n",
    "To print some statement to the screen you are using, use the `print()` function.\n",
    "\n",
    "<span class=\"girk\">Run the two cells below to execute some print statements.</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world\n"
     ]
    }
   ],
   "source": [
    "# print a single string\n",
    "print('hello world')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world\n"
     ]
    }
   ],
   "source": [
    "# print a list of strings\n",
    "print('hello','world')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python 3.X, `print` is a function with the *argument(s)* (here, the string you want printed) enclosed in the function's (round) brackets.\n",
    "\n",
    "<span class=\"girk\">**Exercise**\n",
    "\n",
    "* Copy the print statement from the above code block.\n",
    "\n",
    "* Change the words in the quotes and print them out.\n",
    "\n",
    "* Add some comments to the code block explaining what you have done and seen.</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "good morning\n"
     ]
    }
   ],
   "source": [
    "# Answer\n",
    "\n",
    "'''\n",
    "The print statement below sends text to the output channel (the 'screen' as it were). \n",
    "In this case, it prints the string 'good', followed by a space, then the string 'morning'\n",
    "'''\n",
    "print('good','morning')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables and values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of **variables** is fundamental to any programming. You can think of this as the *name* of *something*, so it is a way of allowing us to refer to some object in the language.\n",
    "\n",
    "What the variable *is* set to is called its **value**.\n",
    "\n",
    "So let's start with a variable we will call (*declare to be*) `x`.\n",
    "\n",
    "We will give a *value* of the string `'one'` to this variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "one\n"
     ]
    }
   ],
   "source": [
    "x = 'one'\n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.4 Exercise**\n",
    "\n",
    "* set the a variable called x to some different string (e.g. 'hello world')\n",
    "    \n",
    "* print the value of the variable `x`\n",
    "    \n",
    "* Try this again, putting some 'newlines' (`\\n`) in the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we set x to the value 1\n",
    "\n",
    "x = 1\n",
    "\n",
    "print(x,'is type',type(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a computing language, the *sort of thing* the variable can be set to is called its **data type**. \n",
    "\n",
    "In python, we can access this with the method `type()` as in the example above.\n",
    "\n",
    "In the example above, the datatype is an **integer** number (e.g. `1, 2, 3, 4`).\n",
    "\n",
    "In 'natural language', we might read the example above as 'x is one'.\n",
    "\n",
    "\n",
    "**E1.1.5 Exercise**\n",
    "\n",
    "* set the a variable called `x` to the integer `5`\n",
    "    \n",
    "* print the value and type of the variable `x`\n",
    "    \n",
    "* change the data type used for `x` to something else (e.g. a string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Setting `x = 1` is different to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 'one'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "because here we have set value of the variable `x` to a **string** (i.e. some text).\n",
    "\n",
    "A string is enclosed in quotes, e.g. `\"one\"` or `'one'`, or even `\"'one'\"` or `'\"one\"'`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"one\")\n",
    "print ('one')\n",
    "print (\"'one'\")\n",
    "print ('\"one\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.6 Exercise**\n",
    "\n",
    "* create a variable `name` containing your name, as a string.\n",
    "        \n",
    "* using this variable and the print function, print out a statement such as `my name is Fred` (if your name were `Fred`)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting `x = 1` or `x = 'one'` is different to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "because here we have set value of the variable `x` to a **floating point** number (these are treated and stored differently to integers in computing).\n",
    "\n",
    "This in turn is different to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where `True` is a **logical** or **boolean** datatype (something is `True` or `False`).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.7 Exercise**\n",
    "\n",
    "* in the code block below, create a variable called `my_var` and set it to some value (your choice of value, but be clear about the data type you intend)\n",
    "           \n",
    "* print the value of the variable to the screen, along with the data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have so far seen four datatypes:\n",
    "\n",
    "- integer (`int`): 32 bits long on most machines\n",
    "- (double-precision) floating point (`float`): (64 bits long)\n",
    "- Boolean (`bool`)\n",
    "- string (`str`)\n",
    "\n",
    "but we will come across more (and even create our own!) as we go through the course.\n",
    "\n",
    "In each of these cases above, we have used the variable `x` to contain these different data types.\n",
    "\n",
    "As we saw above, if you want to know what the data type of a variable is, you can use the method `type()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (type(1));\n",
    "print (type(1.0));\n",
    "print (type('one'));\n",
    "print (type(True));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explicitly convert between data types, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print ('int(1.1) = ',int(1.1))\n",
    "print ('float(1) = ',float(1))\n",
    "print ('str(1) = ',str(1))\n",
    "print ('bool(1) = ',bool(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but only when it makes sense:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"converting the string '1' to an integer makes sense:\",int('1'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"converting the string 'one' to an integer doesn't:\",int('one'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you get an error (such as above), you will need to learn to *read* the error message to work out what you did wrong.\n",
    "\n",
    "**E1.1.8 Exercise**\n",
    "\n",
    "* why did the statement abovce not work?\n",
    "    \n",
    "* type some other data conversions below that *do* work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.3 Arithmetic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often we will want to do some [arithmetic](http://www.tutorialspoint.com/python/python_basic_operators.htm) with numbers in a program, and we use the 'normal' (derived from C) operators for this.\n",
    "\n",
    "Note the way this works for integers and floating point representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "    Some examples of arithmetic operations in Python\n",
    "\n",
    "    Note how, if we mix float and int, the result is raised to float\n",
    "    (as the more general form)\n",
    "'''\n",
    "\n",
    "print (10 + 100)     # int addition\n",
    "print (10. - 100)    # float subtraction\n",
    "print (1./2.)        # float division\n",
    "print (1/2)          # int division\n",
    "print (10.*20.)      # float multiplication\n",
    "print (2 ** 3.)      # float exponent\n",
    "\n",
    "print (65%2)         # int remainder\n",
    "print (65//2)        # floor operation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.9 Exercise**\n",
    "\n",
    "* change the numbers in the examples above to make sure you understand these basic operations.\n",
    "\n",
    "* try combining operations and use brackets `()` to check that that works as expected.\n",
    "\n",
    "* see what happens when you add (i.e. use `+`) strings together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.4 Assignment Operators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "    Assignment operators\n",
    "\n",
    "    x = 3   assigns the value 3 to the variable x\n",
    "    x += 2  adds 2 onto the value of x\n",
    "            so is the same as x = x + 2\n",
    "            similarly /=, *=, -=\n",
    "    x %= 2  is the same as x = x % 2\n",
    "    x **= 2 is the same as x = x ** 2\n",
    "    x //= 2 is the same as x = x // 2\n",
    "\n",
    "    A 'magic' trick\n",
    "    ===============\n",
    "\n",
    "    based on\n",
    "    https://www.wikihow.com/Read-Someone%27s-Mind-With-Math-(Math-Trick)\n",
    "\n",
    "    whatever you put as myNumber, the answer is 42\n",
    "\n",
    "    Try this with integers or floating point numbers ...\n",
    "'''\n",
    "\n",
    "# pick a number \n",
    "myNumber = 34.67\n",
    "\n",
    "x = myNumber\n",
    "\n",
    "x *= 2\n",
    "\n",
    "x *= 5\n",
    "\n",
    "x /= myNumber\n",
    "\n",
    "x -= 7\n",
    "\n",
    "x += 39\n",
    "\n",
    "# The answer will always be 42\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.10 Exercise**\n",
    "\n",
    "* change the number assigned to `myNumber` and check if `42` is still returned\n",
    "* copy and edit the code to print the value of `x` each time you change it, and add comments explaining what is happening for each line of code. This should allow you to follow more carefully what has happened with the arithmetic and also to simplify the code (use fewer statements to achieve the same thing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.5 Logical Operators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logical operators combine boolean variables. Recall from above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (type(True),type(False));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three main logical operators you will use are:\n",
    "\n",
    "    not, and, or\n",
    "    \n",
    "The impact of the `not` opeartor should be straightforward to understand, though we can first write it in a 'truth table':   \n",
    "\n",
    "\n",
    "\n",
    "| A  | not A  | \n",
    "|:---:|:---:|\n",
    "|  T | F | \n",
    "|   F |  T | "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('not True is',not True)\n",
    "print('not False is',not False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.11 Exercise:**\n",
    "    \n",
    "* write a statement to set a variable `x` to `True` and print the value of `x` and `not x`\n",
    "       \n",
    "* what does `not not x` give? Make sure you understand why \n",
    "    \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The operators `and` and `or` should also be quite straightforward to understand: they have the same meaning as in normal english. Note that `or` is 'inclusive' (so, read `A or B` as 'either A or B or both of them')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print ('True and True is',True and True)\n",
    "print ('True and False is',True and False)\n",
    "print ('False and True is',False and True)\n",
    "print ('False and False is',False and False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, `A and B` is `True`, if and only if both `A` is `True` and `B` is `True`. Otherwise, it is `False`\n",
    "\n",
    "We can represent this in a 'truth table':\n",
    "\n",
    "\n",
    "| A  | B  | A and B  | \n",
    "|:---:|:---:|:---:|\n",
    "|  T |  T |  T | \n",
    "|  T |  F |  F | \n",
    "|  F |  T |  F | \n",
    "|  F |  F |  F | \n",
    "\n",
    "\n",
    "**E1.1.12 Exercise:**\n",
    "\n",
    "* draw a truth table *on some paper*, label the columns `A`, `B` and `A and B` and fill in the columns `A` and `B` as above\n",
    "* without looking at the example above, write the value of `A and B` in the third column.\n",
    "* draw another truth table *on some paper*, label the columns `A`, `B` and `A and B` and fill in the columns `A` and `B` as above\n",
    "* write the value of `A or B` in the third column.\n",
    "\n",
    "If you are unsure, test the response using code, below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the testing here e.g.\n",
    "print (True or False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.13 Exercise**\n",
    "\n",
    "* Copy the following truth table onto paper and fill in the final column:\n",
    "\n",
    "| A  | B  | C | ((A and B) or C)  | \n",
    "|:---:|:---:|:---:|:---:|\n",
    "| T|  T |  T |   | \n",
    "| T|  T |  F |   | \n",
    "| T|  F |  T |   | \n",
    "| T |  F |  F |   | \n",
    "| F|  T |  T |   | \n",
    "| F|  T |  F |   | \n",
    "| F|  F |  T |   | \n",
    "| F |  F |  F |   | \n",
    "\n",
    "* Try some other compound statements\n",
    "\n",
    "If you are unsure, or to check your answers, test the response using code, below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the testing here e.g.\n",
    "print ((True and False) or True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.6 Comparison Operators and `if`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A comparison operator 'compares' two terms (e.g. variables) and returns a boolean data type (`True` or `False`).\n",
    "\n",
    "For example, to see if the value of some variable `a` is 'the same value as' ('equivalent to') the value of some variable `b`, we use the equivalence operator (`==`). To test for non equivalence, we use the not equivalent operator `!=` (read the `!` as 'not'):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 100\n",
    "b = 10\n",
    "\n",
    "# Note the use of \\n and \\t in here\n",
    "# \n",
    "print ('a is',a,'and\\nb is',b,'\\n')\n",
    "print ('\\ta is equivalent to b?',a == b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.14 Exercise**\n",
    "\n",
    "* copy the code above and change the values (or type) of the variables `a` and `b` to test their equivalence.\n",
    "* what does the `\\t` in the print statement do?\n",
    "* add a `print` statement to your code that tests for 'non equivalence'\n",
    "* write some code to see if `(a or b)` is equivalent to `(b or a)` or not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A full set of comparison operators is:\n",
    "\n",
    "|symbol| meaning|\n",
    "|:---:|:---:|\n",
    "| is | [is identical to](https://www.geeksforgeeks.org/difference-operator-python/) |\n",
    "| is not | is not identical to |\n",
    "| == | is equivalent to |\n",
    "| != | is not equivalent to |\n",
    "| > | greater than |\n",
    "|>= | greater than or equal to|\n",
    "|<  | less than|\n",
    "|<=  | less than or equal to    |\n",
    "\n",
    "so that, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Comparison examples\n",
    "\n",
    "# is one plus one list identical to two list?\n",
    "print ([1 + 1] is [2])\n",
    "\n",
    "# is one plus one list equal to two list?\n",
    "print ([1 + 1] == [2])\n",
    "\n",
    "# is one less than or equal to 0.999?\n",
    "print (1 <= 0.999)\n",
    "\n",
    "# is one plus one not equal to two?\n",
    "print (1 + 1 != 2)\n",
    "\n",
    "# note the use of single quotes inside a double quoted string here\n",
    "# is 'more' greater than 'less'?\n",
    "print (\"more\" > \"less\")\n",
    "\n",
    "# \"is 100 less than 2?\"\n",
    "print (100 < 2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Aside on string comparisons**\n",
    "\n",
    "In the case of string comparisons, the [ASCII](http://www.asciitable.com) codes of the string characters are compared. So for example the statement \"more\" > \"less\" returns True.\n",
    "\n",
    "Here, the comparison is effectively\n",
    "\n",
    "    m > l\n",
    "\n",
    "Since `m` comes after `l` in the alphabet, the ASCII\n",
    "code for `m` (109) is greater than the ASCII code \n",
    "for `l` (108) (see http://www.asciitable.com) so\n",
    "\n",
    "    109 > 108\n",
    "\n",
    "returns True. Note that ASCII capital letters come before the lower case letters. \n",
    "\n",
    "In practice, we mainly avoid string comparisons (other than to confirm equivalence). So there is little direct use of string comparisons other than `==`. It is useful to know how this works however, in case it crops up or happens 'by accident'. It is also worth understanding what ASCII codes are.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conditional test**\n",
    "\n",
    "One common use of comparisons is for program control, using an `if` statement:\n",
    "\n",
    "    if condition1 is True:\n",
    "        doit1()\n",
    "    elif condition2 is True:\n",
    "        doit2()\n",
    "    else:\n",
    "        doit3()\n",
    "        \n",
    "where `is` compares identity. This allows us to run blocks of code (e.g. the method `doit1()`) only under a particular condition (or set of conditions).\n",
    "\n",
    "In Python, the statement(s) we run on condition (here `doit1()` etc.) are *indented*. \n",
    "\n",
    "The indent can be one or more spaces or a `<tab>` character, the choice is up to the programmer. However, it **must be consistent**.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = [1+1]\n",
    "print('test is {}'.format(test))\n",
    "\n",
    "# initialise retval\n",
    "retval = None\n",
    "\n",
    "# conduct some tests, and set the \n",
    "# variable retval to True if we pass\n",
    "# any test\n",
    "\n",
    "if test is [2]:\n",
    "    retval = True\n",
    "    print('passed test 1: \"if test is [2]\"')\n",
    "elif test == [2]:\n",
    "    retval = True\n",
    "    print('passed test 2: \"if test == [2]\"')\n",
    "else:\n",
    "    retval = False\n",
    "    print('failed both tests')\n",
    "    \n",
    "print('retval is',retval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.1.15 Exercise**\n",
    "\n",
    "* copy the example above, and change it to use other examples from the 'Comparison examples' code block. Change the value of `test` to get different responses and make notes as to why you get the result you do.\n",
    "* try out some more complicated conditions, e.g. multipler tests, combined with an `and` operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.7 Summary\n",
    "\n",
    "In this section, you have had an introduction to the Python programming language, running in a [`jupyter notebook`](http://jupyter.org) environment.\n",
    "\n",
    "You have seen how to write comments in code, how to form `print` statements and basic concepts of variables, values, and data types. You have seen how to maniputae data with arithmetic and assignment operators, as well as the basics in dealing with logic and tests returning logical values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## 1.2  Text and looping\n",
    "In Python, collections of characters (`a`, `b`, `1`, ...) are called strings. Strings and characters are input by surrounding the relevant text in either double (`\"`) or single (`'`) quotes. There are a number of special characters that can be encoded in a string provided they're \"escaped\". For example, some we have come across are:\n",
    "\n",
    "* `\\n`: the carriage return\n",
    "* `\\t`: a tabulator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"I'm a happy string\")\n",
    "print ('I\\'m a happy string') # the apostrophe has been escaped as not to be confused by end of string\n",
    "print (\"\\tI'm a happy string\")\n",
    "print (\"I'm\\na\\nhappy\\nstring\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do a number of things with strings, which are very useful. These so-called string methods are defined on all strings by Python by default, and can be used with every string.\n",
    "For one, we can concatenate strings using the `+` symbol as we saw above.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.1  `len`\n",
    "\n",
    "Gives the length of the string as number of characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = ''\n",
    "print ('the length of',t,'is',len(t))\n",
    "\n",
    "\n",
    "s = \"Hello\" + \"there\" + \"everyone\"\n",
    "\n",
    "print ('the length of',s,'is',len(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise E1.2.1**\n",
    "\n",
    "* what does a zero-length string look like?\n",
    "* The `Hello there everyone` example above has no spaces between the words. Copy the code to the block below and modify it to have spaces.\n",
    "* confirm that you get the expected increase in length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.2 `for ... in ...` and `enumerate`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very commonly, we need to iterate or 'loop' over some set of items.\n",
    "\n",
    "The basic stucture for doing this (in Python, and many other languages) is `for item in group:`, where `item` is the name of some variable and `group` is a set of values. \n",
    "\n",
    "The loop is run so that `item` takes on the first value in `group`, then the second, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for loop\n",
    "group = [4,3,2,1]\n",
    "\n",
    "for item in group:\n",
    "    '''print counter in loop'''\n",
    "    print(item)\n",
    "    \n",
    "print ('blast off!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `group` in this example is the list of integer numbers `[4,3,2,1]`. A `list` is a group of comma-separated items contained in square brackets `[]`.\n",
    "\n",
    "In Python, the statement(s) we run whilst looping (here `print(item)`) are *indented*. \n",
    "\n",
    "The indent can be one or more spaces or a `<tab>` character, the choice is up to the programmer. However, it **must be consistent**.\n",
    "\n",
    "**Exercise 1.2.1**\n",
    "\n",
    "* generate a list of strings called `group` with the names of (some of) the items in your pocket or bag (or make some up!)\n",
    "* set up a `for` loop to go through and print each item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quite often, we want to keep track of the 'index' of the item in the loop (the 'item number').\n",
    "\n",
    "One way to do this would be to use a variable (called `count` here). \n",
    "\n",
    "Before we enter the loop, we initialise the value to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for loop\n",
    "group = ['hat','dog','keys']\n",
    "\n",
    "# initialise a variable count\n",
    "count = 0\n",
    "\n",
    "for item in group:\n",
    "    '''print counter in loop'''\n",
    "    \n",
    "    print('item',count,'is',item)\n",
    "    \n",
    "    # add 1 onto count\n",
    "    count += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1.2.2**\n",
    "\n",
    "* copy the code above, and check to see if the value of `count` at the end of the loop is the same as the length of the list. Why should this be so?\n",
    "* change the code so that the counting starts at 1, rather than 0. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since counting in loops is a common task, we can use the built in method `enumerate()` to achieve the same thing as above. The syntax is then:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for loop\n",
    "group = ['hat','dog','keys']\n",
    "\n",
    "for count,item in enumerate(group):\n",
    "    '''print counter in loop'''\n",
    "    print('item',count,'is',item)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1.2.3**\n",
    "\n",
    "* copy the code above, and check to see if the value of `count` at the end of the loop is the same as the length of the list.\n",
    "* change the code so that the printed count starts at 1, rather than 0. \n",
    "\n",
    "Hint: how can you make it print `count+1` rather than `count`? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.3 `slice` \n",
    "\n",
    "A string can be thought of as an ordered 'array' of characters. \n",
    "\n",
    "So, for example the string `hello` can be thought of as a construct containing `h` then `e`, `l`, `l`, and `o`. \n",
    "\n",
    "We can index a string, so that e.g. `'hello'[0]` is `h`, `'hello'[1]` is `e` etc.\n",
    "\n",
    "We have seen above the idea of the 'length' of a string. In this example, the length of the string `hello` is 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "string = 'hello'\n",
    "\n",
    "# length\n",
    "slen = len(string)\n",
    "print('length of {} is {}'.format(string,slen))\n",
    "\n",
    "# select these indices\n",
    "indices = 0,1,3\n",
    "\n",
    "# loop over each item in indices\n",
    "for index in indices:\n",
    "    print('character {} of {} is {}'.format(index,string,string[index]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise E1.2.4**\n",
    "\n",
    "* copy the code above, and see what happens if you set a value in `indices` that is the value of length of the string. Why does it respond so?\n",
    "* make the code robust to this issue, but using an `if` statement to test if `index` is in the required range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "We can use the idea of a 'slice' to access particular elements within the string.\n",
    "\n",
    "For a slice, we can specify:\n",
    "\n",
    "* start index (0 is the first)\n",
    "* stop index (not including this)\n",
    "* skip (do every 'skip' character)\n",
    "\n",
    "When specifying this as array access, this is given as, e.g.:\n",
    "\n",
    "`array[start:stop:skip]`\n",
    "\n",
    "* The default start is 0\n",
    "* The default stop is the length of the array\n",
    "* The default skip is 1\n",
    "\n",
    "You can specify a slice with the default values by leaving the terms out:\n",
    "\n",
    "`array[::2]`\n",
    "\n",
    "would give values in the array `array` from 0 to the end, in steps of 2.\n",
    "\n",
    "This idea is fundamental to array processing in Python. We will see later that the same mechanism applies to all ordered groups.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"Hello World\"\n",
    "print (s,len(s))\n",
    "\n",
    "start = 0\n",
    "stop  = 11\n",
    "skip  = 2\n",
    "print (s[start:stop:skip])\n",
    "\n",
    "# use -ve numbers to specify from the end\n",
    "# use None to take the default value\n",
    "\n",
    "start = -3\n",
    "stop  = None\n",
    "skip  = 1\n",
    "print (s[start:stop:skip])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise E1.2.5**\n",
    "\n",
    "The example above allows us to access an individual character(s) of the array.\n",
    "\n",
    "* copy the example above, and print the string starting from the default start value, up to the default stop value, in steps of `2`.\n",
    "* write code to print out the 4$^{th}$ letter (character) of the string `s`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.4 `replace`\n",
    "\n",
    "We can replace all occurrences of a string within a string by some other string. We can also replace a string by an empty string, thus in effect removing it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"I'm a very happy string\".replace(\"happy\", \"unhappy\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise E1.2.6**\n",
    "\n",
    "* copy the statement above, and use the `replace` method to make it print out `\"I'm a happy string\"`. \n",
    "\n",
    "Hint: you want to replace the string `very ` with, effectively, nothing, i.e. a zero-length string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.5 `find`\n",
    "\n",
    "Quite often, we might want to find a string inside another string, and potentially give the location (as in characters from the start of the string) where this string occurs. We can use the `find` method, which will return either a `-1` if the string isn't found, or an integer giving the index of where the string starts (for the first time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"I'm a very happy string\".find(\"a\"))\n",
    "print (\"I'm a very happy string\".find(\"happy\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the idea of `find()` to sort out a messy table of data that we get from a web page.\n",
    "\n",
    "First, we need to import the package `requests` to access some information from a [URL](https://en.wikipedia.org/wiki/URL) (from a web page). The data we get will be in [html](https://en.wikipedia.org/wiki/HTML).\n",
    "\n",
    "The data we will examine is a dataset of [ENSO](https://en.wikipedia.org/wiki/ENSO) values for each month of the year from January 1950 to present, made available by [NOAA](https://en.wikipedia.org/wiki/NOAA)/\n",
    "\n",
    "If you visit  you will see the data table we are interested in. So, how do we 'grab' this?\n",
    "\n",
    "The [URL](https://en.wikipedia.org/wiki/URL) points to [html](https://en.wikipedia.org/wiki/HTML) code. When you display this in a browser, it is rendered appropriately. \n",
    "\n",
    "If you access the html directly, you will get the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Web scraping example\n",
    "\n",
    "import requests\n",
    "\n",
    "url = \"http://www.esrl.noaa.gov/psd/enso/mei.old/table.html\"\n",
    "\n",
    "# This line will pull the URL data as a string\n",
    "txt = requests.get(url).text\n",
    "\n",
    "# show the first 1000 characters (see 'slice' above: this is the same as [None:1000:None])\n",
    "print(txt[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice the presence of html codes in the text string (e.g. `<html>`, `<pre>`). There are particular packages for neatly parsing html (scraping information from web pages), one of the most common being [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/). This will tend to be more useful if the html is well fomatted, and the data contained in `<table>` sections, or similar structures. Here, we just have a block of text in the `<pre>` section.\n",
    "\n",
    "If we want to *just* access the dataset here then, we might notice that the data we want to access starts when we see the string `YEAR`.\n",
    "\n",
    "We can use `find()` to discover the index of this in the string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = txt.find('YEAR')\n",
    "\n",
    "print('start of useful data at index {}\\n---------------------------------'.format(start))\n",
    "print(txt[start:start+1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look again at the web page [http://www.esrl.noaa.gov/psd/enso/mei.old/table.html](http://www.esrl.noaa.gov/psd/enso/mei/table.html), we might notice that the end of the useful data is delimited by two newlines and the string `(1)`, i.e., as a string `\\n\\n(1)`. So we should be able to use `find()` again to get the location of the end of the data (i.e. `stop`, in the sense of a slice).\n",
    "\n",
    "**Exercise 1.2.6**\n",
    "\n",
    "* use this observation to form a string called `data_table`, containing all of the useful data (i.e. `txt[start:stop]`).\n",
    "* print the string `data_table`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This exercise is a very good example of [web scraping](https://en.wikipedia.org/wiki/Web_scraping). Web scraping is often rather messy (you have to work out some 'key' to reliably delimit the information you want) but can be extreemely valuable for accessing datasets that are not cleanly presented. We have only gonbe part of the way to extracting a useful dataset here, because the dataset we are interested in (the ENSO data) are still represented as a string, whereas we really want them to be a set of floating point numbers. We will deal with this later.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.5 `split` and `splitlines`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first 'line' of `` should contain the 'header' information, i.e. the title of the data columns (`YEAR`, `DECJAN` etc.). We want to separate the header from the numbers in the data table, so we want to 'split' the string called `data_table` into a header string and data string. \n",
    "\n",
    "One approach to this would be split the string into 'lines' of text (rather than one block). Effectively that means splitting into multiple strings whenever we hit a `\\n` character. Rather than do that explicitly, we use the `splitlines()` method:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "url = \"http://www.esrl.noaa.gov/psd/enso/mei.old/table.html\"\n",
    "txt = requests.get(url).text\n",
    "\n",
    "# copy the useful data\n",
    "start = txt.find('YEAR')\n",
    "stop  = txt.find('\\n\\n(1)')\n",
    "data_table = txt[start:stop]\n",
    "\n",
    "# split into a list of strings\n",
    "data_lines = data_table.splitlines()\n",
    "\n",
    "# tell me something useful\n",
    "print(type(data_lines),len(data_lines))\n",
    "\n",
    "# loop over some examples\n",
    "for i in 0,1,len(data_lines)-1:\n",
    "    print('line {} {}\\n\\t{}'.format(i,type(data_lines[i]),data_lines[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This splits each 'line' of text into an entry in a `list`, so that the header data is now given in the first entry (`data_lines[0]`) and the lines containinmg data, after that.\n",
    "\n",
    "\n",
    "From the print out above, we notice that the final 'data line' (index `-1`) is shorter than (has fewer entries than) the other lines. This is because we are only part way through this year!.\n",
    "\n",
    "In 'real' datasets, we quite often have 'messy' lines of data such as this (or data missing for other reasons). How you want to deal with the 'messy bits' depends on the sort of analysis you want to do. \n",
    "\n",
    "One option (the simplest) would be to simply remove the last line (ignore this year's data):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "header = data_lines[0]\n",
    "\n",
    "# select the data block as being from entry 1 to -1\n",
    "# so, **not including the last row**\n",
    "data = data_lines[1:-1]\n",
    "\n",
    "print('header:',header)\n",
    "\n",
    "for i in 0,1,len(data)-1:\n",
    "    print('line {} {}\\n\\t{}'.format(i,type(data[i]),data[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1.2.7**\n",
    "\n",
    "* copy the code from above and explore the response using line indices `-1` and `-2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to manipulate or plot the information contained in this (the numbers), we need to convert each of the string representations to a floating point number, e.g. the number `-1.03` rather than the string `'-1.03'`.\n",
    "\n",
    "Each entry in the list `data` is a string, as we saw above.\n",
    "\n",
    "We can split an individual string (such as `data[0]` into a list of strings, using the string method `split()`. By default, this splits on 'white space' (i.e. spaces or tab characters), so, e.g.:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = data[0].split()\n",
    "print(data[0])\n",
    "print(line,len(line))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we have split the long string into 13 strings in a list. \n",
    "\n",
    "We want to generate a new list with 13 corresponding floating point values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the line on whitespace\n",
    "line = data[0].split()\n",
    "\n",
    "# make a new list of the same length\n",
    "# by copying the variable line\n",
    "float_data = line.copy()\n",
    "\n",
    "for index,line_data in enumerate(line):\n",
    "    # insert the cast float into the list\n",
    "    # in the right order (use index)\n",
    "    float_data[index] = float(line_data)\n",
    "    \n",
    "# this is the string list\n",
    "print(line)\n",
    "\n",
    "# this is the float list\n",
    "print(float_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1.2.8**\n",
    "\n",
    "* set a variable to be the string `\"2, 3, 5, 7, 11, 13, 17, 19, 23, 29\"`\n",
    "* use the approach above to generate a **list of integers** of the first 10 prime numbers. \n",
    "* print the list with syntax of the pattern of 'prime number 3 is 7'\n",
    "\n",
    "Make sure you convert each prime number to an integer, rather than leaving it as a string!\n",
    "\n",
    "Hint: We can still use the method `split()` to do split the string into a list of strings, but this time the [separator](https://python-reference.readthedocs.io/en/latest/docs/str/split.html) is a comma, rather than whitespace. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here\n",
    "pstring = \"2, 3, 5, 7, 11, 13, 17, 19, 23, 29\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, we wouldn't go to the trouble of first copying the list. \n",
    "\n",
    "Instead, **where the contents of the loop are simple** (e.g. a single statement) we would use a different way of using a `for` loop, called an **implicit loop**.\n",
    "\n",
    "In this case:\n",
    "\n",
    "    for item in group:\n",
    "        doit(group)\n",
    "        \n",
    "becomes:\n",
    "\n",
    "    [doit(group) for item in group]\n",
    "    \n",
    "with the additional feature that everything returned by `doit(group)` for each item of `group` is put in a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the line on whitespace\n",
    "\n",
    "# implicit for loop\n",
    "float_data = [float(line_data) for line_data in data[0].split()]\n",
    "    \n",
    "# this is the string list\n",
    "print(line)\n",
    "# this is the float list\n",
    "print(float_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The statement:\n",
    "\n",
    "    float_data = [float(line_data) for line_data in line]\n",
    "\n",
    "is much more [Pythonic](https://docs.python-guide.org/writing/style/) than the code above. It is simple, elegant and neat."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can *nest* for statements, i.e. put one for loop inside another. This allows us to treat data of multiple dimensions.\n",
    "\n",
    "In the examples above, we converted only the data in `data[0]` to a list of floating point numbers.\n",
    "If we wanted to process *all* lines of data, we would have to loop over them as well, in an 'outer' loop.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use a step of 10 for illustration purposes\n",
    "# to save space when printing\n",
    "\n",
    "step = 10\n",
    "\n",
    "for index,line in enumerate(data_table.splitlines()[1:-1:step]):\n",
    "    # convert each line to list of floats\n",
    "    float_data = [float(line_data) for line_data in line.split()]\n",
    "    print('line {} is {}'.format(index*step,float_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that whilst we have calculated `float_data` in the loop for each line, it gets over-written with each new line as things stand.\n",
    "\n",
    "We can do the same thing, and generate a list of the responses more neatly, using an implicit loop inside another implicit loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_float_data = [[float(line_data) for line_data in line.split()] for line in data_table.splitlines()[1:-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable `all_float_data` is now a sort of 'two dimensional' list, within which we can refer to individual items as e.g. `all_float_data[10][3]` for row `10`, column `3`.\n",
    "\n",
    "Let's use this idea to print out column 0 of each row (containing the `YEAR` data). We will use the method `range(nrows)` that (implicitly) generates a list `[0,1,2,3, ..., nrows-1]`. \n",
    "\n",
    "Notice the use of `end=' '` in the `print` statement. This replaces the usual newline by whetever is specified by the keyword `end`. Note also that we have used `{:.0f}` to specify the format term. This indicates that the term is to be printed as a floating point number (the `f`) with zero numbers after the decimal point (`.0`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nrows = len(all_float_data)\n",
    "i = 0\n",
    "\n",
    "print('column {} of the data gives:\\n'.format(i))\n",
    "for row in range(nrows):\n",
    "    print('{:.0f}'.format(all_float_data[row][i]),end=' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1.2.9**\n",
    "\n",
    "* use an implicit loop to create a list of ENSO values in a variable `enso` for the years 1950 up to last year for the period `DECJAN`.\n",
    "* produce a plot of ENSO for `DECJAN` as a function of year (see below on how to do that).\n",
    "\n",
    "Hint: check which column in the header is `DECJAN`. To start you off on this, we give you the implicit loop code for extracting the column containing the `YEAR` data (column 0). We also give you the code to achieve the plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here\n",
    "\n",
    "# generate a list called years of column 0 data\n",
    "years = [all_float_data[row][0] for row in range(nrows)]\n",
    "\n",
    "# you need to put the enso data in here!\n",
    "# this is put in as a dummy that should plot a straight line!\n",
    "enso = years.copy()\n",
    "\n",
    "# for plotting\n",
    "import pylab as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# \n",
    "plt.figure(0,figsize=(12,3))\n",
    "plt.plot(years,enso)\n",
    "plt.xlabel('year')\n",
    "plt.ylabel('ENSO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.6 Summary\n",
    "\n",
    "In section 1.2 you have been introduced to text representation in Python, as strings (type `str`), and shown that this sort of variable can be thought of an an 'array', and that it has a length attribute that can be accessed with `len()`.\n",
    "\n",
    "Other useful string manipulation methods you were introduced to are: `replace()`, `find()`, `split()` and `splitlines()`, though of course there are [many more](https://docs.python.org/3/library/string.html).\n",
    "\n",
    "In an 'array', we can use an index to refer to a particular item (e.g. index 0 for the first item, 1 for the second, -1 for the last). We can use this idea to manipulate strings. \n",
    "\n",
    "In a more general sense, we can take a 'slice' of an array, with the syntax `[start:stop:skip]` giving access to a regularly spaced part of an array. We can use this, for example, to print out every 10th value (`skip=10`).\n",
    "\n",
    "You were also introduced to the idea of looping control structures, using a `for ... in ...:` statement, and the equivalent implicit form. This introduced the idea of [indented code blocks](https://wiki.python.org/moin/Why%20separate%20sections%20by%20indentation%20instead%20of%20by%20brackets%20or%20%27end%27) and (related) nested structures (loops within loops).\n",
    "\n",
    "In passing, you have also been shown how to pull html data from a URL (scraping) using the [`requests`](http://docs.python-requests.org/en/master/) package, and also how to produce a simple data plot, using [`pylab`](https://matplotlib.org/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3. Groups of things\n",
    "Very often, we will want to group items together. There are several main mechanisms for doing this in Python, known as:\n",
    "\n",
    "* string e.g. `hello`\n",
    "* tuple, e.g. `(1, 2, 3)`\n",
    "* list, e.g. `[1, 2, 3]`\n",
    "* numpy array e.g. `np.array([1, 2, 3])`\n",
    "\n",
    "A slightly different form of group is a dictionary:\n",
    "\n",
    "* dict, e.g. `{1:'one', 2:'two', 3:'three'}`\n",
    "\n",
    "You will notice that each of the grouping structures tuple, list and dict use a different form of bracket. The numpy array is fundamental to much work that we will do later.\n",
    "\n",
    "We have dealt with the idea of a string as an ordered collection in the material above, so will deal with the others here.\n",
    "\n",
    "We noted the concept of length (`len()`), that elements of the ordered collection could be accessed via an index, and came across the concept of a slice. All of these same ideas apply to the first set of groups (string, tuple, list, numpy array) as they are all ordered collections.\n",
    "\n",
    "A dictionary is not (by default) ordered, however, so indices have no role. Instead, we use 'keys'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.1 `tuple`\n",
    "A tuple is a group of items separated by commas. In the case of a tuple, the brackets are optional.\n",
    "You can have a group of differnt types in a tuple (e.g. int, int, str, bool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load into the tuple\n",
    "t = (1, 2, 'three', False)\n",
    "\n",
    "# unload from the tuple\n",
    "a,b,c,d = t\n",
    "\n",
    "print(t)\n",
    "print(a,b,c,d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there is only one element in a tuple, you must put a comma , at the end, otherwise it is not interpreted as a tuple:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = (1)\n",
    "print (t,type(t))\n",
    "t = (1,)\n",
    "print (t,type(t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can have an empty tuple though:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = ()\n",
    "print (t,type(t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.3.1 Exercise**\n",
    "\n",
    "* create a tuple called t that contains the integers 1 to 5 inclusive\n",
    "* print out the value of t\n",
    "* use the tuple to set variables a1,a2,a3,a4,a5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 1.3.2  `list`\n",
    "A `list` is similar to a `tuple`. One main difference is that you can change individual elements in a list but not in a tuple.\n",
    "To convert between a list and tuple, use the 'casting' methods `list()` and `tuple()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# a tuple\n",
    "t0 = (1,2,3)\n",
    "\n",
    "# cast to a list\n",
    "l = list(t0)\n",
    "\n",
    "# cast to a tuple\n",
    "t = tuple(l)\n",
    "\n",
    "print('type of {} is {}'.format(t,type(t)))\n",
    "print('type of {} is {}'.format(l,type(l)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can concatenate (join) lists or tuples with the `+` operator:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l0 = [1,2,3]\n",
    "l1 = [4,5,6]\n",
    "\n",
    "l = l0 + l1\n",
    "print ('joint list:',l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.3.2 Exercise**\n",
    "* copy the code from the cell above, but instead of lists, use tuples\n",
    "* loop over each element in the tuple and print out the data type and value of the element\n",
    "\n",
    "Hint: use a `for ... in ...` construct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common method associated with lists or tuples is:\n",
    "* `index()`\n",
    "\n",
    "Some useful methods that will operate on lists and tuples are:\n",
    "* `len()`\n",
    "* `sort()`\n",
    "* `min(),max()`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l0 = (2,8,4,32,16)\n",
    "\n",
    "# print the index of the item integer 4 \n",
    "# in the tuple / list\n",
    "\n",
    "item_number = 4\n",
    "\n",
    "# Note the dot . here\n",
    "# as index is a method of the class list\n",
    "ind  = l0.index(item_number)\n",
    "\n",
    "# notice that this is different\n",
    "# as len() is not a list method, but \n",
    "# does operatate on lists/tuples\n",
    "# Note: do not use len as a variable name!\n",
    "llen = len(l0)\n",
    "\n",
    "# note the use of integers in the braces e.g. {0}\n",
    "# rather than empty braces as before. This allows us to\n",
    "# refer to particular items in the format argument list\n",
    "print('the index of {0} in {1} is {2}'.format(item_number,l0,ind))\n",
    "print('the length of the {0} {1} is {2}'.format(type(l0),l0,llen))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**E1.3.3 Exercise**\n",
    "\n",
    "* copy the code to the block below, and test that this works with lists, as well as tuples\n",
    "* find the index of the integer 16 in the tuple/list\n",
    "* what is the index of the first item?\n",
    "* what is the length of the tuple/list?\n",
    "* what is the index of the last item?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A list has a much richer set of methods than a tuple. This is because we can add or remove list items (but not tuple).\n",
    "\n",
    "* `insert(i,j)` : insert `j` beore item `i` in the list\n",
    "* `append(j)` : append `j` to the end of the list\n",
    "* `sort()` : sort the list\n",
    "\n",
    "This shows that tuples and lists are 'ordered' (i.e. they maintain the order they are loaded in) so that indiviual elements may be accessed through an 'index'. The index values start at 0 as we saw above. The index of the last element in a list/tuple is the length of the group, minus 1. This can also be referred to an index `-1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l0 = [2,8,4,32,16]\n",
    "\n",
    "# insert 64 at the begining (before item 0)\n",
    "# Note that this inserts 'in place'\n",
    "# i.e. the list is changed by calling this\n",
    "l0.insert(0,64)\n",
    "\n",
    "\n",
    "# insert 128 *before* the last item (item -1)\n",
    "l0.insert(-1,128)\n",
    "\n",
    "# append 256 on the end\n",
    "l0.append(256)\n",
    "\n",
    "# copy the list \n",
    "# and sort the copy\n",
    "# Note the use of the copy() method here\n",
    "# to create a copy\n",
    "l1 = l0.copy()\n",
    "\n",
    "# Note that this sorts 'in place'\n",
    "# i.e. the list is changed by calling this\n",
    "l1.sort()\n",
    "\n",
    "print('the list {0} once sorted is {1}'.format(l0,l1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E1.3.4 Exercise**\n",
    "\n",
    "* copy the above code and try out some different locations for inserting values (e.g. what does index `-2` mean?)\n",
    "* what happens if you take off the `.copy()` statement in the line `l1 = l0.copy()`, i.e. just use `l1 = l0`?  [Why is this?](https://www.afternerd.com/blog/python-copy-list/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.3 `np.array`\n",
    "\n",
    "An array is a group of objects of the same type. Because they are of the same type, they can be stored efficiently in compter memory, and also accessed efficiently.\n",
    "\n",
    "Whilst there are different ways of forming arrays, the most common is to use numpy arrays, using the package `numpy`. To use this, we must first import the package into the current workspace. We do this with the `import` method. Using the optional `as` statement allows us to use a shorter (or more suitable) name for the package. We will generally call numpy `np`, so we use:\n",
    "\n",
    "`import numpy as np`\n",
    "\n",
    "to import ('load') the numpy package. \n",
    "\n",
    "Often, we will read data from a file/URL as we did above for the ENSO dataset. In that case, we had to step through each item to convert from string form to floating point number.\n",
    "\n",
    "This sort of thing is much more simply done using methods associated with numpy arrays. \n",
    "\n",
    "A particularly useful numpy method is `np.loadtxt(file)` that loads an ASCII table of data straight into a numpy array.\n",
    "\n",
    "Whilst this is designed to load data from a file, we can use `io.StringIO()` from the `io` package to make data that we already have as a string seem to `np.loadtxt` as if it were a file. This is a useful 'trick' for using methods that expect data in a file. The `unpack=True` option makes sure the data array is compoised the way we would expect it. The `usecols` option lets us select only those data columns we wish to read (0 and 1 here).\n",
    "\n",
    "\n",
    "An alternative to `np.loadtxt()` is `np.genfromtxt()`. This has some additional features, such the `invalid_raise` flag. If this is set `False`, the loading is made somewhat tolerant to data errors (e.g. inconsistent number of columns). Further, we can explicitly set what will indicate `missing_values` in the input and what we would like to replace them with (`filling_values`) which can be useful for tidying up datasets.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import numpy as np\n",
    "import io\n",
    "\n",
    "# access dataset as above\n",
    "url = \"http://www.esrl.noaa.gov/psd/enso/mei.old/table.html\"\n",
    "txt = requests.get(url).text\n",
    "\n",
    "# copy the useful data\n",
    "start_head = txt.find('YEAR')\n",
    "start_data = txt.find('1950\\t')\n",
    "stop_data  = txt.find('2018\\t')\n",
    "\n",
    "# select a data column\n",
    "data_column = 1\n",
    "\n",
    "header = txt[start_head:start_data].split()\n",
    "data = np.loadtxt(io.StringIO(txt[start_data:stop_data]),unpack=True,usecols=[0,data_column])\n",
    "\n",
    "# so data[0] is the year data\n",
    "#    data[1] is the enso data for column data_column\n",
    "# print some attributes of the data array\n",
    "\n",
    "print('array type',type(data))\n",
    "print('data type',data.dtype)\n",
    "print('number of dimensions',data.ndim)\n",
    "print('data shape',data.shape)\n",
    "print('data size',data.size)\n",
    "\n",
    "# for plotting\n",
    "import pylab as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# \n",
    "plt.figure(0,figsize=(12,3))\n",
    "plt.plot(data[0],data[1],label=header[data_column])\n",
    "plt.xlabel('year')\n",
    "plt.ylabel('ENSO')\n",
    "plt.title('ENSO data from {0}'.format(url))\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw in the example above that a numpy array (`<class 'numpy.ndarray'>`) has a set of attributes that include `shape`, `ndim`, `dtype` and `size` that we can use to query information about the array. We will learn morre about processing data with numpy arrays later in the course, but you should already see that they are a useful construct for manipulating multi-dimensional datasets.\n",
    "\n",
    "**Exercise 1.3.4**\n",
    "\n",
    "* copy the code from the block above and modify it to plot the ENSO data for the period `FEBMAR`. Check this by looking at the data in the [original table](http://www.esrl.noaa.gov/psd/enso/mei/table.html).\n",
    "* modify the code to produce a plot of *all* periods (so the graph should have 12 lines, correctly labelled)\n",
    "\n",
    "Hint: You will need to consider what, if anything to set of `usecols` (what happends if you don't set `usecols`?) and provide a looping structure for the plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.4 `dict`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The collections we have used so far have all been ordered. This means that we can refer to a particular element in the group by an index, e.g. `array[10]`.\n",
    "\n",
    "A dictionary is not (by default) ordered. Instead of indices, we use 'keys' to refer to elements: each element has a key associated with it. It can be very useful for data organisation (e.g. databases) to have a key to refer to, rather than e.g. some arbitrary column number in a gridded dataset.\n",
    "\n",
    "A dictionary is defined as a group in braces (curley brackets). For each elerment, we specify the key and then the value, separated by `:`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = {'one': 1, 'two': 2, 'three': 3}\n",
    "\n",
    "# we then refer to the keys and values in the dict as:\n",
    "\n",
    "print ('a:\\n\\t',a)\n",
    "print ('a.keys():\\n\\t',a.keys())     # the keys\n",
    "print ('a.values():\\n\\t',a.values()) # returns the values\n",
    "print ('a.items():\\n\\t',a.items())   # returns a list of tuples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because dictionaries are not ordered, we cannot guarantee the order they will come out in a `for` loop, but we will often use such a loop to iterate over the items in a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key,value in a.items():\n",
    "    print(key,value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We refer to specific items using the key e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a['one'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add to a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.update({'four':4,'five':5})\n",
    "print(a)\n",
    "\n",
    "# or for a single value\n",
    "a['six'] = 6\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quite often, you find that you have the keys you want to use in a dictionary as a list or array, and the values in another list.\n",
    "\n",
    "In such a case, we can use the method `zip(keys,values)` to load into the dictionary. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "values = [1,2,3,4]\n",
    "keys = ['one','two','three','four']\n",
    "\n",
    "a = dict(zip(keys,values))\n",
    "\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use this idea to make a dictionary of our ENSO dataset, using the items in the header for the keys. In this way, we obtain a  more elegant representation of the dataset, and can refer to items by names (keys) instead of column numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import numpy as np\n",
    "import io\n",
    "\n",
    "# access dataset as above\n",
    "url = \"http://www.esrl.noaa.gov/psd/enso/mei.old/table.html\"\n",
    "txt = requests.get(url).text\n",
    "\n",
    "# copy the useful data\n",
    "start_head = txt.find('YEAR')\n",
    "start_data = txt.find('1950\\t')\n",
    "stop_data  = txt.find('2018\\t')\n",
    "\n",
    "header = txt[start_head:start_data].split()\n",
    "data = np.loadtxt(io.StringIO(txt[start_data:stop_data]),unpack=True)\n",
    "\n",
    "# use zip to load into a dictionary\n",
    "data_dict = dict(zip(header, data))\n",
    "\n",
    "key = 'MAYJUN'\n",
    "# plot data\n",
    "plt.figure(0,figsize=(12,7))\n",
    "plt.title('ENSO data from {0}'.format(url))\n",
    "plt.plot(data_dict['YEAR'],data_dict[key],label=key)\n",
    "plt.xlabel('year')\n",
    "plt.ylabel('ENSO')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1.3.5**\n",
    "\n",
    "* copy the code above, and modify so that datasets for months `['MAYJUN','JUNJUL','JULAUG']` are plotted on the graph\n",
    "\n",
    "Hint: use a for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also usefully use a dictionary with a printing format statement. In that case, we refer directly to the key in ther format string. This can make printing statements much easier to read. We don;'t directly pass the dictionary to the `fortmat` staterment, but rather `**dict`, where `**dict` means \"treat the key-value pairs in the dictionary as additional named arguments to this function call\".\n",
    "\n",
    "So, in the example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import numpy as np\n",
    "import io\n",
    "\n",
    "# access dataset as above\n",
    "url = \"http://www.esrl.noaa.gov/psd/enso/mei/table.html\"\n",
    "txt = requests.get(url).text\n",
    "\n",
    "# copy the useful data\n",
    "start_head = txt.find('YEAR')\n",
    "start_data = txt.find('1950\\t')\n",
    "stop_data  = txt.find('2018\\t')\n",
    "\n",
    "header = txt[start_head:start_data].split()\n",
    "data = np.loadtxt(io.StringIO(txt[start_data:stop_data]),unpack=True)\n",
    "\n",
    "# use zip to load into a dictionary\n",
    "data_dict = dict(zip(header, data))\n",
    "print(data_dict.keys())\n",
    "\n",
    "# print the data for MAYJUN\n",
    "print('data for MAYJUN: {MAYJUN}'.format(**data_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line `print('data for MAYJUN: {MAYJUN}'.format(**data_dict))` is equivalent to writing:\n",
    "\n",
    "    print('data for {MAYJUN}'.format(YEAR=data_dict[YEAR],DECJAN=data_dict[DECJAN], ...))\n",
    "    \n",
    "In this way, we use the keys in the dictionary as keywords to pass to a method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another useful example of such a use of a dictionary is in saving a numpy dataset to file.\n",
    "\n",
    "If the data are numpy arrays in a dictionary as above, we can store the dataset using:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import numpy as np\n",
    "import io\n",
    "\n",
    "# access dataset as above\n",
    "url = \"http://www.esrl.noaa.gov/psd/enso/mei/table.html\"\n",
    "txt = requests.get(url).text\n",
    "\n",
    "# copy the useful data\n",
    "start_head = txt.find('YEAR')\n",
    "start_data = txt.find('1950\\t')\n",
    "stop_data  = txt.find('2018\\t')\n",
    "\n",
    "header = txt[start_head:start_data].split()\n",
    "data = np.loadtxt(io.StringIO(txt[start_data:stop_data]),unpack=True)\n",
    "\n",
    "# use zip to load into a dictionary\n",
    "data_dict = dict(zip(header, data))\n",
    "\n",
    "filename = 'enso_mei.npz'\n",
    "\n",
    "# save the dataset\n",
    "np.savez_compressed(filename,**data_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we load from the file is a dictionary-like object `<class 'numpy.lib.npyio.NpzFile'>`.\n",
    "\n",
    "If needed, we can cast this to a dictionary with `dict()`, but it is generally more efficient to keep the original type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset\n",
    "\n",
    "filename = 'enso_mei.npz'\n",
    "\n",
    "loaded_data = np.load(filename)\n",
    "\n",
    "print(type(loaded_data))\n",
    "\n",
    "# test they are the same using np.array_equal\n",
    "for k in loaded_data.keys():\n",
    "    print('\\t',k,np.array_equal(data_dict[k], loaded_data[k]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1.3.6**\n",
    "\n",
    "* Using what you have learned above, access the Met Office data file (`https://www.metoffice.gov.uk/hadobs/hadukp/data/monthly/HadSEEP_monthly_qc.txt`)[https://www.metoffice.gov.uk/hadobs/hadukp/data/monthly/HadSEEP_monthly_qc.txt] and create a 'data package' in a numpy`.npz` file that has keys of `YEAR` and each month in the year, with associated datasets of Monthly Southeast England precipitation (mm).\n",
    "* confirm that tha data in your `npz` file is the same as in your original dictionary\n",
    "* produce a plot of October rainfall using these data for the years 1900 onwards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.5 Summary\n",
    "\n",
    "In this section, we have extended the types of data we might come across to include groups . We dealt with ordered groups of various types (`tuple`, `list`), and introduced the numpy package for numpy arrays (`np.array`). We saw dictionaries as collections with which we refer to individual items with a key.\n",
    "\n",
    "We learned in the previous section how to pull apart a dataset presented as a string using loops and various using methods and to construct a useful dataset 'by hand' in a list or similar structure. It is useful, when learning to program, to know how to do this.\n",
    "\n",
    "Here, we saw that packages such as numpy provide higher level routines that make reading data easier, and we would generally use these in practice. We saw how we can use `zip()` to help load a dataset from arrays into a dictionary, and also the value of using a dictionary representation when saving numpy files."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "nbTranslate": {
   "displayLangs": [
    "*"
   ],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "en",
   "targetLang": "fr",
   "useGoogleTranslate": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "207px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}