day_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Programming for Scientists - Day 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "In the second day we will consider:\n",
    "\n",
    "* formatting print output\n",
    "* reading/writing text files\n",
    "* reading/writing binary (HDF5) files\n",
    "* modules\n",
    "* installing new modules with pip\n",
    "* the standard library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [1] Input and Output\n",
    "\n",
    "### Formatting (print) output\n",
    "\n",
    "There are several ways to present the output of a program: data can be printed in a human-readable form, or written to a file for future use. So far we have seen the `print()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 5\n",
    "print('starting')\n",
    "print(x)\n",
    "print('x = ', x, ' which is great!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print strings and variables, which is enough for some simple use cases, but there are three ways to have more control with print:\n",
    "1. \"Old\" formatting with `%`\n",
    "2. The `.format()` method\n",
    "3. \"f-strings\"\n",
    "\n",
    "The first, \"old formatting\", is easy to use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = 2.2\n",
    "print('x = %d' % x)\n",
    "print('x = %d, y = %f' % (x,y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expressions `%d` and `%f` are examples of [format codes](https://docs.python.org/3/library/string.html#formatspec), which tell the `print()` function that this should be replaced by the value of a variable.\n",
    "\n",
    "The variable(s) are specified listing them using the format: `string % (var1,var2,var3)`.\n",
    "\n",
    "The most common and useful formatting codes are:\n",
    "* `%s` for a string\n",
    "* `%d` for an integer number\n",
    "* `%4d` for an integer number, padding it with spaces to be at least 4 characters long.\n",
    "* `%04d` for an integer number, padding it with zeros to be at least 4 characters long.\n",
    "* `%e` scientific notation, for any floating number.\n",
    "* `%f` for a floating number, automatic number of digits shown.\n",
    "* `%.2f%` for a floating number, showing two digits after the decimal place.\n",
    "* `%8f` for a floating number, showing eight digits in total.\n",
    "* `%6.2f` for a floating number, showing six digits in total, with two after the decimal place, leaving three before the decimal place (which counts as one character)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Assign the number 123 to a variable and print it padded to five characters with zeros. Then, print it using the `%s` format code - what happens, and why?\n",
    "\n",
    "Assign the number 4.2311e2 to a variable, and print it with three different representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second method uses the same format codes, but in a slightly different syntax with `.format()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('We say {} and think that x = {}'.format('thanks',x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('We say {word} and think that x = {myvar}'.format(word='hi',myvar=x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('The second entry is {1} while the first is {0}'.format(x,y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ages = {'tom':22, 'sam':23, 'philip':18}\n",
    "print('Tom is {tom:d} years old, while Philip is {philip:d}'.format(**ages))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last method is using [f-strings](https://docs.python.org/3/reference/lexical_analysis.html#f-strings) (i.e. \"formatted strings\"). This is a modern and suggested way to do formatted output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = 'never!'\n",
    "print(f'We say {word} and think that x = {x}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'We chose {word = } and know that {x = }')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'{x = :4d} and {y = :.2f}, and also y = {y:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of the three methods, you can use whichever seems the most natural and easiest to read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Write a function that takes six integer inputs: (year, month, day, hour, minutes, and seconds) and converts them to a string with format `2006-03-22 13:12:55`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Write a function that takes a string formatted as `2006-03-22 13:12:55` and returns a 6-tuple of integers giving the corresponding year, month, day, hour, minutes, and seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading (text) files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we have only been working with variables and data which we have explicitly written. In data processing and data analysis, most often you will need to load data from \"external\" files (on the filesystem).\n",
    "\n",
    "The simplest case, which always works well for very small datasets, is a simple text file. When the file contains values separated by spaces or commas, we call this a [CSV file](https://en.wikipedia.org/wiki/Comma-separated_values) (comma-separated value).\n",
    "\n",
    "There are many ways to read such files in Python. First, we can use the built-in `open()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/day2_stars.txt','r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first argument is a string containing the filename. The second argument is another string containing a few characters describing the way in which the file will be used. mode can be `'r'` when the file will only be read, `'w'` for only writing (an existing file with the same name will be erased), and `'a'` opens the file for appending; any data written to the file is automatically added to the end. `'r+'` opens the file for both reading and writing.\n",
    "\n",
    "Normally, files are opened in \"text mode\", that means, you read and write strings from and to the file. You can append `'b'` to the mode to open a file in binary mode: now the data is read and written in the form of bytes objects. This mode should be used for all files that don’t contain text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# f is an 'open' file object, we should use it to read, and then close\n",
    "lines = f.readlines()\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python it is good practice to use the **with keyword** when dealing with file objects. The advantage is that the file is guaranteed to be properly closed, even if an error occurs at some point while reading. (You should always make sure to close a file object, otherwise problems!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/day2_stars.txt','r') as f:\n",
    "    lines = f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The syntax works as follows: the `open()` function is run, and its return is assigned to the variable `f`. The indented code block which follows can then use this `f` file object. As soon as the indented code is finished, the file is automatically closed.\n",
    "\n",
    "Note that `readlines()` simply reads the entire file, one line at a time, and returns a list, where each item is a single line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Two other functions exist, `readline()` and `read()`. Try each, loading the same file as above, and compare the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another very pythonic way to read the lines of a file is to loop over them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/day2_stars.txt','r') as f:\n",
    "    for line in f:\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List (and dict) comprehension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A useful shorthand in Python, which can often save you from writing a loop to construct a list a dict, is called [list comprehension](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions). The syntax looks like:\n",
    "\n",
    "    [item for item in iterable_object] makes a list\n",
    "    {key:val for key in iterable_object} makes a dict\n",
    "    \n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "squares = [i**2 for i in range(10)]\n",
    "print(squares)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {i:'ok' for i in range(5)}\n",
    "print(my_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {i:i**3 for i in range(5)}\n",
    "print(my_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't go crazy with list comprehensions - you will often see Python experts writing this in such complex ways they are impossible to understand. Used in moderation, however, they can be quite helpful.\n",
    "\n",
    "Importantly, \"iterable_object\" is an idea you will often see in Python. It means anything which can return one element at a time, i.e. which can be \"iterated\" over. Lists and generators are two examples, as are file-like objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/day2_stars.txt','r') as f:\n",
    "    lines = [line for line in f]\n",
    "    \n",
    "print(len(lines))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regardless, we have the data \"loaded\", but so far it is just a list of strings, one string per line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Construct a list which contains only (and all of) the `Jmag` values. Then, compute the sum. Hint: loop over the lines (strings) we have loaded, use `.split()` to split each (using whitespace). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Construct a dictionary, which stores the RA ([right ascension](https://en.wikipedia.org/wiki/Right_ascension), i.e. position on the sky) as the values, using the NAME as the key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing (text) files\n",
    "\n",
    "Instead of reading a text file, we can write to a text file, simply by changing the mode from `'r'` to `'w'` and using the `.write(string)` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('output.txt','w') as f:\n",
    "    f.write(\"just a little test\\n\")\n",
    "    f.write(\"and some more\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As above, we can also use `writelines()` to write a list of strings all at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('output.txt','w') as f:\n",
    "    f.writelines([str(x) for x in squares])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Open up the \"output.txt\" file we have just made - use either a terminal, or the file explorer on the left. What's wrong? Fix it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Look at the text file `data/day2_gallazzi.txt`, identify the rows which are header (metadata), the number of columns, and so on.\n",
    "\n",
    "Load the file, and write a text file which contains only the stellar mass $M_\\star$ (first column) and metallicity $Z$ (second column) from the original file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading and writing (large) binary data\n",
    "\n",
    "For large numeric datasets, you rarely will want to read or write these as text (strings): the resulting files are larger to store, and precision may be lost if not enough digits are stored, for example.\n",
    "\n",
    "The alternative is to write \"binary\" data (meaning it is just a series of bytes representing numbers, rather than representing strings).\n",
    "\n",
    "In python you can read binary by changing the mode from `'r'` to `'rb'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/day2_numbers.bin','rb') as f:\n",
    "    data_bytes = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data_bytes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The contents of this file are just a series of bytes, which are impossible interpret without some prior knowledge. If we know that each byte encodes the value of a single integer, we can actually use the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for byte in data_bytes:\n",
    "    print(int(byte))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_numbers = [int(byte) for byte in data_bytes]\n",
    "print(data_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such files are called \"raw binary\", and they are difficult to work with. Ahead of time, you have no idea what the contents of the file are, nor how to read it correctly, unless it comes along with an exact description of how data has been arranged in the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In science and scientific computing, there are a number of binary data formats in common use - this depends on the particular field. Some common formats are:\n",
    "* [HDF5](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) in general, especially for simulations.\n",
    "* [FITS](https://en.wikipedia.org/wiki/FITS) in astrophysics.\n",
    "* [CDF](https://cdf.gsfc.nasa.gov/) in space sciences, earth sciences.\n",
    "* many others...\n",
    "\n",
    "These are all \"self-describing\" binary formats, meaning that they adhere to a well-known standard, so that you can understand the structure of a given file without any prior knowledge.\n",
    "\n",
    "In a single file they can store multiple datasets e.g. 1D arrays, 2D arrays, 3D arrays, etc, together with metadata (number of elements in an array, its physical units, and so on).\n",
    "\n",
    "## HDF5\n",
    "\n",
    "\n",
    "Let's look at a quick example with **HDF5**. First, we import the [h5py Python library](https://docs.h5py.org/en/stable/index.html) used to read and write HDF5 files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's **write** (create) a new HDF5 file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('test.hdf5','w') as f:\n",
    "    f['dataset1'] = squares\n",
    "    f['more_data'] = [float(sq) for sq in squares]\n",
    "    f['a_third_dataset'] = np.array([33,22,11])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have just created a new binary file named `test.hdf5` which contains two one-dimensional datasets (i.e. arrays).\n",
    "\n",
    "Now let's **read** an existing HDF5 file:\n",
    "\n",
    "### Exercise\n",
    "\n",
    "Open a terminal, and type `h5ls test.hdf5` to list the contents of this file. Try `h5ls -r test.hdf5` and `h5ls -rv test.hdf5` as well, for \"recursive\" and \"verbose\".\n",
    "How many datasets are there? How many entries in each? What is the data type of each?\n",
    "\n",
    "In the notebook, use `h5py` to read the first number from the `more_data` dataset in our newly created file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we don't know what datasets are in the file, or what their names are? We can use `.keys()`, just as with a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('test.hdf5','r') as f:\n",
    "    dset_names = list(f.keys())\n",
    "\n",
    "print(dset_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, we can read an entire dataset using the following syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('test.hdf5','r') as f:\n",
    "    entire_dataset = f['more_data'][()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But one of the powerful aspects of HDF5 and similar binary data formats is that you can load specific subsets of data.\n",
    "\n",
    "Imagine that you had an enormous 100GB data file, which is too large to load into the memory all at once. To load only the second through fifth entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('test.hdf5','r') as f:\n",
    "    data_subset = f['more_data'][1:5]\n",
    "    \n",
    "print(data_subset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how we are using the same indexing and slicing syntax that we have seen already, to tell h5py what subset of the dataset to load.\n",
    "\n",
    "> HDF5 is a rich format with many other features - take a look at the [h5py quickstart guide](https://docs.h5py.org/en/stable/quick.html#quick)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## pickle\n",
    "\n",
    "Often you just want to save some variable in python to a file, so that you can load it again later (or send it to a colleague).\n",
    "\n",
    "The `pickle` module makes this possible: it can convert -any- Python object (e.g. variable) into a piece of data which can be saved, and loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = {'a':'Some data','b':'Additional data'}\n",
    "y = [x, 33]\n",
    "\n",
    "# save\n",
    "with open('test.pickle','wb') as f:\n",
    "    pickle.dump(y,f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load\n",
    "with open('test.pickle','rb') as f:\n",
    "    y_loaded = pickle.load(f)\n",
    "print(y_loaded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing remote (online) data using APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is becoming increasingly popular, and powerful, to access data from remote, online resources (that is, websites).\n",
    "\n",
    "Services providing this functionality are called APIs (\"application program interfaces\"). For example:\n",
    "\n",
    "* You could use the [Twitter API](https://developer.twitter.com/en/docs) to search and retrieve all tweets containing a certain word posted on a certain day.\n",
    "* You could use a [RKI COVID API](https://api.corona-zahlen.org/docs/) to retrieve details on coronavirus infections in Germany.\n",
    "* You could retrieve historical data on the price of Bitcoin from the [Coinbase API](https://docs.cloud.coinbase.com/).\n",
    "\n",
    "APIs can not only provide data, but also services. For example:\n",
    "\n",
    "* You could use a [machine-learning image recognition API](https://docs.imagga.com/#introduction) to classify what an image (that you send) contains.\n",
    "\n",
    "Modern web APIs are almost always based on [REST](https://en.wikipedia.org/wiki/Representational_state_transfer), which simply means that:\n",
    "\n",
    "1. You send and recieve information with a standard \"web request\" to a particular URL (the same as going to the URL in your browser).\n",
    "2. Each request is \"stateless\", meaning that it is completely independent from any prior or future requests.\n",
    "\n",
    "They are very easy to use. Just to get a feel, let's look at the [randomuser.me API](https://randomuser.me). This is just a \"toy\" API for understanding the concepts. It generates random profiles of people."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [requests library](https://docs.python-requests.org/en/latest/) makes it very easy to make \"web\" (HTTP) requests in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify the specific API URL to access\n",
    "url = \"http://randomuser.me/api/\"\n",
    "\n",
    "# make request\n",
    "response = requests.get(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that the [response code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) indicates success (200)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can look at the actual text of the response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that this looks a bit like a dictionary, with a key called `results`. In fact, this is a string which is encoded in [JSON](https://en.wikipedia.org/wiki/JSON) format.\n",
    "\n",
    "**JSON** is a very common \"human-readable\" text format for sending and receiving data in Python, particularly across the web.\n",
    "\n",
    "In this case `requests` can automatically decode this response text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(response.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is, in fact, a normal Python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = response.json()\n",
    "type(r['results'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(r['results'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r['results'][0].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Make five requests to this API. For each, print out the name and phone number of the (automatically generated) person."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [2] Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you exit the Python interpreter and start it again (or select \"Restart Kernel\" in a Jupyter notebook), the definitions you have made (functions and variables) are lost.\n",
    "\n",
    "Therefore, if you want to write a longer and more complex program, the standard approach is to use a text editor to prepare the series of commands, and run that instead. This is known as creating a script.\n",
    "\n",
    "As your program gets longer, you will want to split it into several files for easier maintenance. You may also want to use a common function, written once, in several different programs, without copying its definition into each program.\n",
    "\n",
    "To achieve this, we create a Python **module**. Definitions from a module can be imported into other programs, scripts, or notebooks.\n",
    "\n",
    "We have already seen examples of module imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "# we can then use the math.cos() function of the math module\n",
    "math.cos(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to import a specific function or definition from a module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import cos\n",
    "\n",
    "# we can then use the cos() function without any prefix\n",
    "cos(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we can import every definition from a module into the current program, this is considered poor form and should **never** be done:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import * # puts cos, sin, and hundreds of other names into the current program, could overwrite things"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also rename a module, or a specific function from a module, when we import it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import cos as my_cos # perhaps to avoid a conflict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np # just for convenience"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a custom module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A \"module\", in its simplest form, is just a file with a `.py` extension."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Create a new file in the current directory (use the terminal, or right-click on the file explorer and select \"New File\"), name it `fibonacci.py`. Paste the following contents into it:\n",
    "\n",
    "    # Fibonacci numbers module\n",
    "\n",
    "    def fib(n):    # write Fibonacci series up to n\n",
    "        a, b = 0, 1\n",
    "        while a < n:\n",
    "            print(a, end=' ')\n",
    "            a, b = b, a+b\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we should be able to import it, and run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fibonacci"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fibonacci.fib(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A module can contain executable statements, as well as function definitions. Any executable statements are intended to initialize the module (generally, don't do this!), and are only executed the first time the module is imported.\n",
    "\n",
    "Modules can import other modules. It is customary but not required to place all import statements at the beginning of a module (or script, for that matter)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a custom module from the command-line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may want to run a piece of python code in the terminal like `python fibonacci.py <arguments>`. This may be useful for testing, for submitting the code to a compute cluster, and so on.\n",
    "\n",
    "To do this, we need to add the following code to the bottom of our `fibonacci.py` file:\n",
    "\n",
    "    if __name__ == \"__main__\":\n",
    "        import sys\n",
    "        fib(int(sys.argv[1]))\n",
    "        \n",
    "What is going on? The `if` statement is not within a function, so it is immediately executed when the module is imported, or when the file is run with the command `python fibonacci.py`.\n",
    "\n",
    "When the module is imported, the esoteric sounding condition `__name__ == \"__main__\"` is False, and nothing happens. But when run from the command line, this is True, in which case the `fib()` function is run, and the first command-line argument is passed as the argument for this function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Add this code to the bottom of our module file, and use the command-line (File -> New -> Terminal) to run it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note on how python finds modules: When you type `import mymodule`, python first searches for a built-in module with that name. If not found, it then searches for a file named `mymodule.py` in a list of directories given by the variable sys.path, which contains these locations:\n",
    "> * The directory containing the input script (or the current directory when no file is specified).\n",
    "> * The `PYTHONPATH` environment variable (a list of directory names, with the same syntax as the shell variable PATH).\n",
    "> * The default search paths for your python installation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will often want to install a module (or library) that you've found online, with some functionality that you want to use.\n",
    "\n",
    "There are a few different ways to do it, depending on the situation (these are terminal commands, not in a notebook):\n",
    "\n",
    "1. From source code, i.e. found in a github repository:\n",
    "\n",
    "> git clone https://www.github.com/username/module_name\n",
    ">\n",
    "> cd module_name\n",
    ">\n",
    "> python setup.py --install --user\n",
    "\n",
    "2. From the [pypi](https://pypi.org/) package index (most common):\n",
    "\n",
    "> **pip install --user module_name**\n",
    "\n",
    "3. If you are using an [Anaconda](https://www.anaconda.com/products/individual) installation of python:\n",
    "\n",
    "> conda install module_name\n",
    "\n",
    "Note: the `--user` option is asking that the package be installed locally, in your home directory, rather than \"system-wide\", which is impossible on any shared computer/cluster.\n",
    "\n",
    "(You could install packages system-wide on your personal computer, but this is bad practice).\n",
    "\n",
    "If you don't have `pip` and need to install it, or want to learn about Anaconda `environments`, open:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "## If you need to install pip:\n",
    "\n",
    "Most typical python installations, on clusters or your own system, will have the `pip` command available. This is a package manager, i.e. it installs, upgrades, uninstalls, and keeps track of libraries on the [pypi index](https://pypi.org/).\n",
    "\n",
    "On the KIP JupyterLab server, `pip` is not available by default. We can install it:\n",
    "\n",
    "1. Open a terminal (File -> New -> Terminal).\n",
    "2. Type `wget https://bootstrap.pypa.io/get-pip.py` (download the install script).\n",
    "3. Type `python get-pip.py`.\n",
    "\n",
    "You will notice a message about the pip executable not being on the current PATH.\n",
    "\n",
    "1. Type `echo 'export PATH=/cipuser/zah/wu533/.local/bin:$PATH' >> .bash_profile` (**change the path /zah/wu533/** to your username. this adds this path to the PATH environment variable, telling the OS where to search for executables).\n",
    "2. Type `source .bash_profile` (to execute this command immediately, it will automatically be done in the future when you log in).\n",
    "\n",
    "After this, we could do:\n",
    "\n",
    "    pip install --user memory_profiler\n",
    "    \n",
    "(we will use this package on day 5)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "## Conda environments, installing packages with `conda`\n",
    "\n",
    "It is fairly common these days to use **Anaconda** to install a working python environment. Anaconda combines at least two features: independent \"environments\", and a nice package manager (like pip).\n",
    "\n",
    "The first step, done only once, is to create an empty environment (optionally, with a specific version):\n",
    "\n",
    "```python\n",
    "    conda create --prefix=~/.local/envs/myenv python=3.9\n",
    "```\n",
    "\n",
    "Then you \"activate\" the environment (usually, by adding this line to your `.bashrc` file, so that it is always active when you start):\n",
    "\n",
    "```bash\n",
    "    source activate ~/.local/envs/myenv\n",
    "```\n",
    "\n",
    "Then, any packages you install, and e.g. their specific versions, are contained within this environment:\n",
    "\n",
    "```bash\n",
    "    conda install package_name\n",
    "```\n",
    "\n",
    "This is nice, for example, if you are working on more than one project, but different projects require different libraries, or conflicting versions of libraries -- you can keep two separate environments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [3] The \"standard library\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every python installation comes with a number of [standard library](https://docs.python.org/3/library/index.html) modules, many of which are essential to use. Let's explore some of the most important:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Files, directories, and filesystem interaction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `os` module provides dozens of functions for interacting with the operating system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.path.isfile('test.hdf5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.rename('test.hdf5','test_new.hdf5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `glob` module can search for files or directories, also using wildcards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = glob.glob('*.ipynb')\n",
    "for file in files:\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dates and times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `datetime` module supplies classes for manipulating dates and times, including handling timezones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "now = datetime.date.today()\n",
    "print(now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "now.strftime(\"%m-%d-%y. %d %b %Y is a %A on the %d day of %B.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `time` module is useful for recording how long it takes for a piece of code to run (i.e. for performance benchmarking)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "for i in range(100000):\n",
    "    x = [j**2 for j in range(10)]\n",
    "    \n",
    "print('Execution took %.2f seconds.' % (time.time()-start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mathematics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: the standard math/random/stats libraries are rarely used in practice, since numpy (tomorrow!) is better.\n",
    "\n",
    "The `math` module gives access to the underlying C library functions for floating point math:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "math.cos(math.pi / 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "math.log10(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `random` module provides tools for making random selections:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.choice(['apple', 'pear', 'banana'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.random() # float between 0.0 and 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `statistics` module calculates basic statistical properties (the mean, median, variance, etc.) of numeric data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [2.75, 1.75, 1.25, 0.25, 0.5, 1.25, 3.5]\n",
    "statistics.median(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Does the `math.cos` funtion take radians or degrees? Are there functions that can convert between radians and degrees? Use these to find the cosine of 60 degrees, and the sine of pi/6 radians.\n",
    "\n",
    "> Hint: in a Jupyter notebook, or in an IPython console, type `math.` and then hit the `TAB` key. This will show you a list of definitions (i.e. functions) within the math module.\n",
    ">\n",
    "> Similarly, you can also type `?math` to print out some documenation for a module, or `?math.cos` to print out documenation for a specific function.\n",
    ">\n",
    "> Beyond this, you should find and consult the online documentation of whatever module or library you are using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regular expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Regular expressions](https://en.wikipedia.org/wiki/Regular_expression) (or \"regex\") are very powerful, and complicated, tools for searching and manipulating text. If you do extensive processing of complex text, they will be essential.\n",
    "\n",
    "The `re` module provides regular expression tools for advanced string processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regex = r'\\bf[a-z]*' # we want to match: any sequence of characters which \"is a word\" (starts with a word boundary, e.g. space, \"\\b\"), then the letter \"f\", then any number of letters between a-z\n",
    "string_to_search = 'which foot or hand fell fastest'\n",
    "\n",
    "re.findall(regex, string_to_search)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Write your email address as a string variable. Then, separate the username and domain name into two separate strings (without re). Do it again using re."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other standard libraries\n",
    "\n",
    "Many others exist (see [documentation](https://docs.python.org/3/library/index.html)), you will generally discover them when googling with a specific problem or question. They fall into the categories:\n",
    "\n",
    "* Text processing\n",
    "* Binary data (including **struct** for reading/writing raw structured binary data)\n",
    "* Data types (including **OrderedDict**)\n",
    "* Numeric and math\n",
    "* Functional programming (other ways of looping, making functions)\n",
    "* File, directory, and OS access (including **subprocess** to run commands)\n",
    "* Data persistence (saving/loading/databases)\n",
    "* Compression (zip)\n",
    "* File loading (CSV, **configparser**)\n",
    "* Encryption and hashing\n",
    "* Concurrent execution (parallel programming, including **threading** and **multiprocessing** - Friday!)\n",
    "* Networking and internet communication (low-level)\n",
    "* GUI tools\n",
    "* Documentation and (unit) testing tools\n",
    "* Debugging, profiling, and packaging tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid #bbb; margin: 30px 0\"> </hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Day 2 Practice Problem - Average Temperatures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Use the [data/day2_munich_temps.txt](data/day2_munich_temps.txt) data file, which gives the temperature in Munich every day (one line each), for several years. \n",
    "\n",
    "## Task A\n",
    "\n",
    "Read in the data, and print out the minimum, average, maximum temperature for each year, e.g:\n",
    "\n",
    "    1995: -3C  10C  35C\n",
    "    1996: ...\n",
    "    \n",
    "Hint: you could use a dictionary to help store values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task B\n",
    "\n",
    "For the year 2000, print out the same three statistics, averaging over each of the twelve months, e.g:\n",
    "\n",
    "    January: -15C 0C 3C\n",
    "    February: ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task C\n",
    "\n",
    "For each month, print out the same three statistics, but averaging over all of the years available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid #bbb; margin: 30px 0\"> </hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Day 2 Challenge Problem - IllustrisTNG API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IllustrisTNG project is a suite of \"cosmological\" galaxy formation simulations. Each simulation in IllustrisTNG evolves a large swath of a mock Universe from soon after the Big-Bang until the present day while taking into account a wide range of physical processes that drive galaxy formation. The simulations can be used to study a broad range of topics surrounding how the Universe — and the galaxies within it — evolved over time. \n",
    "\n",
    "Step 1. Sign up for a [public data access](https://www.tng-project.org/data/) account.\n",
    "\n",
    "Step 2. Follow the [Data Access API tutorial](https://www.tng-project.org/data/docs/api/) to get familiar with accessing such scientific data using a web-based API.\n",
    "\n",
    "## Task A\n",
    "\n",
    "Use the API to search the TNG100-1 simulation at snapshot 99 (redshift zero) for all galaxies with total mass between $10^{12.0} M_\\odot$ and $10^{12.2} M_\\odot$ (see Task \\#2 on the API webpage for hints).\n",
    "\n",
    "For each, compute and print the gas fraction, defined as the ratio of gas mass to dark matter mass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task B\n",
    "\n",
    "Download a \"particle cutout\" of one of the galaxies you have found from the previous task. Open the resulting HDF5 file with `h5py` and examine its contents. Print the number of particles of each type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid #bbb; margin: 30px 0\"> </hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Day 2 Challenge Problem - Bitcoin Price API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Use the public Coinbase API to download the current exchange rate (i.e. price) for Bitcoin, in different currencies.\n",
    "\n",
    "The URL:\n",
    "\n",
    "    https://api.coinbase.com/v2/exchange-rates?currency=BTC\n",
    "    \n",
    "provides a JSON response.\n",
    "\n",
    "## Task A\n",
    "\n",
    "Create a loop to query this API twenty or thirty times. Between each query, pause the program for three seconds (use `sleep(3.0)`).\n",
    "\n",
    "For each query, parse the JSON response to obtain the current BTC price in EUR, and save it into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task B\n",
    "\n",
    "We want to test a strategy for trading BTC:\n",
    "* At the time corresponding to the first data point you have from above, assume you start with \"wallet\" of 1000 euros.\n",
    "* Walk forward in time, through the price data series. At each step:\n",
    "  * if the current price is greater than a factor `(1+frac)` times the previous price, then we take this as a sign the value is increasing, and we \"buy\". Use your entire wallet balance, if it is in EUR, to buy BTC, i.e. use the current price to convert from EUR to BTC.\n",
    "  * if the current price is less than a factor `(1+frac)` times the previous price, take this as a sign the value is decreasing, and \"sell\". Use your entire wallet balance, if it is in BTC, to sell, i.e. use the current price to convert from BTC to EUR.\n",
    "  \n",
    "Choose a reasonable value for `frac`.\n",
    "  \n",
    "At the end of the time period (when you are out of data), convert your wallet balance into EUR using the final price, if it isn't already. Have you made or lost money?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your solution here\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}