tu013-Numpy-Advanced-GeneratingData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a38d572b",
   "metadata": {},
   "source": [
    "## Advanced `numpy`: Understanding and creating data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30775168",
   "metadata": {},
   "source": [
    "### Learning goals:\n",
    " \n",
    " - Understand the basic distribution of individual variables\n",
    " - [Data scraping](https://en.wikipedia.org/wiki/Web_scraping).\n",
    " - How to organize data (or [data wrangling](https://en.wikipedia.org/wiki/Data_wrangling))\n",
    " - Use existing data to predict future data (data simulation)\n",
    " - Generate correlated datasets from existing data parameters.\n",
    " - Advanced understanding of indexing into `numpy` `arrays`\n",
    " - Practice with loops and advanced plots using `matplotlib`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51a8a963",
   "metadata": {},
   "source": [
    "In previous tutorials, we have mostly focussed on operations using small `lists` or `arrays`, with only a few dozens of observations. In general, when handling data in the real world, arrays and lists are larger (hundreds to thousands of observations).\n",
    "\n",
    "In this tutorial we will learn how to think about data and how to use `numpy` to create data that respects some properties of an underlying data set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb536d3",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d921de",
   "metadata": {},
   "source": [
    "First thing first, we will import the basic libraries we need. \n",
    "\n",
    "*Note that we will import also `pandas` even though we have not really learned much abotu this library yet. This is because we will used some of its functionality to copy data from online.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a95ca534",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7a47cca",
   "metadata": {},
   "source": [
    "### Simulating the height of USA presidents\n",
    "\n",
    "Imagine being asked to guess the height of the next president of the USA. How would you go about it if you were a data scientists? Can we use existing data to make an educated guess?\n",
    "\n",
    "Wikipedia offers a table with the records of the height of all presidents. You can find the [table here](https://en.wikipedia.org/wiki/Heights_of_presidents_and_presidential_candidates_of_the_United_States). \n",
    "\n",
    "The tallest president elected to office (Wikepedia reports) was Abraham Lincoln with a height of `6 ft 3+3⁄4 in` (or `192.4 cm`). The shortest president elected to office was James Madison with a height of `5 ft 4 in` or `163 cm`).\n",
    "\n",
    "Our goal is to try to generate a sensible (believable) height of future presidets. How can we do that? How can we predict the height of future presidents? \n",
    "\n",
    "One approach could be to use the past to predict the future. So, to do that, we could use the height of the previous presidents as an educated guess for the height of the future presidents (this is not perfect ebcause there might be changes over time to the average height of human populations, see [this article for example](https://en.wikipedia.org/wiki/Human_height)). \n",
    "\n",
    "Although not perfect, we can use the distribution of height of the past presidents to make a prediction of the future president's height. Let' take a dive on how we can do this."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eddc8645",
   "metadata": {},
   "source": [
    "### Web scraping "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b264d82",
   "metadata": {},
   "source": [
    "First of all, we will want to capture some data from the web, this is called [Web scraping](https://en.wikipedia.org/wiki/Web_scraping).\n",
    "\n",
    "Our web scraping will be limited, we will want to copy the data from the [table in this Wikipedia article](https://en.wikipedia.org/wiki/Heights_of_presidents_and_presidential_candidates_of_the_United_States). \n",
    "\n",
    "To do so, we can use `pandas` `read_clipboard` to read the entire table of data into a pandas dataframe. \n",
    "\n",
    "Please select and copy to clipboard the table found on [this wikipedia article](https://en.wikipedia.org/wiki/Heights_of_presidents_and_presidential_candidates_of_the_United_States) You might remember that we have used `read_clipboard()` in a previous tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc2d951a",
   "metadata": {},
   "outputs": [],
   "source": [
    "presidents_heights = pd.read_clipboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6739651a",
   "metadata": {},
   "source": [
    "OK, if no errors were displayed, somehting should have happened. If errors were displayed, please make sure to have selected the table and only the table from the Wikipedia article. \n",
    "\n",
    "If errors still persist try first copying the table in a spreadsheet software (Excell or Google Spreadsheet) and then copy the data agin from the streadsheet and try the above line for one more time, hoefully that will work. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83ac341b",
   "metadata": {},
   "source": [
    "Next, let's take a look at the top and bottom 5 rows of the `pandas` `data frame` the copy operation should have created. We can use the methods `.head()` and `.tail()` respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46bf75a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "presidents_heights.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc3be930",
   "metadata": {},
   "outputs": [],
   "source": [
    "presidents_heights.tail() # bottom 5 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4d1482e",
   "metadata": {},
   "source": [
    "Our main goal is to study and use the distribution of heights of the past presidents, and se that distribution as a model for the future presidents.\n",
    "\n",
    "For example, the most basic thing we can do is to use the mean of the previosu rpesidents as a predictor of the likely height of the enxt president. That seems sensible, doesn't it?\n",
    "\n",
    "Before we do that we would like to plot the data, to plot the distribution of heights, and that plot would be something like histogram plot. `pyplot` has a method called [`hist`](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.hist.html), we can use that, but as it turns out the data we will need to be prepared before we can use that. So let's do that first, get the data we need, and only the data we need out of the full table. \n",
    "\n",
    "This is techincally called, [data wrangling](https://en.wikipedia.org/wiki/Data_wrangling)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ddaca28",
   "metadata": {},
   "source": [
    "### Data wrangling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d1e7df9",
   "metadata": {},
   "source": [
    "We have gotten some data, but it is not in the format we need it. We need to find a way to change the data so that it can be suable. A few things we will want to do to the data before it can be used for numerical operations:\n",
    "\n",
    "  - Extract the data from the full table\n",
    "  - Make sure the data extracted is in a usable format for numerical operations (say an `int` or a `float`)\n",
    "\n",
    "We will work with the metric system so we will want to extract the height from the columncontaining the height in cm, this columne can be addressed in `pandas` as `Height (cm)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9834c8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "height_cm = presidents_heights[\"Height (cm)\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "304206b2",
   "metadata": {},
   "source": [
    "And we can check that the operation worked out well, for example by looking at the first few elements of the new dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ac22a85",
   "metadata": {},
   "outputs": [],
   "source": [
    "height_cm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13c6dfbf",
   "metadata": {},
   "source": [
    "Great, now that we have extracted the data we want, the next thing we will want to do, so as to be able to handle the data and be confident about it, is to dump the `pandas dataframe` column we just extracted. \n",
    "\n",
    "We can do this by using the handy `pandas` method `to_numpy()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99c0424b",
   "metadata": {},
   "outputs": [],
   "source": [
    "presidents_height_str = height_cm.to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66b32543",
   "metadata": {},
   "source": [
    "Exfellent, `pandas` seems quite powerful doesn't  it?\n",
    "\n",
    "Yet, we are still not ready to do what we want; compute the mean and the standard deviation of the distribution of heights of the presidents of the united states. \n",
    "\n",
    "This is because the data given to us are strings (`str`) and have a trailing series of character that are not numeric and disadvantageous for numerical operations: ` cm`.\n",
    "\n",
    "Take a look at the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf53b3a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "presidents_height_str"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf15ed89",
   "metadata": {},
   "source": [
    " We need to find a way to remove the units (` cm` the trailing characters) and after that change the format of that data to a numeric one, for example to `int` (`float` would also work for numerical operations). \n",
    " \n",
    "As it turns out, we can remove the trailing characters using `numpy`'s [`rstrip(a[, chars])`](https://numpy.org/doc/stable/reference/generated/numpy.char.rstrip.html#numpy.char.rstrip).\n",
    " \n",
    " This will be a slightly complicate series of operations that, yet, will use all operations we have used before. Let's dig into it:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8848285",
   "metadata": {},
   "source": [
    "First, let's compute the number of presidents (the `len` of the `numpy` array), this number will be helpful to initialize arrays and use `for` loops: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d37392ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "numPresidents = len(presidents_height_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63d40841",
   "metadata": {},
   "source": [
    "Next, let's create an `numpy array` filled with `0`'s, to use to store the new numerical values of the height of the presidents. We will make the array filled with `int` and we will want a 1-D array of `len` equal the number of presidents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09caf840",
   "metadata": {},
   "outputs": [],
   "source": [
    "presidents_height_int = np.zeros((numPresidents,), dtype=int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d9dcd2d",
   "metadata": {},
   "source": [
    "The next things we need to do are the most challanging ones. We will need to remove the trailing characters and change the data type to `int`. We will do this using a `for` loop. \n",
    "\n",
    "Even though the next oeprations are a bit complicated, we will be using only operations that we have encountered before, and a single neat new method that `numpy` offers: `char.strip`. The method will allow us to strips away the last characters from the list. \n",
    "\n",
    "So let's do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27a92998",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(0,numPresidents) : # we loop over from 0 to the number of presidents\n",
    "    temp = np.char.strip(presidents_height_str[i], ' cm') # we strip away *space+cm*\n",
    "    presidents_height_int[i] = int(temp) # we change the format from char to it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0657f0ee",
   "metadata": {},
   "source": [
    "Excellent, if all worked out well above, we can test now the type of the output variable, it should be `int`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "527cbee9",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Check that the type of the output array from the above operations is an `int` as expected. Use the followign cell to return the result.\n",
    "  \n",
    "*Note that I am nto explicitly telling you the name of the variable I would like you to test, because I am itnerested in checking that you understand which one is the output variable and the end result of all the operations above (sneaky prof).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8c28384",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "47c36dab",
   "metadata": {},
   "source": [
    "\n",
    "  - Repeat the same operations above, using the `for` loop, but change the data type from `int` to `float`. Call the output variable `presidents_height_float`\n",
    "  \n",
    "Use the cell below to show your work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "426bff1d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a0b45015",
   "metadata": {},
   "source": [
    "### Estimating key parameters from existing data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6809ef02",
   "metadata": {},
   "source": [
    "Once the data has been mapped to an appropriate format for numerical operations (`int` in our case), we can start exploring the data and estimating key parameters that we can use for our task.\n",
    "\n",
    "To explore the data we can use a plot. A histogram of the data would give us a nice idea of the distribution of the data. We have imported `pyplot` which offers `hist()`, let's use that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dccd689",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(presidents_height_int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16f02493",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Edit the plot about adding a title, and labels for the y and x-axis\n",
    "  \n",
    "  Use the cell below to plot the the histogram again with the requested edits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7093f105",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0aed6f85",
   "metadata": {},
   "source": [
    "The distribution seems well-behaved, normal, or better normally distributed. When the histogram of a dataset is close enough to a normal distribution we can use the mean and standard deviation of the data to estimate the central tendency of the data and the spread around that central tendency.\n",
    "\n",
    "Let's do that next, let' estimate the mean height of the past presidents and the variability around that mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d3c915b",
   "metadata": {},
   "outputs": [],
   "source": [
    "height_mu = np.mean(presidents_height_int)\n",
    "height_sd = np.std(presidents_height_int)\n",
    "\n",
    "print('The mean height of the presidents of the USA as of today is', height_mu, 'cm')\n",
    "print('The average variability around that mean has a standard deviation of', height_sd, 'cm')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd24976",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Are the mean and STD of the height of the presidents `int`? \n",
    "  - Show code below demonstrating how to test the type\n",
    "  \n",
    "  Use the cell below to show your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f04b136f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0421894f",
   "metadata": {},
   "source": [
    "  If the type is not `int` explaing why that is the case using the cell below:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3135a8fd",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "aa656a26",
   "metadata": {},
   "source": [
    "### Predicting the height of future presidents "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e60604e",
   "metadata": {},
   "source": [
    "Now that we have the standard mean and standard deviation of the height fo past presidents, we are in position to make a prediction of the height of future presidents.\n",
    "\n",
    "Under the assumption that no change int he height of males in the USA happens over a couple of hundreds of years (this is likely an unfair assumption, but fine enough for our experiment), we can estimate the mean height of the future president by generating random data centered at the height of the ast presidents with the same variability of the distribution of the meausred height.\n",
    "\n",
    "This really just means that the most likely president in the future is very likely to have a height of 180 cm. But that some variability can happen around that value.\n",
    "\n",
    "We can make a nmerical guess. We can guess that then ext president will have a height of 1800, plus some random factor that will make that height variate from the mean as controlled by the standard deviation.\n",
    "\n",
    "In other words we can say that the future president comes from the same distribution of previous presidents plus some randomness."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6cca0e0",
   "metadata": {},
   "source": [
    "The above can be implemented inpython using `random` anf `rand()`, the random generator that generates normally distributed data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d48eea5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "future_president_height = height_mu + height_sd*np.random.randn()\n",
    "\n",
    "print('Our educated guess for the height of the next president given the height of the past presidents is:', future_president_height)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c46dd2fc",
   "metadata": {},
   "source": [
    "Every time we execute the previosu cell we get a different prediction. The average prediction should be 180 cm, because that is the mean and we are setting that mean to be 180 by adding `height_mu` to the numbers generated by `np.random.randn`. Also, we are using the variabilty int he past data `height_sd` to make the data generate variate as if the new height were to be coming from a distribution with the same spread fo the past distribution. We are setting the spread of the distirbution by multiplying the numbers outputted by  `np.random.randn` by the `height_sd`.\n",
    "\n",
    "In other words, adding a number to the random numer will shift the center (the value of the randome number) and set the mean value. By multiplying the random value we will change the spread or variability of the random value.\n",
    "\n",
    "Well, we can continue talking about this or we can use code to test what we are say."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a9ed3d1",
   "metadata": {},
   "source": [
    "What we had done before is to create a single number, we used that number as *educated* guess of the height of the next president, given the height of the past presidents. But we can repeate the experiemnt multiple times and look at the resutl:=.\n",
    "\n",
    "For example, we can simulate 10 presidents instead of only one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ebc4617",
   "metadata": {},
   "outputs": [],
   "source": [
    "future_president_height = height_mu + height_sd*np.random.randn(5,)\n",
    "print(future_president_height)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e411b12",
   "metadata": {},
   "source": [
    "Great, it worked. We can see above that there is quite some variability in the estimates, but the average shoudl be close to 180 cm. Let's measure that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a60cfe0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.mean(future_president_height))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "174cb1b9",
   "metadata": {},
   "source": [
    "Pretty close, and what about the standard deviation? It should be close to what we set it to about 7 cm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6172a8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.std(future_president_height))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c0b1615",
   "metadata": {},
   "source": [
    "Not we can repeate the experiments above and appreciate what happens when we try not with 1, not with 5, but with 100 or 1000 guesses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "414ec8f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "pres_height_100 = height_mu + height_sd*np.random.randn(100,)\n",
    "print('The mean is',np.mean(pres_height_100))\n",
    "print('The STD is',np.std(pres_height_100))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af466ef",
   "metadata": {},
   "source": [
    "The estimates now are much closer to the numbers we expected; 180 and 7 cm. What about if we try with 1000 guesses or even better 10,000?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e79321d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "pres_height_10000 = height_mu + height_sd*np.random.randn(10000,)\n",
    "print('The mean is',np.mean(pres_height_10000))\n",
    "print('The STD is',np.std(pres_height_10000))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3b016d3",
   "metadata": {},
   "source": [
    "Both the mean and the standard deviation are much closer to what we set them to be 180 and 7 cm. This is because we are computing multiple guesses for the height of the future president and we are then averaging over many guesses, given that the guesses are set to have mean of 180 and a standard deviation fo 7 cm, if we use enough guesses we are doomed to get back what we set the parameters to be.\n",
    "\n",
    "We can make a nice plot of the distrobution and the distribution should look pretty normal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "378681a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = plt.hist(pres_height_10000, 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2944c8fa",
   "metadata": {},
   "source": [
    "The above is a simple example of how we can use data, to generate a data-driven guess, and in doing so, we are effectively encountering a first case of data simulation.\n",
    "\n",
    "We simulated the height of the future president of the USA, given the past data that we collected from online. Pretty cool, data sciency stuff.\n",
    "\n",
    "Note, that the use of `random` and `rand()` is not just cool, is also pretty deep. At this point we are not going to dig too much into the how and the why that operation works. But we might do more of this in the future and hoefully you have encoutnered similar operations in the past, because you will most likely encounter them in the future if you continue working in data science."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60a4fef5",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Simulate the data of the wives of the 200 future USA presidents given that the mean height of the wives in the past has been 163 cm with a standard deviation of 6 cm\n",
    "\n",
    "Use the cell below to show the code. Also plot the historgram of your data and add a title, and labels to the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "269496cb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "625efa06",
   "metadata": {},
   "source": [
    "### Generating correlated datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae26e00",
   "metadata": {},
   "source": [
    "After learning how to create a single datasets based on some simple assumptions on the distribution underlying our process we will next learn how to create two correlated datasets. Think about these datasets as the height of the presidents and their wives. \n",
    "\n",
    "We will run under the assumption I know, I am sorry...) that there is some weird tendency of people with correlated height to marry, if you are tall, you are more likely to marry a tall person, and if you are short you are more likely to marry a shorter person (no statement here, just trying to do some data science).\n",
    "\n",
    "More specifically, we will create a dataset called `x` (the presidents). Each dataset will have the length of `m` (where for example, `m` could be 100 or 1000), this means that, for example, each dataset will have the shape of (m,1) or in our example (1000,1).\n",
    "\n",
    "After that, we will create another dataset called `y` (the wives) of the same shape of `x` (one wife per president). Each one of the `y` dataset data points will have a corresponding `x` datapoint, that it will be correlated with.\n",
    "\n",
    "Let's get started with a hands on method. First we will make the example of a single dataset `x` and a correlated dataset `y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90569e91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The USA Presidents\n",
    "\n",
    "# We first build the dataset `x` \n",
    "# we will use our standard method\n",
    "# based on randn\n",
    "m  = 1000\n",
    "mu = 180\n",
    "sd = 7\n",
    "x  = mu + sd*np.random.randn(m,1)\n",
    "\n",
    "# let take a look at it\n",
    "a = plt.hist(x, 60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afee412a",
   "metadata": {},
   "source": [
    "OK. After generating the first dataset we will generate a second dataset, let's call it `y`. This second dataset will be correlated to the first.\n",
    "\n",
    "To generate a dataset correlated to `x` we will indeed use `x` as our base for the data and add on top of `x` a small amount of noise, let's call it `noise`. `noise` represents the small (or larger) difference between `x` and `y`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78fe035c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The First ladies, USA Presidents\n",
    "\n",
    "err = np.random.randn(m,1)\n",
    "y = (x + err) - 10 # Let's assume the wives are 10 cm shorter than the presidents\n",
    "a = plt.hist(y,60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1035d734",
   "metadata": {},
   "source": [
    "OK. The two histograms seem similar (similar range and height), but it is difficult to judge if `x` and `y` are indeed correlated. To do that we need to make a scatter plot.\n",
    "\n",
    "`matplotlib` has a convenient function for scatter plots, `plt.scatter()`, we will use that function to take a look at whether the two datasets are correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1c3dc64",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1aac1a9",
   "metadata": {},
   "source": [
    "Great, the symbols should be aligned along the major diagonal. This means that they are indeed correlated. To get to understand more what we did above, let's think about `err`.\n",
    "\n",
    "Imagine, if there were no error, e.g., no `err`. That would mean that there would be no difference between `x` and `y`. Literally, the two datasets would be identical.\n",
    "\n",
    "We can do that with the code above by setting `err` to `0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04cbe9a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "err = 0\n",
    "y = x + err - 10\n",
    "plt.scatter(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "932919e2",
   "metadata": {},
   "source": [
    "The symbols should all lay on the major diagonal. So, `err` effectively controls the level of correlation between `x` and `y`. So if we set it to something small, in other words if we add only a small amount of error then the two arrays (`x` and `y`) would be very similar. For example, let's try setting it up to 10% of the original `err`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "553ce0f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "err = np.random.randn(m,1);\n",
    "err = err*0.5 # 0.5 -> scaling factor for the noise, the smaller this factor the lesser the noise\n",
    "y = x + err - 10\n",
    "plt.scatter(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e1c6e9b",
   "metadata": {},
   "source": [
    "OK. It should have worked. The error added is not large, the symbols should lay almost on the diagonal, but not quite.\n",
    "\n",
    "As we increase the `err` the symbols should move away from the diagonal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ffc82ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "err = np.random.randn(m,1);\n",
    "scaling_factor = 0.99\n",
    "err = err*scaling_factor \n",
    "y = x + err - 10 \n",
    "plt.scatter(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c1c2e2c",
   "metadata": {},
   "source": [
    "One way to think about the scaling factor and `err` is that they are related to correlation. Indeed, they are not directly related to correlation (not a one-to-one relationship, but a proxy). \n",
    "\n",
    "The scaling factor is inversely related to correlation because as the scaling factor increases the correlation decreases. Furthermore, they are not directly related to correlation because they both depend on a couple of variables, for example, the variance of the distributions (both `err` and `x` will affect the relationship between the correlation and the scaling factor)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d685b13",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Add a title and labels to the scatter plot above. Use the cell below to make the new plot with the new attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f450704",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "56d49cd6",
   "metadata": {},
   "source": [
    "Python has a method to generate couples of correlated arrays. We will now briefly explore it, but leave a deeper dive on each function to you. You are suggested to further explore the code below and its implications. It might come helpful to us later down the road, you never know!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c7d7803",
   "metadata": {},
   "source": [
    "### A more principled way to make correlated datasets\n",
    "\n",
    "NumPy has a function called `multivariate_normal` that generates pairs of correlated datasets. The correlation values can be specified conveniently. A little bit of thinking is required, though. The function uses the covariance matrix. The covariance matrix is composed of 4 numbers. Two of the numbers describe the variances of the two datasamples we want to generate. The other two values describe the correlation between the samples and are generally called `covariances` (co-variations or co-relations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfa555a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from  numpy.random import multivariate_normal # we import the function\n",
    "x_mu = 180; # we set up the mean of the first set of data points \n",
    "y_mu = 170; # we set up the mean of the second sample\n",
    "x_var = 2; # the variance of the first sample\n",
    "y_var = 2; # the variance of the second sample\n",
    "cov = 0.9; # this is the covariance (can be thought of as correlation)\n",
    "\n",
    "# the function multivariate_normal will need a matrix to control\n",
    "# the relation between the samples, this matrix is called covariance matrix\n",
    "cov_m = [[x_var, cov],\n",
    "         [cov, y_var]]\n",
    "\n",
    "# we now create the two data sets by setting the the proper\n",
    "# means and passing the covariance matrix, we also pass the\n",
    "# requested size of the sample\n",
    "data = multivariate_normal([x_mu, y_mu], cov_m, size=1000)\n",
    "\n",
    "# We can plot the two data sets\n",
    "x, y = data[:,0], data[:,1]\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7d49313",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Simulate two datasets of the walking stride of mothers and 10-years old daugthers. \n",
    "  - We will make a few assumptions. We will assume that:\n",
    "    - the walking stride of the mothers is on average 80 cm with a standard deviation of 2 cm. \n",
    "    - that the daugthers' height at 10 years of age is correlated to the mother's height and it is about 70% of that of the mothers (70% of 80) and with a standard deviation also proportional to that of the mothers (70%)\n",
    "    \n",
    "Reuse the code above but isntert the new parameters suggested here to make a simulation of 1000 mothers and daugthers. Make a correlation plot and add titles and labels to the axis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04fa66e9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "13ec207a",
   "metadata": {},
   "source": [
    "### Creating multiple correlated datasets\n",
    "\n",
    "Imagine now if we were asked to create a series of correlated datasets. Not one, nottwo, more than that.\n",
    "\n",
    "Once the basic code used to build one is known. The rest of the datasets can be generated reusing the same code and putting the code inside a loop. Below we will show how to create 5 datasets using a `while` loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c20772f",
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = 0;\n",
    "n_datasets = 5;\n",
    "siz_datasets = 1000;\n",
    "\n",
    "x_mu = 180;  # mean height of the USA presidents \n",
    "y_mu = 170;  # mean height of the first ladies\n",
    "x_var = 1.5; # the variance of the first dataset\n",
    "y_var = 1.5; # the variance of the second dataset\n",
    "cov = 0.85; # this is the covariance (can be thought of as correlation)\n",
    "\n",
    "# covariance matrix\n",
    "cov_m = [[x_var, cov],\n",
    "         [cov, y_var]]\n",
    "\n",
    "while counter < n_datasets :\n",
    "    data = multivariate_normal([x_mu, y_mu], \n",
    "                               cov_m, \n",
    "                               size=siz_datasets)\n",
    "    x, y = data[:,0], data[:,1]\n",
    "    counter = counter + 1\n",
    "\n",
    "    # Make a plot, show it, wait some time\n",
    "    print(\"Plotting dataset: \", counter)\n",
    "    plt.scatter(x, y);\n",
    "    plt.show() ;\n",
    "    plt.pause(0.05)\n",
    "\n",
    "else:\n",
    "    print(\"DONE Plotting datasets!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16aee6b8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bec019ad",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Use subplot to organize the plots made using the code above. In other words, repeate the plotting made above but organize the plots using subplot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b862ee4b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}