tu011-Numpy-Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "420658c2",
   "metadata": {},
   "source": [
    "# `NumPy`: Numerical operations in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "741f0ffe",
   "metadata": {},
   "source": [
    "This tutorial uses material also found in the [SciPy 1.0 Nature Methods](https://www.nature.com/articles/s41592-019-0686-2) and [NumPy Array IEEE](10.1109/MCSE.2011.37) article and at [https://numpy.org/](https://numpy.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be6f2c2e",
   "metadata": {},
   "source": [
    "### Learning outcomes:\n",
    " - Understand Python Library `NumPy`\n",
    " - Use `numpy arrays`\n",
    " - Apply loops and logical operators on `numpy arrays`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff32a2c3",
   "metadata": {},
   "source": [
    "Data scientists spend a lot of time – wait for it – working with ***data***! To work with **data** it is critical to organize the data in a way that facilitate the work on the potential analyses we might need to do. So organizing data means guessing what type of work we will want to do with the dataset. And, odd is it may seem, good guessing requires some practice. The data organization process will require: \n",
    "\n",
    "* store the data a clear and systematic way\n",
    "* provide methods to access the data that are simple and straightforward\n",
    "* be flexible enough so to and allow to modify the format of the data for various needs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "014aeb3f",
   "metadata": {},
   "source": [
    "NumPy is the fundamental `library` for mathematical operations and computations in Python. \n",
    "\n",
    "The NumPy library provides a series of objects that can be used to store data (represent the numbers in a dataset for Python) and a series of methods to analyze data (provide operations that can compute, say the mean or the standard deviation of some numbers in a dataset).\n",
    "\n",
    "The NumPy `array` is the workhorse of NumPy and we will spend quite some time with it below. The numpy array is a multidimensional array object, that means that it \n",
    "  - (1) is an object for programming (so it has methods and proprties of an object, just like you cup holds coffee and has colors, \n",
    "  - (2) the Numpy array can hold data and have a bunch of properties that we can look as we look at the colors on the cup. The properties of the array help us recognize and understand what is in the array and sometimes that is really important, for example, you would not want to drink from someone else's cup, would you?\n",
    "\n",
    "A variety of fast numerical operations on arrays are provided by NumPy. These include operations that are mathematical, logical, shape manipulation, sorting, selecting, I/O, discrete Fourier transforms, basic linear algebra, basic statistical operations, random simulation and much more. \n",
    "\n",
    "NumPy is the base of scientific computing and data science libraries such as [Pandas](https://pandas.pydata.org/) [scipy.org](https://scipy.org/), and [scikit-learn.org](https://scikit-learn.org/) among many others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f75b21d0",
   "metadata": {},
   "source": [
    "In other (simpler?) words, Numpy arrays are grids or tables for holding, accessing, and manipulating data. They are created and accessed in ways that are very similar to the ways Python `lists` can be accessed.\n",
    "\n",
    "So what we are going to do is to recall how `lists` work (lists are handy!), then we will graduate to `numpy arrays` and see what they can do. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2be116d5",
   "metadata": {},
   "source": [
    "### Python lists"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0099343",
   "metadata": {},
   "source": [
    "We have covered Python `lists` (and other datatypes) in previous tutorials. Python `lists` (a list of things) is build by collecting, ahem, a list of things using `[square brackets]`.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42710c7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist = ['this', 3, 'list', 4+2j, 6.66]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a97c258",
   "metadata": {},
   "source": [
    "We can address elements in a list by using indices and the `:` (colon) operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a06fd7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09ae6a64",
   "metadata": {},
   "source": [
    "We can read this as \"Give me all the elements in the interval between 0 **inclusive** to 3 **exclusive**.\"\n",
    "\n",
    "I know this is weird. But at least for any two indexes `a` and `b`, the number of elements you get back from `mylist[a,b]` is always equal to `b` minus `a`, so I guess that's good!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97c3441e",
   "metadata": {},
   "source": [
    "We can get any consecutive hunk of elements using `:`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24a6dbc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist[2:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "432eb54a",
   "metadata": {},
   "source": [
    "If you omit the indexes, Python will assume you want everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f32671ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "mylist[:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10b541d0",
   "metadata": {},
   "source": [
    "List can obviously host also homogeneous types of data, such as `int` or `float`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b1ab9a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "mylistHomogeneous = [2, 3.14, 10.5, 11.13, 12.7, 4.31]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25008f33",
   "metadata": {},
   "source": [
    "### Numpy Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbeacd1c",
   "metadata": {},
   "source": [
    "Numpy arrays were designed to be lists with superpowers, so everything we learned about  `lists` will apply to `numpy arrays` as well!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2da5f66c",
   "metadata": {},
   "source": [
    "A NumPy array is a multidimensional, uniform collection of elements (that is, all elements occupy the same number of bytes in memory). An array is characterized by\n",
    " - the type of elements it contains and\n",
    " - its shape. \n",
    " \n",
    "For example, a matrix might be represented as an array of shape M×N that contains numbers, such as floating-point or complex numbers. Unlike matrices, NumPy arrays can have up to 32 dimensions; they might also contain other kinds of elements (or even combinations of elements), such as Booleans or dates. [Ref. Van Der Walt et al. IEEE](10.1109/MCSE.2011.37)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b6d0528",
   "metadata": {},
   "source": [
    "Not to state the obvious, but to use `numpy arrays`, we'll need to `import` the library `numpy`. The standard is to import `numpy` as `np`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43a39707",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fd20cd7",
   "metadata": {},
   "source": [
    "`arrays` can be made by simply asking for one and filling it out with values, in much the same way we make a `list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26c71a2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "myarr = np.array([2, 4, 6, 8, 9, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c9118a7",
   "metadata": {},
   "source": [
    "The command `print` can be used also on `NumPy arrays` and it returns the content of the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "582f0f51",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(myarr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "010710d6",
   "metadata": {},
   "source": [
    "By simply returning the array object name (`myarr`) the output is a bit more informative and it returns the type (`array`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5cffedf",
   "metadata": {},
   "outputs": [],
   "source": [
    "myarr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "620cc900",
   "metadata": {},
   "source": [
    "From then on, all the indexing we've learned so far applies directly! Square brackets are used for indexing and the same type of addressing can be used as we have learned for `lists`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d71ad59",
   "metadata": {},
   "outputs": [],
   "source": [
    "myarr[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c363d45e",
   "metadata": {},
   "outputs": [],
   "source": [
    "myarr[-3:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfef2a0a",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - Create a `NumPy array` containing both `int`, `float` and `complex` datatypes.\n",
    "\n",
    "  [Use the cell below to show your code]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "687db55e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9eef86ca",
   "metadata": {},
   "source": [
    "  - Create a `NumPy array` containing both `str` as datatypes.\n",
    "\n",
    "  [Use the cell below to show your code]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75af8e7c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0b02fa87",
   "metadata": {},
   "source": [
    "### Operations on `numpy arrays`: the difference between `lists` and `arrays`\n",
    "Indeed, we can make an array directly out of a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3b92752",
   "metadata": {},
   "outputs": [],
   "source": [
    "myArrFromList = np.array(mylistHomogeneous)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2284fad",
   "metadata": {},
   "outputs": [],
   "source": [
    "myArrFromList"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "617126a6",
   "metadata": {},
   "source": [
    "And then of course we can index it exactly the same way, so... *Wait, why are we making arrays now? What's the difference?*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c860495f",
   "metadata": {},
   "source": [
    "One **huge** difference is that if we wanted to do some math with basic Python lists, the fact that they can hold multiple types of data elements does not assure that the mathematical operations will perform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87a02129",
   "metadata": {},
   "outputs": [],
   "source": [
    "mylistHomogeneous + 5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8691df4b",
   "metadata": {},
   "source": [
    "`numpy` arrays instead contain numerical elements by definition. This definition assures the ability to perform math ith the arrays. So, whereas the addition above did not work when using the `list`, it does work when using the `numpy` array, even though both `list` and `array` contain the same elements!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "056f7707",
   "metadata": {},
   "outputs": [],
   "source": [
    "myArrFromList + 5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5cd22ea",
   "metadata": {},
   "source": [
    "Now **that** seems like it might be useful!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b27b00fd",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - What happens when you add a number to a `NumPy array`? How do the content of the array change?\n",
    "\n",
    "  [Use the cell below to show answer the question]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c1e2e88",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5a773b69",
   "metadata": {},
   "source": [
    "  - Create a new `array` and multiply the array by a complex number:\n",
    "\n",
    "  [Use the cell below to show your code]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81ae359b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d8d02957",
   "metadata": {},
   "source": [
    "    [Use the cell below to describe in your own words what happnes to the elements of the array after multiplying for a complex number]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c8b6acf",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2415291",
   "metadata": {},
   "source": [
    "### More operations on `arrays`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac4de629",
   "metadata": {},
   "source": [
    "Two arrays can be added, or subtracted, or multiplied or whatever!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44e6b598",
   "metadata": {},
   "outputs": [],
   "source": [
    "myarr + myArrFromList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29ebb25a",
   "metadata": {},
   "outputs": [],
   "source": [
    "myarr * myArrFromList"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d3f210c",
   "metadata": {},
   "outputs": [],
   "source": [
    "myarr / myArrFromList"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9514eea5",
   "metadata": {},
   "source": [
    "We can also **combine** our 2 arrays into a single ***two dimensional (2D) array***."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15773b73",
   "metadata": {},
   "outputs": [],
   "source": [
    "twoDarr = np.array([myarr, myArrFromList])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38d0cb66",
   "metadata": {},
   "outputs": [],
   "source": [
    "twoDarr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc081a09",
   "metadata": {},
   "source": [
    "Simple though this may seem, *2D arrays just like this are the bedrock of data analysis!* Arrays of real data are usually larger – sometimes much much larger! – but all the principles are the same and all you as a Data Scientists need to remember is the dimensionality of the data arrays. Python will then compute what you ask for."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "512de187",
   "metadata": {},
   "source": [
    "But, hold on one second. Remembering the dimensionality of the array is ** *very* ** imoortant. Indeed Python can perform some operations if two arrays do *not* have the same dimensions, but other operations are likely to fail.\n",
    "\n",
    "For example, imagine two `arrays` with different dimensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df19c2bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "smallArray = [2, 3, 4]\n",
    "largeArray = [2, 3, 4, 5, 6, 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd3dc621",
   "metadata": {},
   "source": [
    "The two arrays can be added together by using the symbol `+`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46acd3f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "smallArray + largeArray"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0580b3e",
   "metadata": {},
   "source": [
    "Yet, the same two arrays cannot be multiplied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "325e8112",
   "metadata": {},
   "outputs": [],
   "source": [
    "smallArray * largeArray"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1267deda",
   "metadata": {},
   "source": [
    "This is because Python cannot identify elements to match during the multiplication."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ffafdbf",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - What happens when you add two arrays of different dimensions? Say one array with 6 complex numers and one with 4 `float` numbers?\n",
    "\n",
    "      [Use the cell below to show how to create and add the two arrays]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "496cf70b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "063a3fbd",
   "metadata": {},
   "source": [
    "    [Use the cell below to describe in your own words what happens to the arrays as result of the multiplication]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25eb7e66",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8a781202",
   "metadata": {},
   "source": [
    "### Methods of `numpy arrays`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00649964",
   "metadata": {},
   "source": [
    "So the shape of the array (the dimensionality) is key, especially if we plan on doing math with the arrays, whihc is the primary goal of the arrays! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f411c993",
   "metadata": {},
   "source": [
    "`numpy arrays` are Python objects and as such they have `methods`. A variety of methods exist for the array and `shape` is the one that allow us to retrieve the dimensionality of an array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2576396",
   "metadata": {},
   "outputs": [],
   "source": [
    "twoDarr.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1939541",
   "metadata": {},
   "source": [
    "Unlike lists, which are always just lists, arrays can come in any shape. So it's *really* convenient that they can tell us what shape they are straight away."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50a84cdc",
   "metadata": {},
   "source": [
    "Indexing into 2D arrays is a straightforward extension of indexing into 1D arrays or lists. We just provide a second index after a `,` (comma). Like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a175553a",
   "metadata": {},
   "outputs": [],
   "source": [
    "twoDarr[1,3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "073d6afa",
   "metadata": {},
   "source": [
    "The first index refers to the **row index**, and the second to the **column index**. In this case, we're asking for the value in the second row and the fourth column, which is indeed 7 (remember *the first row and column are index=0!*)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23ee8afa",
   "metadata": {},
   "source": [
    "We can play all the same games indexing with 2D arrays as we can with 1D arrays, we just have to remember that everything before the comma `,` refers to the *rows* in that it specifies locations along the *vertical dimension*, and everything after the comma `,` refers to the *columns* in that it specifies locations along the *horizontal dimension*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "750f6f9e",
   "metadata": {},
   "source": [
    "So this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5d9f101",
   "metadata": {},
   "outputs": [],
   "source": [
    "twoDarr[:,0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f3af80b",
   "metadata": {},
   "source": [
    "means \"Give me all the rows (the colon `:`) in the first 3 columns (the \"`0:3`).\"\n",
    "\n",
    "I told you that the colon all alone by itself would end up being useful!!! In this case for example, by using the `:` you do not need to type many indices (one per row) and you even do not need to remmeber how many rows there are, just use `:` and Python will return all the elements."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6a63346",
   "metadata": {},
   "source": [
    "A few more examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8b88d71",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the last row (regardless of the number of rows, \n",
    "# again you do not need to knowhow many rows exist)\n",
    "twoDarr[-1,:] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cec900f",
   "metadata": {},
   "outputs": [],
   "source": [
    "twoDarr[:,-2:] # last two columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cb04019",
   "metadata": {},
   "outputs": [],
   "source": [
    "twoDarr[0,::2] # first row, every other column"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "500e0aba",
   "metadata": {},
   "source": [
    "To get good at this, you don't need natural born talent or anything like that. Like so much in life, the key is *practice, practice, practice*!!! So play around! You can't break your computer or anything!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf0e9f4a",
   "metadata": {},
   "source": [
    "Another neat trick that arrays can do is *transpose* themselves, flipping the rows for columns.\n",
    "\n",
    "(Hold your right hand in front of your face so that you're looking at your palm with your fingers pointing towards the left. Now flip your hand so that you're looking at the back of your hand with your fingers pointing up. You just *transposed* your hand such that the first row (your pointer finger) became the first column!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83665f10",
   "metadata": {},
   "outputs": [],
   "source": [
    "colarr = twoDarr.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40bd0891",
   "metadata": {},
   "outputs": [],
   "source": [
    "colarr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3ae4342",
   "metadata": {},
   "source": [
    "Why would we want to do that? By convention, *variables* in datasets should correspond to the columns, and *observations* should correspond to the rows. So we have taken data in which this was not so and turned it into an array in which the columns are the first few non-prime numbers and the prime numbers, respectively, and the rows correspond to the instances in order (1st , 2nd, 3rd, ....).\n",
    "\n",
    "We have just done a little of what is known as **data wrangling**. While not as fun as data visualization, data wrangling is often a big part of any analysis project!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7f79aa",
   "metadata": {},
   "source": [
    "Now that we have the data into shape, we can unleash all the powers of numpy arrays, powers which pandas DataFrames will inherit and build upon!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "663d153f",
   "metadata": {},
   "source": [
    "For example, who's bigger overall, the primes or the non-primes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7768011",
   "metadata": {},
   "outputs": [],
   "source": [
    "colarr.sum(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f693f3d",
   "metadata": {},
   "source": [
    "The primes win! \n",
    "In `colarr.sum(0)`, the 0 means \"the first (vertical) dimension\", i.e., sum the values *across the rows* within each column. To sum along the second dimension, we do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f1481ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "colarr.sum(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec5f46b4",
   "metadata": {},
   "source": [
    "So any numpy array knows how to add up the numbers in it by row or by column (see what happens if you leave off the dimension, like this `colarr.sum()`. The list of things that numpy arrays can do themselves is pretty impressive.\n",
    "\n",
    "Check it out [here](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html).\n",
    "\n",
    "(or paste this into your browser: https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "338d6e62",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "\n",
    "  - How many methods does a numpy array have? [Report your answer here]\n",
    "  \n",
    "  - Create a new 2-dimensional array, and show the use of two methods not used above (`prod` and `round` could be two simple ones, but no pressure):\n",
    "  \n",
    "  [Use the cell below to show how to create and add the two arrays]\n",
    "  \n",
    "*Hint:* The symbol `?` can be used at the end of a method and that can help understand how to use the method, for example, `myarray.shape?`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de1d9b1f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ea8df3af",
   "metadata": {},
   "source": [
    "### `NumPy` methods to create arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65048d77",
   "metadata": {},
   "source": [
    "Often, we want to create a array that we know we're going to put values in later. For example, we might be planning on doing a computation that will result in 3 sets of 7 values, and we want be able to store them directly into an array. We can pre-make an array filled with zeros with `np.zeros(r, c)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d2cf6a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "myzeros = np.zeros((7, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebbeceb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "myzeros"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "990f9543",
   "metadata": {},
   "source": [
    "Another handy method to create arrays is `ones`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d072457f",
   "metadata": {},
   "outputs": [],
   "source": [
    "myones = np.ones((3,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1a1a3c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "myones"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f29559ec",
   "metadata": {},
   "source": [
    "We will encounter other `NumPy` methods in later tutorials. For the time being one last method that will turn out very handy when modelling data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "531a6494",
   "metadata": {},
   "outputs": [],
   "source": [
    "myRandomNumArray = np.random.randn(10,1)\n",
    "print(myRandomNumArray)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a9bf11a",
   "metadata": {},
   "source": [
    "The `numpy` submodule `random` contains a variety of methods to create arrays containing random numbers. Generating random numbers is helpful in many applications, for example, they can be used to create normally distributed noise, or data with normally distributed noise, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac3fe9aa",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "  \n",
    "  - Create a new 1-dimensional array of uniformly-distributed random number:\n",
    "  \n",
    "  [Use the cell below to show your code]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "619a3b7a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f7fe529a",
   "metadata": {},
   "source": [
    "  \n",
    "  - Create a new  2-dimensional array of normally-distributed random number:\n",
    "  \n",
    "  [Use the cell below to show your code]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e4e7eb6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9b4cf9fc",
   "metadata": {},
   "source": [
    "Let's now create a simple 1-D array and explore what happens when we add a number to the values and what happens when we multiply the numbers in the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c45aea8",
   "metadata": {},
   "outputs": [],
   "source": [
    "size  = 20\n",
    "origArray = np.random.randn(size,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76464a58",
   "metadata": {},
   "source": [
    "Let's look at the values in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60b2c57b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(origArray)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20d92a80",
   "metadata": {},
   "source": [
    "There seem to be a variety of numbers, some positive, some negative, as expected because `randn` is supposed to generate numbers centered at 0 (i.e., with mean 0) and standard deviation of 1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d78aba3b",
   "metadata": {},
   "source": [
    "Let's compute the standard deviation and mean of these numbers. Numpy provides to handy methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cb0b1d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "mean = np.mean(origArray)\n",
    "sd = np.std(origArray)\n",
    "print(['The mean is:', mean])\n",
    "print(['the STD is:', sd])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aac5969",
   "metadata": {},
   "source": [
    "Well, okay, the mean is not quite close to 0, but perhaps close enough? The standard deviation seems pretty close to the expected value of 1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "107aa591",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "  \n",
    "  - Create a new 1-dimensional array of 100 normally-distributed random numbers:\n",
    "  \n",
    "  [Use the cell below to show your code]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8db274a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "da0fb155",
   "metadata": {},
   "source": [
    "   - What happens to the mean and standard deviation after increasing the size of the array? Are they closer of further from the expected values? Why? \n",
    "        \n",
    "  [Use the cell below to show your code]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71a3bdb7",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ad4a5ff3",
   "metadata": {},
   "source": [
    "What happens if we add 5 to the array?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57e99de9",
   "metadata": {},
   "outputs": [],
   "source": [
    "x  = 5 + origArray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ee8c164",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d065eaf0",
   "metadata": {},
   "source": [
    "It looks like the values shifted. But how much? It looks like they recentered at 5, the value we added. So we can perhaps assume that now the distribution of numbers is normally distributed but with a mean of 5. The standard deviation has not bee changed. It is still at 1, trust me for the moment and let try multiplying the numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a462f822",
   "metadata": {},
   "outputs": [],
   "source": [
    "x  = 2 * origArray\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e5a455",
   "metadata": {},
   "source": [
    "It looks like the values increased. There seem to be a larger variality, more bigger numbers, both negative and positive. So perhaps the STD is not at 1 anymore. Could it be at 2?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee254e49",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "  \n",
    "  - Compute the mean, std and median of `x`:\n",
    "  \n",
    "  [Use the cell below to show your code]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbd75e7e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3ef0de78",
   "metadata": {},
   "source": [
    "  \n",
    "  - What are the mean, std and median of `x`? Why, what is going on here?\n",
    "  \n",
    "  [Use the cell below to show answer in your own words]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c595912",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "85056a39",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af07f25",
   "metadata": {},
   "source": [
    "So in this tutorial we have shown how to organize and manipulate data using Python `numpy` `arrays`.\n",
    "\n",
    "So those are the basics of numpy arrays. They:\n",
    "\n",
    "* store values in rows and columns\n",
    "* each dimension starts at index zero (like lists)\n",
    "* can be accessed using\n",
    "    - square brackets `[]` with row and column indexes separated by a comma\n",
    "    - integer indexes (including negative \"start from the end\" indexes)\n",
    "    - a colon `:` (or two if you want a step value other than 1)\n",
    "* can have maths done to every element in one go\n",
    "* can be added, subtracted, etc. from one another\n",
    "* have superpowers! they can compute stuff along their rows and columns!\n",
    "\n",
    "The operations that are available for these two data types will be the base for many things that you might need to do as a Data Scientist. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd832e70",
   "metadata": {},
   "source": [
    "`Numpy arrays` Can host a variety of data types. Although this might be too much for now, below a table of all the data types an `array` can support:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9b0ce9d",
   "metadata": {},
   "source": [
    "| Type\t| Description |\n",
    "| --- | --- |\n",
    "| bool |\tBoolean (True or False) stored as a bit (0, 1) |\n",
    "| inti\t| Platform integer (normally either int32 or int64) |\n",
    "| int8\t| Byte (-128 to 127) |\n",
    "| int16\t| Integer (-32768 to 32767) |\n",
    "| int32\t| Integer (-2 ** 31 to 2 ** 31 -1) |\n",
    "| int64\t| Integer (-2 ** 63 to 2 ** 63 -1) |\n",
    "| uint8\t| Unsigned integer (0 to 255) |\n",
    "| uint16\t| Unsigned integer (0 to 65535) |\n",
    "| uint32\t| Unsigned integer (0 to 2 ** 32 – 1) |\n",
    "| uint64\t| Unsigned integer (0 to 2 ** 64 – 1) |\n",
    "| float16\t| Half precision float: sign bit, 5 bits exponent, and 10 bits mantissa |\n",
    "| float32\t| Single precision float: sign bit, 8 bits exponent, and 23 bits mantissa |\n",
    "| float64 or float\t| Double precision float: sign bit, 11 bits exponent, and 52 bits mantissa |\n",
    "| complex64\t| Complex number, represented by two 32-bit floats (real and imaginary components) |\n",
    "| complex128 or complex\t| Complex number, represented by two 64-bit floats (real and imaginary components) |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57c4bbcc",
   "metadata": {},
   "source": [
    "$\\color{blue}{\\text{Complete the following exercise.}}$\n",
    "  \n",
    "  - Generate an 1-D array of mean = 10 and std = 1.5:\n",
    "  \n",
    "  [Use the cell below to show your code]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f0ea518",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}