lecture_12_numpy_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: center\">\n",
    "    <div style=\"font-size: xxx-large ; font-weight: 900 ; color: rgba(0 , 0 , 0 , 0.8) ; line-height: 100%\">\n",
    "        NumPy &amp; Pandas\n",
    "    </div>\n",
    "    <div style=\"font-size: x-large ; padding-top: 20px ; color: rgba(0 , 0 , 0 , 0.5)\">\n",
    "        Data Processing\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NumPy** is a library, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.\n",
    "\n",
    "- For many tasks you could use Python's lists and numerical data types (int, float) to compute something. However, in reality, Python can be quite slow if you want to compute many values.\n",
    "- NumPy also provides you with a syntax and commands that are very close to how you would write mathematical equations. It therefore allows an easier transfer from equations into code. NumPy is often a lot faster for larger datasets.\n",
    "\n",
    "**Pandas** is a library for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It builds on NumPy.\n",
    "\n",
    "- Pandas is especially useful when you are handling tabular data (with rows and columns, such as a `.csv` file).\n",
    "- It also works nicely with the plotting software `seaborn` (next lecture)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing NumPy and Pandas\n",
    "\n",
    "You will find many examples for the usage of NumPy and Pandas online. Often you will encounter examples such as:\n",
    "```python\n",
    "import numpy as np\n",
    "np.max()\n",
    "```\n",
    "or\n",
    "```python\n",
    "import pandas as pd\n",
    "pd.read_csv()\n",
    "```\n",
    "\n",
    "Therefore a good practice is to always import NumPy with `import numpy as np` and pandas with `import pandas as pd`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading text files with NumPy\n",
    "\n",
    "Documentation: [numpy.loadtxt](https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html)\n",
    "\n",
    "Load data from a text file.\n",
    "\n",
    "Each row in the text file must have the same number of values.\n",
    "\n",
    "The following lists all arguments that `loadtxt` accepts:\n",
    "```python\n",
    "numpy.loadtxt(fname, dtype=<class 'float'>, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None, unpack=False, ndmin=0, encoding='bytes', max_rows=None)\n",
    "```\n",
    "\n",
    "We will use `fname`, `unpack`, and `usecols`:\n",
    "\n",
    "`fname` : file, str, or pathlib.Path\n",
    "\n",
    "    File or filename to read.  If the filename extension is ``.gz`` or ``.bz2``, the file is first decompressed.\n",
    "\n",
    "`unpack` : bool\n",
    "\n",
    "    If True, the returned array is transposed, so that arguments may be unpacked using x, y, z = loadtxt(...). When used with a structured data-type, arrays are returned for each field. Default is False.\n",
    "\n",
    "\n",
    "`usecols` : int or sequence\n",
    "\n",
    "    Which columns to read, with 0 being the first. For example, usecols = (1,4,5) will extract the 2nd, 5th and 6th columns. The default, None, results in all columns being read.\n",
    "\n",
    "    When a single column has to be read it is possible to use an integer instead of a tuple. E.g usecols = 3 reads the fourth column the same way as usecols = (3,) would.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**How does the raw txt file look like?**\n",
    "\n",
    "This is a simple Python way of reading all lines from a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# time ns ew ud\n",
      "\n",
      "1574331571.7485132 1.124 148.1 104.1\n",
      "\n",
      "1574331572.7485132 1.224 149.1 105.1\n",
      "\n",
      "1574331573.7485132 1.324 150.1 106.1\n",
      "\n",
      "1574331574.7485132 1.524 151.1 107.1\n"
     ]
    }
   ],
   "source": [
    "with open('lecture_12/station_data.txt') as f:\n",
    "    for line in f.readlines():\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading all columns**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.57433157e+09, 1.12400000e+00, 1.48100000e+02, 1.04100000e+02],\n",
       "       [1.57433157e+09, 1.22400000e+00, 1.49100000e+02, 1.05100000e+02],\n",
       "       [1.57433157e+09, 1.32400000e+00, 1.50100000e+02, 1.06100000e+02],\n",
       "       [1.57433157e+09, 1.52400000e+00, 1.51100000e+02, 1.07100000e+02]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data = np.loadtxt('lecture_12/station_data.txt')\n",
    "station_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: \"e+09\" is a scientific number notation and means \"*10^9\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading all columns (unpack)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: [1.57433157e+09 1.57433157e+09 1.57433157e+09 1.57433157e+09]\n",
      "ns: [1.124 1.224 1.324 1.524]\n",
      "ew: [148.1 149.1 150.1 151.1]\n",
      "ud: [104.1 105.1 106.1 107.1]\n"
     ]
    }
   ],
   "source": [
    "time, ns, ew, ud = np.loadtxt('lecture_12/station_data.txt', unpack=True)\n",
    "print(f\"Time: {time}\")\n",
    "print(f\"ns: {ns}\")\n",
    "print(f\"ew: {ew}\")\n",
    "print(f\"ud: {ud}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading 2 columns**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ns: [1.124 1.224 1.324 1.524]\n",
      "ud: [104.1 105.1 106.1 107.1]\n"
     ]
    }
   ],
   "source": [
    "ns, ud = np.loadtxt('lecture_12/station_data.txt', unpack=True, usecols=(1, 3))\n",
    "print(f\"ns: {ns}\")\n",
    "print(f\"ud: {ud}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading 1 column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time: [1.57433157e+09 1.57433157e+09 1.57433157e+09 1.57433157e+09]\n"
     ]
    }
   ],
   "source": [
    "time = np.loadtxt('lecture_12/station_data.txt', unpack=True, usecols=0)\n",
    "print(f\"Time: {time}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy Data Types\n",
    "Because NumPy handles data separately to how Python does it, special NumPy data types exist (they are similar to the Python ones).\n",
    "\n",
    "Numpy commands that create arrays therefore support the `dtype` argument which allows specifying the type of the array.\n",
    "\n",
    "There are some more types than the ones shown below, but these are the most useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ints = np.array([1,2,3], dtype=np.int)\n",
    "floats = np.array([1,2,3], dtype=np.float)\n",
    "strings = np.array(['a', 'b', 'c'], dtype=np.str)\n",
    "bools = np.array([True, False], dtype=np.bool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with NumPy arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating an array (from a list)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 5, 4])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_data_1d = [1, 2, 5, 4]\n",
    "np.array(list_data_1d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating multi-dimensional arrays (i.e. a matrix) (from a list of lists)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3, 4],\n",
       "       [4, 5, 6, 7]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_data_2d = [\n",
    "    [1,2,3,4],\n",
    "    [4,5,6,7]\n",
    "]\n",
    "np.array(list_data_2d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Getting the shape of an array**\n",
    "\n",
    "Useful to check input shapes, or when iterating through the data manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4,)\n",
      "(2, 4)\n"
     ]
    }
   ],
   "source": [
    "print(np.array(list_data_1d).shape)\n",
    "print(np.array(list_data_2d).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting multiple rows**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5 6 7 8 9]\n",
      "[5 6 7 8 9]\n",
      "[0.  0.5 1.  1.5 2.  2.5 3.  3.5 4.  4.5]\n"
     ]
    }
   ],
   "source": [
    "# Create a range of values (similar to python's range())\n",
    "print(np.arange(10))\n",
    "print(np.arange(5,10))\n",
    "print(np.arange(0, 5, 0.5)) # But allows decimal steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creatings arrays of ones or zeros**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1., 1., 1.],\n",
       "       [1., 1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.ones([2,5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.zeros([5,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating a random array of a specific size**\n",
    "\n",
    "Among others there are `normal` and `uniform` distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.79431066, 0.35815881, 0.53235965, 0.39973807, 0.73490048],\n",
       "        [0.25634318, 0.31949962, 0.46617447, 0.70935159, 0.13732785],\n",
       "        [0.25856235, 0.59593606, 0.7027836 , 0.9836858 , 0.51018931]],\n",
       "\n",
       "       [[0.97593766, 0.56500033, 0.27418398, 0.49473663, 0.50331682],\n",
       "        [0.4634583 , 0.83537112, 0.31873157, 0.86070633, 0.54592973],\n",
       "        [0.02528392, 0.44929906, 0.27123426, 0.43028796, 0.40986868]]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random = np.random.uniform(size=[2,3,5])\n",
    "random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3, 5)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Iterating through the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.79431066 0.35815881 0.53235965 0.39973807 0.73490048]\n",
      " [0.25634318 0.31949962 0.46617447 0.70935159 0.13732785]\n",
      " [0.25856235 0.59593606 0.7027836  0.9836858  0.51018931]]\n",
      "[[0.97593766 0.56500033 0.27418398 0.49473663 0.50331682]\n",
      " [0.4634583  0.83537112 0.31873157 0.86070633 0.54592973]\n",
      " [0.02528392 0.44929906 0.27123426 0.43028796 0.40986868]]\n"
     ]
    }
   ],
   "source": [
    "for row in random:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.79431066 0.35815881 0.53235965 0.39973807 0.73490048]\n",
      "[0.25634318 0.31949962 0.46617447 0.70935159 0.13732785]\n",
      "[0.25856235 0.59593606 0.7027836  0.9836858  0.51018931]\n",
      "[0.97593766 0.56500033 0.27418398 0.49473663 0.50331682]\n",
      "[0.4634583  0.83537112 0.31873157 0.86070633 0.54592973]\n",
      "[0.02528392 0.44929906 0.27123426 0.43028796 0.40986868]\n"
     ]
    }
   ],
   "source": [
    "for row in random:\n",
    "    for value in row:\n",
    "        print(value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Array slicing**\n",
    "\n",
    "Similar to Python lists, but more flexible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1,  2,  3,  4],\n",
       "       [ 5,  6,  7,  8],\n",
       "       [ 9, 10, 11, 12]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array = np.array([\n",
    "    [1,2,3,4],\n",
    "    [5,6,7,8],\n",
    "    [9,10,11,12]\n",
    "])\n",
    "array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2,  3],\n",
       "       [ 6,  7],\n",
       "       [10, 11]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can slice each dimension (also called axis) individually,\n",
    "# simply separate each axis by a \",\"\n",
    "# array[first_dimension, second_dimenstion, third, ...]\n",
    "# \":\" will select all rows, \"1:3\" will select elements 1 and 2\n",
    "array[:,1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Overriding values in an array**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[100,   2,   3,   4],\n",
       "       [  5,   6,   7,   8],\n",
       "       [  9,  10,  11,  12]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Individual values\n",
    "array[0,0] = 100\n",
    "array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[100,   2,   3,   4],\n",
       "       [  5,   6,   7,   8],\n",
       "       [ -1,  -1,  -1,  -1]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Or larger parts by specifying an array of matching size\n",
    "array[2,:] = np.array([-1,-1,-1,-1])\n",
    "array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Math with arrays (*, /, -, +, ...)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Multiply all values in an array with a scalar value**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 4, 6, 8])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.arange(5) * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Full list here: https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.math.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Element-wise multiplication**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  6, 14, 24, 36])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.arange(5) * np.arange(5,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Matrix multiplication**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2., 2., 2.],\n",
       "       [2., 2., 2.],\n",
       "       [2., 2., 2.],\n",
       "       [2., 2., 2.]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.matmul(np.ones([4,2]), np.ones([2,3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Axis 0: [1 2 3 4]\n",
      "Axis 1: [1 5 9]\n"
     ]
    }
   ],
   "source": [
    "# min, max, mean, standard deviation\n",
    "array = np.arange(10)\n",
    "array.min(), array.max(), array.mean(), array.std()\n",
    "# -> Alternatively: np.mean(array)\n",
    "\n",
    "# If you have a matrix you can compute these on indidvidual axis\n",
    "array = np.array([\n",
    "    [1,2,3,4],\n",
    "    [5,6,7,8],\n",
    "    [9,10,11,12]\n",
    "])\n",
    "print('Axis 0:', array.min(axis=0)) # 0 = Columns\n",
    "print('Axis 1:', array.min(axis=1)) # 1 = Rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and loading NumPy arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup\n",
    "times = np.array([1574331571.7485132, 1574331572.7485132, 1574331573.7485132, 1574331574.7485132])\n",
    "ns = np.array([1.124, 1.224, 1.324, 1.524])\n",
    "ew = np.array([148.1, 149.1, 150.1, 151.1])\n",
    "ud = np.array([104.1, 105.1, 106.1, 107.1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Save a single array**\n",
    "\n",
    "`.npy` is the numpy file ending for single array data. You can also omit the `.npy` in the path name and it will be automatically added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save('lecture_12/station_data-times.npy', times)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Save multiple arrays in a single file**\n",
    "\n",
    "Sometimes you want to save multiple arrays. Those will be saved in files ending with `.npz` which basically is a `ZIP-file` with multiple `.npy` files in it.\n",
    "\n",
    "Each `key` you specify while saving will be a `.npy` file in the archive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.savez_compressed(\n",
    "    'lecture_12/station_data.npz',\n",
    "    times=times,\n",
    "    ns=ns,\n",
    "    ew=ew,\n",
    "    ud=ud\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Load NumPy arrays from a drive**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded: [1.57433157e+09 1.57433157e+09 1.57433157e+09 1.57433157e+09]\n"
     ]
    }
   ],
   "source": [
    "loaded_array = np.load('lecture_12/station_data-times.npy')\n",
    "print(\"Loaded:\", loaded_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When loading an archive the returned object can be used like a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded: <numpy.lib.npyio.NpzFile object at 0x0000020EF3881278>\n",
      "Archive entries: ['times', 'ns', 'ew', 'ud']\n",
      "Times: [1.57433157e+09 1.57433157e+09 1.57433157e+09 1.57433157e+09]\n"
     ]
    }
   ],
   "source": [
    "loaded_archive = np.load('lecture_12/station_data.npz')\n",
    "print(\"Loaded:\", loaded_archive)\n",
    "print(\"Archive entries:\", list(loaded_archive.keys()))\n",
    "print(\"Times:\", loaded_archive['times'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading text files with Pandas\n",
    "\n",
    "Documentation: [pandas.read_csv](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)\n",
    "\n",
    "Read a comma-separated values (csv) file into DataFrame (a DataFrame is Pandas way of storing tabular data).\n",
    "\n",
    "Pandas is much faster than `np.loadtxt` for larger files.\n",
    "\n",
    "The following lists some of the arguments that `read_csv` accepts:\n",
    "```python\n",
    "pandas.read_csv(filename, sep=\",\", header=\"infer\", names=None, usecols=None, skiprows=None, comment=None)\n",
    "```\n",
    "\n",
    "sep : str, default ‘,’\n",
    "\n",
    "    Delimiter to use.\n",
    "\n",
    "header : int, list of int, default ‘infer’\n",
    "\n",
    "    Row number(s) to use as the column names, and the start of the data. Default behavior is to infer the column names: if no names are passed the behavior is identical to header=0 and column names are inferred from the first line of the file, if column names are passed explicitly then the behavior is identical to header=None. Explicitly pass header=0 to be able to replace existing names. The header can be a list of integers that specify row locations for a multi-index on the columns e.g. [0,1,3]. Intervening rows that are not specified will be skipped (e.g. 2 in this example is skipped). Note that this parameter ignores commented lines and empty lines if skip_blank_lines=True, so header=0 denotes the first line of data rather than the first line of the file.\n",
    "\n",
    "names : array-like, optional\n",
    "\n",
    "    List of column names to use. If file contains no header row, then you should explicitly pass header=None. Duplicates in this list are not allowed.\n",
    "\n",
    "usecols : list-like, optional\n",
    "\n",
    "    Return a subset of the columns. If list-like, all elements must either be positional (i.e. integer indices into the document columns) or strings that correspond to column names provided either by the user in names or inferred from the document header row(s). For example, a valid list-like usecols parameter would be [0, 1, 2] or ['foo', 'bar', 'baz']. Element order is ignored, so usecols=[0, 1] is the same as [1, 0].\n",
    "    \n",
    "skiprows : list-like, int, optional\n",
    "\n",
    "    Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file.\n",
    "    \n",
    "comment : str, optional\n",
    "\n",
    "    Indicates remainder of line should not be parsed. If found at the beginning of a line, the line will be ignored altogether. This parameter must be a single character. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading all columns from .txt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time     ns     ew     ud\n",
       "0  1.574332e+09  1.124  148.1  104.1\n",
       "1  1.574332e+09  1.224  149.1  105.1\n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\n",
    "    'lecture_12/station_data.txt',\n",
    "    sep=' ',\n",
    "    header=None,\n",
    "    comment='#',\n",
    "    names=['time', 'ns', 'ew', 'ud']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading 2 columns from .txt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ns</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.124</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.224</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.324</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.524</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ns     ud\n",
       "0  1.124  104.1\n",
       "1  1.224  105.1\n",
       "2  1.324  106.1\n",
       "3  1.524  107.1"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\n",
    "    'lecture_12/station_data.txt',\n",
    "    sep=' ',\n",
    "    usecols=(1, 3),\n",
    "    comment='#',\n",
    "    names=['ns', 'ud']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading 1 column from .txt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time\n",
       "0  1.574332e+09\n",
       "1  1.574332e+09\n",
       "2  1.574332e+09\n",
       "3  1.574332e+09"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(\n",
    "    'lecture_12/station_data.txt',\n",
    "    sep=' ',\n",
    "    usecols=(0,),\n",
    "    comment='#',\n",
    "    names=['time']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loading all columns from .csv**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time     ns     ew    ud \n",
       "0  1.574332e+09  1.124  148.1  104.1\n",
       "1  1.574332e+09  1.224  149.1  105.1\n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data = pd.read_csv('lecture_12/station_data.csv')\n",
    "station_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Pandas DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating a DataFrame manually (from a dict)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time     ns     ew     ud\n",
       "0  1.574332e+09  1.124  148.1  104.1\n",
       "1  1.574332e+09  1.224  149.1  105.1\n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {\n",
    "    \"time\": [1574331571.7485132,1574331572.7485132,1574331573.7485132,1574331574.7485132],\n",
    "    \"ns\":   [1.124,1.224,1.324,1.524],\n",
    "    \"ew\":   [148.1,149.1,150.1,151.1],\n",
    "    \"ud\":   [104.1,105.1,106.1,107.1]\n",
    "}\n",
    "pd.DataFrame(data) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating a DataFrame manually (from a list of lists)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>times</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          times     ns     ew     ud\n",
       "0  1.574332e+09  1.124  148.1  104.1\n",
       "1  1.574332e+09  1.224  149.1  105.1\n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [[1574331571.7485132,1.124,148.1,104.1],\n",
    "        [1574331572.7485132,1.224,149.1,105.1],\n",
    "        [1574331573.7485132,1.324,150.1,106.1],\n",
    "        [1574331574.7485132,1.524,151.1,107.1]]\n",
    "columns = [\"times\", \"ns\", \"ew\", \"ud\"]\n",
    "pd.DataFrame(data, columns=columns) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating a DataFrame manually (from a numpy array)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>times</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          times     ns     ew     ud\n",
       "0  1.574332e+09  1.124  148.1  104.1\n",
       "1  1.574332e+09  1.224  149.1  105.1\n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = np.array(\n",
    "        [[1574331571.7485132,1.124,148.1,104.1],\n",
    "        [1574331572.7485132,1.224,149.1,105.1],\n",
    "        [1574331573.7485132,1.324,150.1,106.1],\n",
    "        [1574331574.7485132,1.524,151.1,107.1]])\n",
    "columns = [\"times\", \"ns\", \"ew\", \"ud\"]\n",
    "pd.DataFrame(data, columns=columns) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Specifying an INDEX column**\n",
    "\n",
    "The index is basically a row label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>times</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Station 1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Station 2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Station 3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Station 4</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  times     ns     ew     ud\n",
       "Station 1  1.574332e+09  1.124  148.1  104.1\n",
       "Station 2  1.574332e+09  1.224  149.1  105.1\n",
       "Station 3  1.574332e+09  1.324  150.1  106.1\n",
       "Station 4  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = np.array(\n",
    "        [[1574331571.7485132,1.124,148.1,104.1],\n",
    "        [1574331572.7485132,1.224,149.1,105.1],\n",
    "        [1574331573.7485132,1.324,150.1,106.1],\n",
    "        [1574331574.7485132,1.524,151.1,107.1]])\n",
    "columns = [\"times\", \"ns\", \"ew\", \"ud\"]\n",
    "index = [\"Station 1\", \"Station 2\", \"Station 3\", \"Station 4\"]\n",
    "station_data_with_index = pd.DataFrame(data, columns=columns, index=index) \n",
    "station_data_with_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting a single column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.574332e+09\n",
       "1    1.574332e+09\n",
       "2    1.574332e+09\n",
       "3    1.574332e+09\n",
       "Name: time, dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data['time']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting multiple columns**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time     ns\n",
       "0  1.574332e+09  1.124\n",
       "1  1.574332e+09  1.224\n",
       "2  1.574332e+09  1.324\n",
       "3  1.574332e+09  1.524"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data[['time', 'ns']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting a single row**\n",
    "\n",
    "The first \"unnamed\" column is called the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time    1.574332e+09\n",
       "ns      1.324000e+00\n",
       "ew      1.501000e+02\n",
       "ud      1.061000e+02\n",
       "Name: 2, dtype: float64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# iloc = index location\n",
    "station_data.iloc[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting multiple rows**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time     ns     ew    ud \n",
       "1  1.574332e+09  1.224  149.1  105.1\n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data.iloc[1:4] # Note, this follows the same syntax as list slicing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting a row given an index label**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "times    1.574332e+09\n",
       "ns       1.124000e+00\n",
       "ew       1.481000e+02\n",
       "ud       1.041000e+02\n",
       "Name: Station 1, dtype: float64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data_with_index.loc['Station 1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting multiple rows given an index label**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>times</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Station 1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Station 2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  times     ns     ew     ud\n",
       "Station 1  1.574332e+09  1.124  148.1  104.1\n",
       "Station 2  1.574332e+09  1.224  149.1  105.1"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data_with_index.loc[['Station 1', 'Station 2']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting rows based on a condition**\n",
    "\n",
    "You can use any conditional operator here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time     ns     ew    ud \n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data.loc[station_data['ns'] > 1.224]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Assigning data**\n",
    "\n",
    "You can not only access data stored in `DataFrame`s, you can also manipulate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.574332e+09</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           time     ns     ew    ud \n",
       "0  1.574332e+09  1.124  148.1  104.1\n",
       "1  1.574332e+09  1.224  149.1  105.1\n",
       "2  1.574332e+09  1.324  150.1  106.1\n",
       "3  1.574332e+09  1.524  151.1  107.1"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1.324</td>\n",
       "      <td>150.1</td>\n",
       "      <td>106.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1.524</td>\n",
       "      <td>151.1</td>\n",
       "      <td>107.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   time     ns     ew    ud \n",
       "0     0  1.124  148.1  104.1\n",
       "1     0  1.224  149.1  105.1\n",
       "2     0  1.324  150.1  106.1\n",
       "3     0  1.524  151.1  107.1"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data['time'] = 0\n",
    "station_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ns</th>\n",
       "      <th>ew</th>\n",
       "      <th>ud</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1.124</td>\n",
       "      <td>148.1</td>\n",
       "      <td>104.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1.224</td>\n",
       "      <td>149.1</td>\n",
       "      <td>105.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>3.324</td>\n",
       "      <td>152.1</td>\n",
       "      <td>108.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>3.524</td>\n",
       "      <td>153.1</td>\n",
       "      <td>109.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   time     ns     ew    ud \n",
       "0     0  1.124  148.1  104.1\n",
       "1     0  1.224  149.1  105.1\n",
       "2     2  3.324  152.1  108.1\n",
       "3     2  3.524  153.1  109.1"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data.loc[station_data['ns'] > 1.224] += 2\n",
    "station_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Convert pandas back to numpy**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0.   ,   1.124, 148.1  , 104.1  ],\n",
       "       [  0.   ,   1.224, 149.1  , 105.1  ],\n",
       "       [  2.   ,   3.324, 152.1  , 108.1  ],\n",
       "       [  2.   ,   3.524, 153.1  , 109.1  ]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "station_data.to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many more useful operations you can perform on a Dataframe:\n",
    "- Compute min, max, mean, std\n",
    "- Get unique values\n",
    "- Group columns and perform above operations on the groups \n",
    "- Pivot tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "* You know when to use NumPy\n",
    "* You know when to use Pandas\n",
    "* You know two ways to load data from files\n",
    "* You know the basics of NumPy\n",
    "  - Different ways to create arrays\n",
    "  - Ways to iterate through an array\n",
    "  - Ways to select parts of an array\n",
    "  - Ways to change values of an array\n",
    "  - Mathematical operators on arrays and matrices\n",
    "* You know the basics of Pandas\n",
    "  - Creating a DataFrame\n",
    "  - Selecting rows and columns\n",
    "  - Conditional selects\n",
    "  - Conversion between NumPy and Pandas\n",
    "  - Data assignment\n",
    "* You can convert data between NumPy and Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next excercise: [Exercise 12](exercise_12_numpy_pandas.ipynb)\n",
    "### Next lecture: [Python - Matplotlib & Seaborn](lecture_13_matplotlib_seaborn.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TODO Beispiele mit open read + write"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "##### Authors:\n",
    "* [Julian Niedermeier](https://github.com/sleighsoft)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}