notebooks/02.01-Dealing-with-Data-Using-Python-NumPy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--BOOK_INFORMATION-->\n",
    "<a href=\"https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv\" target=\"_blank\"><img align=\"left\" src=\"data/cover.jpg\" style=\"width: 76px; height: 100px; background: white; padding: 1px; border: 1px solid black; margin-right:10px;\"></a>\n",
    "*This notebook contains an excerpt from the book [Machine Learning for OpenCV](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv) by Michael Beyeler.\n",
    "The code is released under the [MIT license](https://opensource.org/licenses/MIT),\n",
    "and is available on [GitHub](https://github.com/mbeyeler/opencv-machine-learning).*\n",
    "\n",
    "*Note that this excerpt contains only the raw code - the book is rich with additional explanations and illustrations.\n",
    "If you find this content useful, please consider supporting the work by\n",
    "[buying the book](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv)!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "< [Working with Data in OpenCV](02.00-Working-with-Data-in-OpenCV.ipynb) | [Contents](../README.md) | [Loading External Datasets in Python](02.02-Loading-External-Datasets-in-Python.ipynb) >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dealing with Data Using Python's NumPy Package\n",
    "\n",
    "If you followed the advice outlined in the previous chapter and installed the Python Anaconda stack, you already have NumPy installed and are ready to go. If you are more the do-it-yourself type, you can go http://www.numpy.org and follow the installation instructions found there.\n",
    "\n",
    "I informed you earlier that it would be okay if you weren't a Python expert yet. Who knows,\n",
    "perhaps you're just now switching from OpenCV's C++ API. This is all fine. I wanted to give\n",
    "you a quick overview on how to get started with NumPy. If you are a more advanced\n",
    "Python user, you may simply skip this subsection.\n",
    "\n",
    "Once you are familiar with NumPy, you will find that most scientific computing tools in the\n",
    "Python world are built around it. This includes OpenCV, so the time spent on learning\n",
    "NumPy is guaranteed to benefit you in the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing NumPy\n",
    "\n",
    "Once you start a new IPython or Jupyter session, you can import the NumPy module and\n",
    "verify its version as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Recall that in the Jupyter Notebook you can hit Ctrl + Enter to execute a\n",
    "cell once you have typed the command. Alternatively, Shift + Enter\n",
    "executes the cell and automatically inserts or selects the cell below it.\n",
    "Check out all the keyboard shortcuts by clicking on Help > Keyboard\n",
    "Shortcut, or take a quick tour by clicking on Help > User Interface Tour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.12.1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numpy.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the pieces of the package discussed here, I would recommend using NumPy version 1.8\n",
    "or later. By convention, you'll find that most people in the scientific Python world will\n",
    "import NumPy using `np` as an alias:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.12.1'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding NumPy arrays\n",
    "\n",
    "Python is a **weakly-typed language**.  This means that you do\n",
    "not have to specify a data type when you create a new variable. For example, the following\n",
    "will automatically be represented as an integer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can double-check the variable type as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going a step further, we can create a list of integers as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_list = list(range(10))\n",
    "int_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similary, we can create a list of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str_list = [str(i) for i in int_list]\n",
    "str_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, lists are not very flexible to do math on. Let's say, for example, we wanted to\n",
    "multiply every element in <tt>int_list</tt> by a factor of 2. A naive approach might be to do the\n",
    "following – but see what happens to the output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_list * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not really what we wanted!\n",
    "\n",
    "Instead, operating on every element in the list gets really easy with NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr = np.array(int_list)\n",
    "int_arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, every NumPy array comes with a list of attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "int_arr ndim:  1\n",
      "int_arr shape:  (10,)\n",
      "int_arr size:  10\n",
      "int_arr dtype:  int64\n"
     ]
    }
   ],
   "source": [
    "print(\"int_arr ndim: \", int_arr.ndim)\n",
    "print(\"int_arr shape: \", int_arr.shape)\n",
    "print(\"int_arr size: \", int_arr.size)\n",
    "print(\"int_arr dtype: \", int_arr.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can display all attributes and methods of a NumPy array by typing out the name of the array, add a period, and then hit <tt>&lt;TAB&gt;</tt>:\n",
    "\n",
    "<tt>In [X]: int_arr.&lt;TAB&gt;</tt>\n",
    "\n",
    "This will display a dropdown menu with a whole lot of attributes, such as the ones mentioned above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing single array elements by indexing\n",
    "\n",
    "If you are familiar with Python's standard list indexing, indexing in NumPy will feel quite\n",
    "familiar.\n",
    "In a one-dimensional array, the i-th value (counting from zero) can be accessed by\n",
    "specifying the desired index in square brackets, just as with Python lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To index from the back of the array, you can use negative indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a few other cool tricks.\n",
    "\n",
    "For example, return all elements from index 2 up to index 5, but don't include index 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 3, 4])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[2:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return all elements from the beginning of the array up to index 5 - 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return all elements from index 5 up to the end of the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 6, 7, 8, 9])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return every other element, starting at index 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 4, 6, 8])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[::2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return all elements of the array, but in reverse order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_arr[::-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating multi-dimensional arrays\n",
    "Arrays can be N-dimensional.\n",
    "In machine learning, we will often deal with at least 2-D arrays, where the column index stands for the values of a particular feature, and the rows contain the actual feature values.\n",
    "\n",
    "Let's say we want to create an array with 3 rows and 5 columns, with all elements initialized to zero. If we don't specify a data type, NumPy will default to using floats:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.],\n",
       "       [ 0.,  0.,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_2d = np.zeros((3, 5))\n",
    "arr_2d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you probably know from your OpenCV days, this could be interpreted as a 3x5 grayscale image with all pixels set to 0 (black). Analogously, if we wanted to create a tiny 2x4 pixel image with 3 color channels (R, G, B), but all pixels set to white, we would use NumPy to create a 3-D array with dimensions 2x4x3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 1.,  1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.,  1.]],\n",
       "\n",
       "       [[ 1.,  1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.,  1.]],\n",
       "\n",
       "       [[ 1.,  1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.,  1.]]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_float_3d = np.ones((3, 2, 4))\n",
    "arr_float_3d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the first dimension defines the color channel (R, G, B). Thus, if this were real image data, we could easily grab the red color information by slicing the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  1.,  1.,  1.],\n",
       "       [ 1.,  1.,  1.,  1.]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_float_3d[0, :, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In OpenCV, images either come as 32-bit float arrays with values between 0 and 1, or they come as 8-bit integer arrays with values between 0 and 255. Hence, we can also create a 2x4 pixel, all-white RGB image using 8-bit integers, by specifying the dtype attribute of the NumPy array, and by multiplying all the 1's in the array with 255:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[255, 255, 255, 255],\n",
       "        [255, 255, 255, 255]],\n",
       "\n",
       "       [[255, 255, 255, 255],\n",
       "        [255, 255, 255, 255]],\n",
       "\n",
       "       [[255, 255, 255, 255],\n",
       "        [255, 255, 255, 255]]], dtype=uint8)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr_uint_3d = np.ones((3, 2, 4), dtype=np.uint8) * 255\n",
    "arr_uint_3d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reminder about built-in documentation\n",
    "\n",
    "As you read through this chapter, don't forget that IPython gives you the ability to quickly explore the contents of a package (by using the tab-completion feature), as well as the documentation of various functions\n",
    "(using the <tt>?</tt> character)\n",
    "\n",
    "For example, to display all the contents of the numpy namespace, you can type this:\n",
    "\n",
    "    $ ipython\n",
    "    In [1]: import numpy as np\n",
    "    In [2]: np.<TAB>\n",
    "\n",
    "Try it out yourself in the empty cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to display NumPy's built-in documentation, you can use this:\n",
    "\n",
    "    In [3]: np?\n",
    "\n",
    "Then hit <tt>Shift+Enter</tt> to execute the cell.\n",
    "\n",
    "Try it out yourself in the empty cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's another neat trick. Let's say you're typing away at a command to create linearly spaced values in an interval using NumPy's arange function, but you forgot the exact syntax. What you do is you start typing the function's name, then use Shift+TAB to display the function signature:\n",
    "In [4]: np.arange(<Shift+TAB>\n",
    "Try it out yourself in the empty cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More detailed documentation, along with tutorials and other resources, can be found at http://www.numpy.org."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "< [Working with Data in OpenCV](02.00-Working-with-Data-in-OpenCV.ipynb) | [Contents](../README.md) | [Loading External Datasets in Python](02.02-Loading-External-Datasets-in-Python.ipynb) >"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}