Linear Algebra with Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "This guide was written in Python 3.6.\n",
    "\n",
    "\n",
    "## Python and Pip\n",
    "\n",
    "Download [Python](https://www.python.org/downloads/) and [Pip](https://pip.pypa.io/en/stable/installing/).\n",
    "\n",
    "\n",
    "## Libraries\n",
    "\n",
    "We'll be working with `numpy` and `scipy`, so make sure to install them. Pull up your terminal and insert the following: \n",
    "\n",
    "```\n",
    "pip3 install scipy\n",
    "pip3 install numpy\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Linear Algebra is a branch of mathematics that allows you to concisely describe coordinates and interactions of planes in higher dimensions, as well as perform operations on them. \n",
    "\n",
    "Think of it as an extension of algebra into an arbitrary number of dimensions. Linear Algebra is about working on linear systems of equations. Rather than working with scalars, we work with matrices and vectors. This is particularly important to the study of computer science because vectors and matrices can be used to represent data of all forms - images, text, and of course, numerical values.\n",
    "\n",
    "## Why Learn Linear Algebra?\n",
    "\n",
    "<b>Machine Learning</b>: A lot of Machine Learning concepts are tied to linear algebra concepts. Some basic examples, PCA - eigenvalue, regression - matrix multiplication. As most ML techniques deal with high dimensional data, they are often times represented as matrices.\n",
    "\n",
    "<b>Mathematical Modeling</b>: for example, if you want to capture behaviors (sales, engagement, etc.) in a mathematical model, you can use matrices to breakdown the samples into their own subgroups. This requires some basic matrix manipulation, such as matrix inversion, derivation, solving partial differential, or first order differential equations with matrices, for example. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scalars & Vectors\n",
    "\n",
    "You'll see the terms scalar and vector throughout this course, so it's very important that we learn how to distinguish between the two. A scalar refers to the <b>magnitude</b> of an object. In contrast, a <i>vector</i> has <b>both</b> a magnitude and a <b>direction</b>. \n",
    "\n",
    "An intuitive example is with respect to distance. If you drive 50 miles north, then the scalar value is `50`. Now, the vector that would represent this could be something like `(50, N)` which indicates to us both the direction <i>and</i> the magnitude. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "Using the [distance formula](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html) and [trigonometry functions](https://docs.python.org/3/library/math.html) in Python, calculate the magnitude and direction of a line with the two coordinates, `(5,3)` and `(1,1)`.\n",
    "\n",
    "For more information on [distance formula](http://www.purplemath.com/modules/distform.htm) and [trigonometry functions](http://www2.clarku.edu/~djoyce/trig/formulas.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectors\n",
    "\n",
    "A vector is typically an ordered tuple of numbers which have both a magnitude and direction. It's important to note that vectors are an <b>element</b> of a vector space. \n",
    "\n",
    "In the next section, we'll learn about matrices, which are a rectangular array of values. A vector is simply a one dimensional matrix. \n",
    "\n",
    "Sometimes what we're given, however, isn't a neat two value tuple consisting of a magnitude and direction value. Sometimes we're given something that resembles a list, and from there, we can create a tuple that describes the vector. \n",
    "\n",
    "With that said, we <i>can</i> represent a vector with a list, for example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "A = [2.0, 3.0, 5.0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's write this same vector with `numpy` instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "v = np.array([2.0, 3.0, 5.0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "Write code for two vectors with five values of your choice. The first should be written as a regular one-dimensional list. The other should be be written with numpy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a Norm? \n",
    "\n",
    "Remember the distance formula from the intro section? That's what a norm is, which is why that function we used from scipy was called `linalg.norm`. With that said, just to review, a norm just refers to the magnitude of a vector, and is denoted with ||u||. With numpy and scipy, we can do calculate the norm as follows: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.4772255750516612"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "v = np.array([1,2,3,4])\n",
    "nln.norm(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actual formula looks like: \n",
    "\n",
    "$$ ||v|| = \\sqrt{v_1^2 + ... + v_n^2} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "Find the norm of the vector `[3, 9, 5, 4]` using the actual formula above. You should write a function `find_norm(v1)` that returns this value as a float and then call it on the provided variable `n1`. You should not use `scipy`, but you may use the `math` module. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrices\n",
    "\n",
    "A Matrix is a 2D array that stores real or complex numbers. You can think of them as multiple vectors in an array! You can use numpy to create matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "matrix1 = np.matrix(\n",
    "    [[0, 4],\n",
    "     [2, 0]]\n",
    ")\n",
    "matrix2 = np.matrix(\n",
    "    [[-1, 2],\n",
    "     [1, -2]]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "Using `numpy`, create a 3 x 5 matrix with values of your choice. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.0 Linear Equations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gauss-Jordan Elimination\n",
    "\n",
    "Gaussian Elimination helps to put a matrix in row echelon form, while Gauss-Jordan Elimination puts a matrix in reduced row echelon form?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solving Systems of Equations\n",
    "\n",
    "Consider a set of m linear equations in n unknowns:\n",
    "\n",
    "...\n",
    "\n",
    "We can let:\n",
    "\n",
    "...\n",
    "\n",
    "And re-write the system: \n",
    "\n",
    "```\n",
    "Ax = b\n",
    "```\n",
    "This reduces the problem to a matrix equation and now we can solve the system to find $A^{-1}$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Systems of Equations with Python\n",
    "\n",
    "Using numpy, we can solve systems of equations in Python. Each equation in a system can be represented with matrices. For example, if we have the equation `3x - 9y = -42`, it can be represented as `[3, -9]` and `[-42]`. If we add another equation to the mix, for example,  `2x + 4y = 2`, we can merge it with the previous equation to get `[[3, -9], [2, 4]]` and `[-42, 2]`. Now let's solve for the x and y values.\n",
    "\n",
    "Now, let's put these equations into numpy arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "A = np.array([ [3,-9], [2,4] ])\n",
    "b = np.array([-42,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can use the `linalg.solve()` function to solve the x and y values. Note that these values will be\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-5.,  3.])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.solve(A,b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which means `x = -5` and `y = 3`. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a vector space?\n",
    "\n",
    "A vector space V is a set that contains all linear combinations of its elements. In other words, if you have a set A, the space vector V includes all combinations of the elements in A. \n",
    "\n",
    "With that said, there are three properties that <b>every</b> vector space must follow:\n",
    "\n",
    "1. Additive Closure: If vectors u and v &isin; V, then u + v &isin; V <br>\n",
    "When we earlier stated that the vector space has all combinations of the elements in set A, one of the operations we meant by 'combinations' was <i>vector</i> addition. For example if we have two vectors in set A, let's say (4, 5) and (3, 1), then the vector space of A must have those two vectors, as well as the vector (4+3, 5+1), or `(7, 6)`. This has to be true for any two vectors in set A. \n",
    "\n",
    "2. Scalar Closure: If u &isin; V, then &alpha; &middot; u must &isin; &Nu; for any scalar &alpha; <br>\n",
    "Recall that a scalar is a magnitude value with no direction, such as 5. For a vector space to be a vector space, that means for every vector in the original set A, that vector multiplied by any number (or constant or scalar) must be in the vector space V. \n",
    "\n",
    "3. Additive Identity: There exists a &middot; 0 &isin; V such that u + 0 = u for any u &isin; V <br>\n",
    "In other words, the vector space of set A must contain the zero vector.\n",
    "\n",
    "4. Additive Associativity: If u, v, w &isin; V, then u + (v + w) = (u + v) + w <br>\n",
    "Regardless of the order in which you add multiple vectors, their results should be the same\n",
    "\n",
    "5. Additive Inverse: If u &isin; V, then there exists a vector âˆ’u &isin; V so that u + (âˆ’u) = 0. <br>\n",
    "For example, if the vector (2, 3) &isin; V, then its additive inverse is (-2, -3) and must also be an element of the vector space V. \n",
    "\n",
    "\n",
    "The dimension of a vector space V is the cardinality. It's usually denoted as superscript, for example, &real;<sup>n</sup>. Let's break down what this looks like:\n",
    "\n",
    "- &real;<sup>2</sup> refers to your typical x, y systems of equations. <br>\n",
    "\n",
    "- &real;<sup>3</sup> adds an extra dimension, which means adding another variable, perhaps z."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.0 Linear Transformations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Operations\n",
    "\n",
    "What makes matrices particularly useful is the fact that we can perform operations on them. While it won't be necessarily intuitive why these operations are important right now, it will become obvious in later content.\n",
    "\n",
    "### Addition\n",
    "\n",
    "Matrix addition works very similarlty to normal addition. You simply add the corresponding spots together. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[-1,  6],\n",
       "        [ 3, -2]])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix1 + matrix2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiplication\n",
    "\n",
    "To multiply two matrices with numpy, you can use the `np.dot` method: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 4, -8],\n",
       "        [-2,  4]])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(matrix1, matrix2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, simply, you can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 4, -8],\n",
       "        [-2,  4]])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix1 * matrix2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dot product is an operation that takes two coordinate vectors of equal size and returns a single number. The result is calculated by multiplying corresponding entries and adding up those products. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "Write a function `matrix_add(matrix_A, matrix_B)` that performs matrix addition if the dimensionality is valid. Note that the dimensionality is only valid if input matrix A and input matrix B are of the same dimension in both their row and column lengths. \n",
    "\n",
    "For example, you can add a 3x5 matrix with a 3x5 matrix, but you cannot add a 3x5 matrix with a 3x1 matrix. If the dimensionality is not valid, print this error message \"Cannot perform matrix addition between a-by-b matrix and c-by-d matrix\", where you substitute a, b with the dimension of the input matrix A, and c,d with the dimension of the input matrix B.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identity Matrix\n",
    "\n",
    "A Diagonal Matrix is an n x n matrix with 1s on the diagonal from the top left to the bottom right, such as \n",
    "\n",
    "``` \n",
    "[[ 1., 0., 0.],\n",
    "[ 0., 1., 0.],\n",
    "[ 0., 0., 1.]]\n",
    "```\n",
    "\n",
    "We can generate diagonal matrices with the `eye()` function in Python: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0.,  0.,  0.],\n",
       "       [ 0.,  1.,  0.,  0.],\n",
       "       [ 0.,  0.,  1.,  0.],\n",
       "       [ 0.,  0.,  0.,  1.]])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.eye(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a matrix is multiplied by its inverse, the result is the identity matrix. It's important to note that only square matrices have inverses!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "```\n",
    "A =  [[1 2]\n",
    "      [3 4]]\n",
    "      \n",
    "B = [[-2.   1. ]\n",
    "     [ 1.5 -0.5]]\n",
    "     \n",
    "C = [[1 2 3]\n",
    "     [4 5 6]]\n",
    "```    \n",
    "    \n",
    "1. Given matrix A and B,  mutiply AB - call this `mat1`. Mutiply BA - call this `mat2`.  Are these matrix inverses?\n",
    "\n",
    "2. Given matrix C, create an identity matrix - call it `id1`  to multiply  C*id1- call this `mat3`.  \n",
    "\n",
    "3. Given matrix C, create an identity matrix - call it `id2`  to multiply  id2*C- call this `mat4`.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverse Matrices\n",
    "\n",
    "The matrix A is invertible if there exists a matrix A<sub>-1</sub> such that\n",
    "\n",
    "A<sub>-1</sub>A = I and AA<sub>-1</sub> = I\n",
    "\n",
    "Multiplying inverse matrix is like the division; itâ€™s simply the reciprocal function of multiplication. Letâ€™s say here is a square matrix A, then multiplying its inversion gives the identity matrix I.\n",
    "\n",
    "We can get the inverse matrix with numpy: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.    0.5 ]\n",
      " [ 0.25  0.  ]]\n"
     ]
    }
   ],
   "source": [
    "inverse = np.linalg.inv(matrix1)\n",
    "print(inverse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "Write a function `matrix_inverse(matrix_A)` that outputs the inverse matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.0 Subspaces and Their Dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kernels & Images\n",
    "\n",
    "#### Images \n",
    "\n",
    "The image of a function consists of all the values the function takes in its codomain.\n",
    "\n",
    "#### Kernels\n",
    "\n",
    "The kernel of a matrix A is the dimension of the space mapped to zero under the linear transformation that A represents. The dimension of the kernel of a linear transformation is called the nullity.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a subspace?\n",
    "\n",
    "A vector subspace is a subset of a vector space. That subspace is <b>also</b> a vector space, so it follows all the rules we reviewed above. It's also important to note that if W is a linear subspace of V, then the dimension of W must be &le; the dimension of V.\n",
    "\n",
    "The easiest way to check whether it's a vector subspace is to check if it's closed under addition and scalar multiplication. Let's go over an example:\n",
    "\n",
    "Let's show that the set V = {(x, y, z) | x, y, z &isin; &real; and x*x = z*z } is <b>not</b> a subspace of &real;<sup>3</sup>.\n",
    "\n",
    "If V is actually a subspace of &real;<sup>3</sup>, that means it <b>must</b> follow all the properties listed in the beginning of this section. Recall that this is because all subspaces must also be vector spaces. \n",
    "\n",
    "Let's evaluate the first property that stays the following:\n",
    "\n",
    "If vectors u and v &isin; V, then u + v &isin; V\n",
    "\n",
    "Now, is this true of the set we've defined above? Absolutely not. (1, 1, 1) and (1, 1, -1) are both in V, but what about their sum, (1, 2, 0)? It's not! And because it's not, it does not follow the required properties of a vector space. Therefore, we can conluse that it's also not a subspace. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "1. Write the representation of &real;<sup>2</sup> as a list comprehension - use ranges between -10 and 10 for all values of x and y. \n",
    "\n",
    "2. Write the representation of &real;<sup>3</sup> as a list comprehension - use ranges between -10 and 10 for all values of x, y, and z.  \n",
    "\n",
    "3. Write a list comprehension that represents the the set V = {(x, y, z) | x, y, z &isin; &real; and x+y = 11}. Use ranges between -10 and 10 for all values of x, y, and z. \n",
    "\n",
    "4. Choose three values of x, y, and z that show the set V = {(x, y, z) | x, y, z &isin; &real; and x+y = 11} is <b>not</b> a subspace of &real;<sup>3</sup>. These values should represent a tuple that <i>would</i> be in vector V had it been a vector subspace. Each value should also be between -10 and 10. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Linear Dependence? \n",
    "\n",
    "A set of vectors {v<sub>1</sub>,...,v<sub>n</sub>} is linearly independent if there are scalars c<sub>1</sub>...c<sub>n</sub> (which <b>aren't</b> all 0) such that the following is true:\n",
    "\n",
    "c<sub>1</sub>v<sub>1</sub> + ... + c<sub>n</sub>v<sub>n</sub> = 0\n",
    "\n",
    "#### Example\n",
    "\n",
    "Let's say we have three vectors in a set: x<sub>1</sub> = (1, 1, 1), x<sub>2</sub> = (1, -1, 2), and x<sub>3</sub> = (3, 1, 4). \n",
    "\n",
    "These set of vectors are linear dependent because 2x<sub>1</sub> + x<sub>2</sub> - x<sub>3</sub> = 0. Why is this equal to zero? Again, because 2*(1,1,1) + 1(1,-1,2) - (3,1,4) = (2+1-3, 2-1-1, 2+2-4) = (0, 0, 0). If we can find some equation that satisfies a resultant of 0, it's considered linear dependent!\n",
    "\n",
    "### What is Linear Independence? \n",
    "\n",
    "A set of vectors is considered linear dependent simply if they are not linear dependent! In other words, all the constants from the previous section should be equal to zero. c<sub>1</sub> = c<sub>2</sub> = ... = c<sub>n</sub> = 0\n",
    "\n",
    "### What is a basis? \n",
    "\n",
    "Any linearly independent set of n vectors spans an n-dimensional space. This set of vectors is referred to as the basis of &real;<sup>n</sup>. \n",
    "\n",
    "#### Under vs Overdetermined Matrices\n",
    "\n",
    "When `m < n`, the linear system is said to be <b>underdetermined</b>, e.g. there are fewer equations than unknowns. In this case, there are either no solutions or infinite solutions and a unique solution is not possible.\n",
    "\n",
    "When `m > n`, the system may be <b>overdetermined</b>. In other words, there are more equations than unknowns. They system could be inconsistent, or some of the equations could be redundant. \n",
    "\n",
    "If some of the rows of an m x n matrix are linearly dependent, then the system is <b>reducible</b> and we get get rid of some of the rows. \n",
    "\n",
    "Lastly, if a matrix is square and its rows are linearly independent, the system has a unique solution and is considered <b>invertible</b>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "```\n",
    "A = [[1 1]\n",
    "     [0 1]]\n",
    "\n",
    "B = [[1 2]\n",
    "     [3 6]]\n",
    "```\n",
    "Note: `linalg.det(A)` gives the determinant of matrix A.  \n",
    "\n",
    "Note: `linalg.inv(A)` finds the inverse matrix of A.  \n",
    "\n",
    "Note: `linalg.solve(A,b)` will solve for x.  (Ax = b)\n",
    "\n",
    "Recall: A nonzero determinant is equivacal to the matrix being invertible.  \n",
    "\n",
    "\n",
    "1.  Consider a 2-by-2 identity matrix, I.  We see that this is a basis of &real;<sup>2</sup> spanned by vectors [1,0] and [0,1].  Solve the unique solution for b = [3,4] - call this `sol1`. \n",
    "\n",
    "2.  Given matix A, compute the determinant - call this `det1`.  Find the inverse of this matrix - call this `mat1`.  We see that this is a basis of &real;<sup>dim</sup>.  What integer is `dim`? This is spanned by two vectors, one being [0,1].  What is the other one - call this `bas1`   Solve the unique solution for b = [3,4] - call this `sol2`. (Observe how matrix A and the identity matrix give two different solutions.)\n",
    "\n",
    "3. Given matix B, compute the determinant - call this `det2`. We see that matrix is noninvertible and the rows are linearly dependent. Infact we see this matrix is reducible if we perform the following operation on row 2: `alpha` x row 1 - row 2.  What is the value of `alpha`?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5.0 Orthogonality and Least Squares\n",
    "\n",
    "\n",
    "## QR Decomposition with Gram-Schmidt\n",
    "\n",
    "\n",
    "QR decomposition (also called a QR factorization) of a matrix is a decomposition of a matrix A into a product A = QR of an orthogonal matrix Q and an upper triangular matrix R. QR decomposition is often used to solve the linear least squares problem and is the basis for a particular eigenvalue algorithm, the QR algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "A = np.matrix([[1,1,0], [1,0,1], [0,1,1]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Q,R = np.linalg.qr(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[-0.70710678,  0.40824829, -0.57735027],\n",
       "        [-0.70710678, -0.40824829,  0.57735027],\n",
       "        [-0.        ,  0.81649658,  0.57735027]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[-1.41421356, -0.70710678, -0.70710678],\n",
       "        [ 0.        ,  1.22474487,  0.40824829],\n",
       "        [ 0.        ,  0.        ,  1.15470054]])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.0 Determinants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trace and Determinant\n",
    "\n",
    "The trace of a matrix A is the sum of its diagonal elements. It's important because it's an invariant of a matrix under change of basis and it defines a matrix norm. \n",
    "\n",
    "In Python, this can be done with the `numpy` module: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "print(np.trace(matrix1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which in this case is just `0`. \n",
    "\n",
    "The determinant of a matrix is defined to be the alternating sum of permutations of the elements of a matrix. The formula is as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{bmatrix} \n",
    "    a      & b  \\\\\n",
    "    c      & d  \\\\\n",
    "\\end{bmatrix}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "$$ = ad - bc $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In python, you can use the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-8.0\n"
     ]
    }
   ],
   "source": [
    "det = np.linalg.det(matrix1)\n",
    "print(det)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that an n×n matrix A is invertible $\\iff$ det(A) $\\ne$ 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "Write a function `matrix_det(matrix_A)` that calculates the determinant (an integer) of matrix A if it is a 1x1 or 2x2 matrix. \n",
    "\n",
    "- If the input matrix is not square, print this error message \"Cannot calculate determinant of a non-square matrix.\" \n",
    "- If the input matrix is square but has a higher dimension, print the error message \"Cannot calculate determinant of a n-by-n matrix\", where you substitute n with the dimension of the input matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Types\n",
    "\n",
    "### Under vs Overdetermined Matrices\n",
    "\n",
    "When `m < n`, the linear system is said to be <b>underdetermined</b>, e.g. there are fewer equations than unknowns. In this case, there are either no solutions or infinite solutions and a unique solution is not possible.\n",
    "\n",
    "When `m > n`, the system may be <b>overdetermined</b>. In other words, there are more equations than unknowns. They system could be inconsistent, or some of the equations could be redundant. \n",
    "\n",
    "### Row, Column, and Null Space \n",
    "\n",
    "The <b>column space</b> C(A) of a matrix A (sometimes called the range of a matrix) is the span (set of all possible linear combinations) of its column vectors.\n",
    "\n",
    "The <b>row space</b> of an m x n matrix, A, denoted by R(A) is the set of all linear combinations of the row vectors of A.\n",
    "\n",
    "The <b>null space</b> of an m x n matrix, A, denoted by null(A) is the set of all solutions, x, of the equation Ax = 0<sub>m</sub>.\n",
    "\n",
    "### Rank\n",
    "\n",
    "The rank of a matrix A is the dimension of its column space - and - the dimension of its row space. These are equal for any matrix. Rank can be thought of as a measure of non-degeneracy of a system of linear equations, in that it is the dimension of the image of the linear transformation determined by A.\n",
    "\n",
    "### Transpose\n",
    "\n",
    "The transpose of a matrix is another operation that can be accomplished with `numpy`. Its importance isn't so obvious as first, but they're used to <HERE>. Now, let's take a look at an example. Given the matrix below, let's find what its transpose looks like. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2],\n",
       "       [3, 4]])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([[1, 2], [3, 4]])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the the 2 and 3 flipped positions. The transpose essentially flips the positions of a matrix in a particular way. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 3],\n",
       "       [2, 4]])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what does that look like when we have a matrix that's not 2x2 dimensions? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6]])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([[1, 2, 3], [4, 5, 6]])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of dimensions is now flipped as well. Instead of three columns, we now have two. And instead of two rows, we now have three. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 4],\n",
       "       [2, 5],\n",
       "       [3, 6]])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.0 Eigenvalues & Eigenvectors\n",
    "\n",
    "Let A be an `n x n` matrix. The number &lambda; is an eigenvalue of `A` if there exists a non-zero vector `C` such that\n",
    "\n",
    "Av = &lambda;v\n",
    "\n",
    "In this case, vector `v` is called an eigenvector of `A` corresponding to &lambda;. You can use numpy to calculate the eigenvalues and eigenvectors of a matrix: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.82842712475 -2.82842712475\n"
     ]
    }
   ],
   "source": [
    "eigenvecs, eigvals = np.linalg.eigvals(matrix1)\n",
    "print(eigenvecs, eigvals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to note that eigenvectors do not change direction in the transformation of the matrix.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge\n",
    "\n",
    "1. Given the matrix below, find the eigenvalues (name these variable `eig1` and `eig2`). For each eigenvalue find its eigenvector (call these variables `eigenvector1` and `eigenvector2`).\n",
    "\n",
    "``` \n",
    "    1    4\n",
    "    3    5\n",
    "```\n",
    "* Don't forget to create the matrix above.\n",
    "\n",
    "2. Consider the rotation matrix for two dimensions. (for example - we see that for zero degrees this matrix is just a 2-by-2 identity matrix.) Find the eigenvalues for a 45 degree rotation (call these variables `eig_rot1` and `eig_rot2`).  For each eigenvalue find its eigenvalue (call these variables `eigenvector_rot1` and `eigenvector_rot2`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}