02 - Linear Algebra.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Deep Learning Book (Simplified)\n",
    "## Part I - Applied Math and Machine Learning basics\n",
    "*This is a series of blog posts on the [Deep Learning book](http://deeplearningbook.org) where we  are attempting to provide a summary of each chapter highlighting the concepts that we found to be most important so that other people can use it as a starting point for reading the chapters, while adding further explanations on few areas that we found difficult to grasp. Please refer [this](http://www.deeplearningbook.org/contents/notation.html) for more clarity on notation.*\n",
    "\n",
    "## Chapter 2: Linear Algebra\n",
    "A good understanding of linear algebra is essential for understanding and working with many machine learning algorithms, especially deep learning algorithms. The chapter aims to teach you enough for understanding deep learning and hence, omits many key topics in Linear Algebra that are not relevant.\n",
    "\n",
    "The sections present in this chapter are listed below. Feel free to navigate as you like: <br>\n",
    "\n",
    "**1. Scalars, Vectors, Matrices and Tensors** <br>\n",
    "**2. Multiplying Matrices and Vectors** <br>\n",
    "**3. Identity and Inverse Matrices** <br>\n",
    "**4. Linear Dependence and Span** <br>\n",
    "**5. Norms** <br>\n",
    "**6. Special Kinds of Matrices and Vectors** <br>\n",
    "**7. Eigendecomposition** <br>\n",
    "**8. Singular Value Decomposition (SVD)** <br>\n",
    "**9. The Moore Penrose Pseudoinverse** <br>\n",
    "**10. The Trace Operator** <br>\n",
    "**11. The Determinant** <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Scalars, Vectors, Matrices and Tensors\n",
    "\n",
    "\n",
    "The study of linear algebra involves several types of mathematical objects:\n",
    "\n",
    "\n",
    "**Scalars**: A scalar is just a single number which may take up different types of values, e.g. real-valued (slope of a line), natural number (number of units), etc.\n",
    "\n",
    "\n",
    "**Vectors**: A vector is an array of numbers of the same type (e.g. $x$ ∈ ℝ), arranged in order and indexed as $x_1, x_2$, etc. They are expressed as:\n",
    "\n",
    "\n",
    "$$x= \\begin{bmatrix}x_1 \\\\ x_2 \\\\ . \\\\ . \\\\ . \\\\ x_n\\end{bmatrix}$$\n",
    "\n",
    "**Matrices**: A matrix is a 2-D array of elements, each element being indexed by two numbers. A real-valued matrix **A** of height *m* and width *y* is represented as $ A \\in ℝ ^ {mxn} $. The element in the $i^{th}$ row and $j^{th}$ column is indexed as $A_{i, j}$. $f(A)_{i, j}$ represents the element $(i, j)$ of the matrix computed by applying the function $f$ to $A$.\n",
    "\n",
    "\n",
    "**Tensors**: An array of numbers arranged in a regular grid with a variable number of axes is known as a tensor. The element at coordinates $(i, j, k)$ of a tensor $A$ is represented as $A_{i, j, k}$.\n",
    "\n",
    "An important operation on any matrix $A$ is the **transpose** (denoted by $A^T$) which is its mirror image across the [main diagonal](https://www.wikiwand.com/en/Main_diagonal). It can be better understood through this animation:\n",
    "![transpose gif](https://upload.wikimedia.org/wikipedia/commons/e/e4/Matrix_transpose.gif?1509056334524)\n",
    "\n",
    "A column vector of n-dimension can be thought of as a matrix of shape $NX1$. Thus, the transpose of a row-vector gives a column vector: $x = [x_1 \\hspace{.1cm} x_2 ... \\hspace{.1cm} x_n]^T$. We can add matrices of the same size ($C_{i, j} = A_{i,j} + B_{i,j}$) and add or multiply a matrix by a scalar ($D = a.B + c$). We can even add a vector to a matrix: $C = A + b$ where $C_{i, j} = A_{i, j} + b_{j}$, i.e. the vector $b$ is added to each row of $A$, an operation known as [broadcasting](http://deeplearning.net/software/theano/tutorial/broadcasting.html). The diagram below provides a visual explanation for broadcasting:  <br>\n",
    "![broadcasting](http://deeplearning.net/software/theano/_images/bcast.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Multiplying Matrices and Vectors\n",
    "Matrix multiplication is one of the important operations involving operations. For a matrix multiplication between two matrices $A_{mn}$ and $B_{kp}$ to exist, we must have $n == k$. The resulting matrix $C (= AB)$ has the shape $m$ x $p$. <br>\n",
    "**Element-wise product** is given by $A \\odot B$ and has the same shape as $A$ (and $B$). <br>\n",
    "**Dot-product** between two vectors $x$ and $y$ of the same dimension is a scalar given by $x^Ty$.\n",
    "\n",
    "Useful properties:\n",
    "\n",
    "1. $A(B+C) = AB + AC$ (Distributive)\n",
    "2. $A(BC) = (AB)C$    (Associative)\n",
    "3. $AB \\ne BA$ (not commutative, in general)\n",
    "4. $(AB)^T = B^TA^T$\n",
    "5. $x^Ty = (x^Ty)^T = y^Tx$\n",
    "\n",
    "System of linear equations:\n",
    "\n",
    "$  Ax = B \\tag{1} $\n",
    "\n",
    "where $A \\in ℝ^{mxn}$ & $b \\in ℝ^{m}$ are known and $x \\in ℝ^{n}$ is to be found. We have $m$ linear equations where the $i^{th}$ equation is given by: <br>\n",
    "\n",
    "$A_{i,1}x_1 + A_{i,2}x_2 + ... + A_{i,n}x_n = b_i$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Identity and Inverse Matrices\n",
    "\n",
    "**Identity Matrix** is a matrix which doesn't change a vector when multiplied by the vector. The entries along its main diagonal is 1. All other entries are zero.\n",
    "![Identity Matrix](https://wikimedia.org/api/rest_v1/media/math/render/svg/e1a4218ab6975ad1809415aa168ab6371b91bafc)\n",
    "\n",
    "Now we define the inverse of a matrix, $A^{-1}$ as:\n",
    "\n",
    "$A^{-1}A = AA^{-1} = I_n$\n",
    "\n",
    "We use this to solve the previously defined system of equations:\n",
    "\n",
    "$$  Ax = b \\\\ \n",
    "A^{-1}Ax = A^{-1}b \\\\\n",
    "I_nx = A^{-1}b \\\\ \n",
    "x = A^{-1}b \\\\\n",
    "$$\n",
    "\n",
    "The next section covers the conditions required for $A^{-1}$ to exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Linear Dependence and Span\n",
    "For $A^{-1}$ to exist, equation (1) must have exactly one solution for every real value of $b$. The system can have infinitely many solutions for some values of b, but can't have a finite number of solutions greater than one.\n",
    "\n",
    "**Linear Combination**\n",
    "\n",
    "We know:\n",
    "\n",
    "$$ Ax = \\begin{bmatrix} {A_{1,1}x_1 + A_{1, 2}x_2 + ... A_{1, n}x_n} \\\\ {A_{2,1}x_1 + A_{2, 2}x_2 + ... A_{2, n}x_n } \\\\ . \\\\ . \\\\ . \\\\ {A_{m,1}x_1 + A_{m, 2}x_2 + ... A_{m, n}x_n} \\end{bmatrix} $$\n",
    "\n",
    "This can now be written as:\n",
    "$$ \\begin{bmatrix} {A_{1,1}x_1 + A_{1, 2}x_2 + ... A_{1, n}x_n} \\\\ {A_{2,1}x_1 + A_{2, 2}x_2 + ... A_{2, n}x_n } \\\\ . \\\\ . \\\\ . \\\\ {A_{m,1}x_1 + A_{m, 2}x_2 + ... A_{m, n}x_n} \\end{bmatrix} = x_1 \\begin{bmatrix} A_{1,1} \\\\ A_{2,1}\\\\ . \\\\ . \\\\ . \\\\ A_{m,1}  \\end{bmatrix} + x_2 \\begin{bmatrix} A_{1,2} \\\\ A_{2,2}\\\\ . \\\\ . \\\\ . \\\\ A_{m,2}  \\end{bmatrix} + ... x_n \\begin{bmatrix} A_{1,n} \\\\ A_{2,n}\\\\ . \\\\ . \\\\ . \\\\ A_{m,n}  \\end{bmatrix} = \\sum_{i=1}^{n} x_iA_{:, i}$$ <br> where $A_{:, i}$ is the $i^{th}$ column of $A$. Thus:\n",
    "$$ Ax = \\sum_{i=1}^{n} x_iA_{:, i}  $$\n",
    "\n",
    "This kind of operation is called a linear combination.\n",
    "\n",
    "**Span** of a set of vectors is the set of all points that can be obtained from the linear combination of the vectors. The span of the columns of $A$ is called the column space or range of $A$. For equation (1) to have a solution, $b$ must belong to this column space. Hence, in order for the system to have a solution for all $b ∈ ℝ^m$, the column space of $A$ must be all of $ℝ^m$. Thus, the number of columns (n) should be >= m. This is a necessary condition, but not a sufficient one as we might have redundant columns and that doesn't add to the column space. Here, the concept of **linear dependency** is introduced. A set of vectors are **linearly independent** if none of the vectors is a linear combination of the other vectors. Thus, for the column space of $A$ to be all of $ℝ^m$, it must contain atleast one *set* of m linearly independent columns. The system must have  *at most* one solution for each value of $b$, hence, $A$ must have at most m columns, or else there are multiple ways of parameterizing each solution. But we started this section stating that we need exactly one solution for $A^{-1}$ to exist. Hence, m = n, i.e., the matrix must be **square** and its columns are linearly independent for its inverse to exist. Such a matrix is said to be **singular**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Norms\n",
    "\n",
    "The size of a vector is given by functions called norms. We can have different types of norms, more commonly used, is the $L^p$ norm given by:\n",
    "$$ ||\\mathbf{x}||_p = (\\sum_{i} |x_i|^p)^{\\frac{1}{p}} $$\n",
    "\n",
    "A norm function $f$ satisfies the following properties:\n",
    "\n",
    "- $f(x) = 0 \\Rightarrow x = 0$\n",
    "- $f(x+y) \\leq f(x) + f(y)$ (The **triangle inequality**)\n",
    "- $\\forall \\alpha \\in ℝ, \\hspace{.1cm} f(\\alpha x) = |\\alpha|f(x)$\n",
    "\n",
    "Different types of norms:\n",
    "\n",
    "- **Euclidean Norm**: This is the $L^2$ norm, which is heavily used in machine learning, and can be also calculated as $x^Tx$.\n",
    "- **L1 Norm**: It is used when the difference between the zero and non-zero elements is very important.\n",
    "- **Max Norm (also known as the $L^{\\infty}$ norm)**: Absolute value of the largest magnitude in the vector:\n",
    "$||x||_{\\infty} = \\displaystyle \\max_{i}|x_i|$\n",
    "- **Frobenius Norm**: Used to measure the size of a matrix (similar to the $L^2$ norm):\n",
    "$||A||_F = \\sqrt{\\displaystyle \\sum_{i,j} A_{i,j}^2} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Special Kinds of Matrices and Vectors\n",
    "\n",
    "- **Diagonal Matrices**: These matrices have non-zero entries only along the main diagonal, e.g. the identity matrix $I_n$. Some of the key features are: <br>\n",
    " - A square diagonal matrix can be represented as: $diag(v)$ where  the vector $v$ represents the elements along tha main diagonal.\n",
    " - Multiplying by a diagonal matrix is computationally efficient. $Dx$ can be calculated by simply scaling each $x_i$ by $v_i$.\n",
    " - A diagonal matrix need not be square.\n",
    " \n",
    "- **Symmetric Matrix**: $A = A^T$\n",
    "- **Unit vector**: A vector which has **unit norm**, i.e. $||x||_2 = 1$.\n",
    "- **Orthogonal vectors**: Two vectors $x$ and $y$ are orthogonal if $x^Ty = 0$, which means that if both of them have non-zero norm, these vectors are at a 90 degree angle to each other. Orthogonal vectors having unit norm are called **orthonormal vectors**.\n",
    "- **Orthogonal Matrix**: A matrix whose rows are mutually orthonormal (and columns too). Thus:\n",
    "$$ A^TA = AA^T = I \\Rightarrow A^{-1} = A^T $$\n",
    "For orthogonal matrices, the inverse is cheap to compute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Eigendecomposition\n",
    "Matrices can be understood better by breaking them into constituent parts (e.g. breaking a number like 12 into its prime factors: 12 = 2 x 2 x 3) which are universal and not obvious from their representation. One of the widely used matrix decomposition is called **eigendecomposition**, where we decompose a matrix into its **eigenvectors** and **eigenvalues**.\n",
    "\n",
    "An **eigenvector** $v$ of a matrix $A$, is a non-zero vector satisfying the following equation:\n",
    "\n",
    "$$ Av = \\lambda v $$\n",
    "\n",
    "The scalar $\\lambda$ is called the **eigenvalue** corresponding to the eigenvector. Any scaled version of an eigenvector is also an eigenvector with the same eigenvalue, hence we focus on only unit eigenvectors.\n",
    "\n",
    "The eigendecomposition of a matrix $A$, having $n$ linearly independent eigenvectors represented as a matrix $V = [v^{(1)}, ... , v^{(n)}]$ with the corresponding eigenvalues given by the vector $\\lambda = [\\lambda_1, ... , \\lambda_n]$ is given by:\n",
    "\n",
    "$$ A = Vdiag(\\lambda)V^{-1} $$\n",
    "\n",
    "For our  purpose, in the case of $A$ being real-valued, $V$ is an orthogonal matrix. $A$ can be thought of as scaling space by $\\lambda_i$ in the direction $v^{(i)}$. We generally sort the entries of $\\lambda$ in decreasing order. Two types of matrices are specially useful:\n",
    "\n",
    "- **Positive definite**: All eigenvalues are positive and $x^TAx = 0 \\Rightarrow x = 0$.\n",
    "- **Positive semidefinite**: All eigenvalues are non-zero and $\\forall x, \\hspace{0.1cm} x^TAx \\geq 0$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2], [4, 3]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "w, v = np.linalg.eig(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize(u):\n",
    "    for i, x in enumerate(u):\n",
    "        norm = np.sqrt(sum([val**2 for val in x]))\n",
    "        for j, y in enumerate(x):\n",
    "            u[i][j] = y / norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "normalize(v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_values = np.linspace(-1, 1, 1000)\n",
    "y_values = np.array([np.sqrt(1 - (x**2)) for x in x_values])\n",
    "\n",
    "x_values = np.concatenate([x_values, x_values])\n",
    "y_values = np.concatenate([y_values, -y_values])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = np.array([[x, y] for x, y in zip(x_values, y_values)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "trans = np.dot(A, u.T).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgwAAAFkCAYAAABMyWOlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XecVOXZ//HPBdJEgyYGsMWCUZAOgoIKiho1UVCwrZqA\nGsujooEnMT81UWNNMWJsMZggsWRjYcVoEkksixpRmqCjGGPEFmWwPRiRonD//jgzOrt7zpTdOWVm\nvu/X67xg7jln5mKZmb3mLtdtzjlERERE8mkXdwAiIiKSfEoYREREpCAlDCIiIlKQEgYREREpSAmD\niIiIFKSEQURERApSwiAiIiIFKWEQERGRgpQwiIiISEFKGERERKSgUBMGMzvfzOab2Udmljaz+8xs\n1yKuO9rMlpnZGjNbamaHhhmniIiI5Bd2D8O+wPXAnsCBQAfgb2bWJegCMxsB/AG4BRgEzAZmm9nu\nIccqIiIiASzKzafMbCtgJTDKOfdkwDl/BDZ1zo3NaZsHPOucOzOaSEVERCRX1HMYtgAc8EGec0YA\nDzdrm5NpFxERkRhsEtUTmZkB1wJPOudezHNqTyDdrC2dafd73K8ABwOvAWvbHqlITesM7AjMcc69\nH3MsgfS+Fymbot/zkSUMwE3A7sDerbjW8Hom/BwM3NnaoETE1wl4c4mSSu97kfIq+J6PJGEwsxuA\nbwL7OufeKXD6CqBHs7butOx1yHoN4I477qBPnz5tCbNoU6ZMYdq0aZE8V6kUW+skNbao41q2bBkn\nnngiZN5XCfYalP99n9TXQXOKs/wqJdZyx1nKez70hCGTLIwDRjvn3ijiknnAAcB1OW0HZdr9rAXo\n06cPQ4YMaUuoRevWrVtkz1UqxdY6SY0txriS3s0fyvs+qa+D5hRn+VVKrCHGWfA9H2rCYGY3AXXA\nWGC1mWV7DlY559Zmzvk98B/n3AWZ+34FzDWzqcCfM9cPBU4NM1YREREJFvYqiTOALwGNwNs5xzE5\n52xPzoRG59w8vCThNGAJMB4YV2CipIiIiIQo1B4G51zBhMQ5N8anbRYwK5SgREREpGTaS6IV6urq\n4g4hkGJrnaTGltS4qlWl/LwVZ/lVSqxxxhlppccwmNkQYNGiRYsqYsKKSJItXryYoUOHAgx1zi2O\nO54get+LlEcp73n1MIiIiEhBShhERESkICUMIiIiUpASBhERESlICYOIiIgUpIRBREREClLCICIi\nIgUpYRAREZGClDCIiIhIQUoYREREpCAlDCIiIlKQEgYREREpSAmDiIiIFKSEQURERApSwiAiIiIF\nKWEQkdiZ2TZmdruZvWdmn5jZUjMbEndcIvKFTeIOQERqm5ltAfwDeAQ4GHgP+DrwYZxxiUhTShhE\nJG7/D3jDOffdnLbX4wpGRPxpSEJE4nY4sNDM7jaztJktNrPvFrxKRCKlhEFE4rYz8D/AP4FvADcD\n15nZibFGJSJNaEhCROLWDpjvnPtx5vZSM+uLl0TcEV9YIpJLCYOIxO0dYFmztmXA+EIXTpkyhW7d\nujVpq6uro66urnzRiVSJ+vp66uvrm7StWrWq6OuVMIhI3P4B7NasbTeKmPg4bdo0hgzR6kuRYvgl\n04sXL2bo0KFFXa85DCISt2nAXmZ2vpn1MrPjge8CN8Qcl4jkUMIgIrFyzi0EjgTqgOeBC4FznXN/\njDUwEWlCQxIiEjvn3F+Av8Qdh4gEC7WHwcz2NbM/mdl/zGyjmY0tcP7ozHm5xwYz6x5mnCIiIpJf\n2EMSXYElwFmAK/Iah1cWtmfm2No5tzKc8ERERKQYoQ5JOOceAh4CMDMr4dJ3nXMfhROViIiIlCqJ\nkx4NWGJmb5vZ38xsZNwBiYiI1LqkJQzvAKcDE/CKtrwJNJrZoFijEhERqXGJWiXhnHsZeDmn6Wkz\n6wVMASbmu1YV30RK09aqbyJSWxKVMASYD+xd6CRVfBMpTVurvolIbUnakISfQXhDFSIiIhKTUHsY\nzKwrsAveREaAnc1sIPCBc+5NM7sK2MY5NzFz/rnAcuAFoDNwKrA/cFCYcYqIiEh+YQ9J7AE8hldb\nwQG/zLT/HjgZr87C9jnnd8ycsw3wCfAccIBz7vGQ4xQREZE8wq7DMJc8wx7OuZOa3f4F8IswYxIR\nEZHSVcIcBhEREYmZEgYREREpSAmDiIiIFKSEQUREasqMGWCW/zj11LijTJ5KKNwkIiLSZqkUDB0K\n69cXPve3v/X+vOWWcGOqJOphEBGRqtfQAP37F5csZGWTBvEoYRARkarW0AATJrTu2oNUNvBzShhE\nRKRqtSVZAHj44fLFUumUMIiISFUqNlmYObNtSUWtUMIgIiJVp7GxcBJw3XXgHEycCPfeC506RRJa\nxVLCICIiVaWxEfbfP/85c+fC5MlN2954w//cE08sS1gVTwmDiIhUjVQqf7Jg5p0zalTL+7p397/m\nzjvLE1ulUx0GERGpCuk0DBwYfH+XLvDaa8GJgeSnHgYREal46TT06gUbN/rfv+mmxSULmvwYTAmD\niIhUvMMPh9Wr/e/bdFNYvry4noV77y1vXNVECYOIiFS0VAoWLPC/r2vX4pMFyU8Jg4iIVKx0GgYN\n8r+vXTt49VUlC+WihEFERCpSOg277AIbNrS8r107eO45JQvlpFUSIiJSkSZMgI8/btnevj28/baS\nhXJTD4OIiFScdBoWLvS/b+nS1icL6XTrY6p2ShhEJFHM7Hwz22hm18QdiyRTOg1f/zqsW9fyvuHD\noW/f1j/20KGtv7baKWEQkcQws2HAqcDSuGOR5JowAf7735btw4bBAw+07bH/85+2XV/NlDCISCKY\n2WbAHcB3gf+LORxJsHfeadm2994wf77mLYRJCYOIJMWNwAPOuUfjDkSSbeutm97efHNvK+uwnH9+\neI9dSbRKQkRiZ2bHAYOAPeKORZIrlYKRI2HNGm8lxNZbww47eMlCmD0LV14Z3mNXEiUMIhIrM9sO\nuBY4yDn3adzxSHKNHNl07sKqVfDkk/HFU2uUMIhI3IYCXwUWmZll2toDo8zsbKCTc875XThlyhS6\ndevWpK2uro66urow45WYrFmT/7bkV19fT319fZO2VatWFX29EgYRidvDQP9mbTOBZcBPg5IFgGnT\npjFkyJAQQ5Mk6dKlaQ9Dly7lffxUqryPlzR+yfTixYsZWuRa0lAnPZrZvmb2JzP7T2Zd9dgirtnP\nzBaZ2Voze9nMJoYZo4jEyzm32jn3Yu4BrAbed84tizs+iVc6Dfvs421dvcsusNlmsMkm3kTHefPK\n+1yqwZBf2D0MXYElwAxgVqGTzWxH4EHgJuB44EDgt2b2tnPu7+GFKSIJE9irILVlwgT4xz++uL33\n3uHNW1i/PpzHrRahJgzOuYeAhwByxibz+R/gVefceZnb/zSzfYApgBIGkRrhnBsTdwwSP7/yz341\nGCQaSavDsBfeeGauOcCIGGKRIkybBmblP+6/P+5/mYjEKaj8c/MaDGG77rpony/JkpYw9ASab/2R\nBr5kZp1iiEfw3rg77uj/i33q1HCe84gj/J/v+uvDeT4RSRa/8s+dOoVboMnP5MnRPl+SVcIqiexQ\nRt4xTS2vKp9UCoYMgU8TuCL+nHO8I+uKK+CCC+KLp5K1dYmVSFiCdqLcY4/wCjRFnYhUoqQlDCuA\nHs3augMfOefyTkfR8qq2mTYtvN6CMF14oXdkzZ4N48bFF08laesSK5EwBA1FhF3+ecKE8B67WiRt\nSGIecECztm9k2qXMcucfVGKy4Cd3KEPzIEQqz9ix/kMRr7yijaXiFnYdhq5mNtDMBmWads7c3j5z\n/1Vm9vucS24GepnZz8xsNzM7EzgKuCbMOGtJY2N4ScIVV4BzrTtSKejQobzxZJOHHXaAlSvL+9gi\nUn7pNCxY0LI9zKEIKV7YPQx7AM8Ci/DmIPwSWAz8JHN/T2D77MnOudeAb+HVX1iCt5zyFOdc85UT\nUqIZM7xfnvvv3/bHmjnT/5d+W+YS9O3rrYFu/pjptPcLvy3eeAN69FCvg0iSpdNeYabmdT3N4ptf\nMHduPM+bVKEmDM65uc65ds659s2OkzP3n9R8vXXmmqHOuS7Oua87524PM8Zqlx12OOWU1j/G7NlN\nf4lPjLD2Zvfu8NprTZ+/Lcucsr0Ov/994XNFJDpjx8LHH7dsj6J3YcYM//ZRo8J93kqTtDkMUiaX\nXtr6YYeddvKy/ewv6KRNIpw8ue1DGZMmKXEQSYqgoQgzePDB8J+/LV+oaokShiqT7VG4+OLSrstN\nEl59tXLGC3OHMlrTfZhNHDRUIRKfQw9tORQBmruQNEoYqkR2jkIpPQpm3i/ZSksSgowa9UXPw8yZ\npV17xBHQrh08/ngooYlIgHQann22Zftmm0XTuxBku+3ie+6kUsJQ4VIpb8lRKV1qX/ua9ybduLF6\nx+gmTiy918E5GD0arrwyvLhEpKmxAXsY//vf0XyJaWz0b1+0KPznrjRKGCrYtGnQv3/xO6xdd533\nS/H11yu/N6FY2V6HVAo6dizumgsv1PwGkSikUjB/fsv2rl2j+4waE7DNWa18RpZCCUMFytZSKHb4\nIZso1HJN9L59vcpxpSQOkybBzjurhoNIGNJpGDjQ/75nnokuDr+5E+JPCUOFmTGj+FoK2YmMtZwo\nNJdNHIodqli+3KvhoLkNIuU1dqw3LNrcsGHe+1SSRwlDhUinYfvti5ur0KmT9026GiYyhiU7VFFs\n4jB6tHbKFCmXoGWUEO1Ex6D5CyrY5E8JQwVoaICePeGttwqfO3MmrF2rDL1Y2cShmGJQ55wDnTvD\nCy+EH5dINQtaRjloULRfcoLmL1TrZPC2UsKQcDNmFLeLWnaeQpRVGKvJ5MnFzW9Ytw769VPdBpHW\nSqX8l1GawZw50cai+QulUcKQYNOmFTcEMXu25imUQ3Z+QzG9DUccoaRBpDVGjPBvV5Gm5FPCkEDp\nNOy4Y+FVENl6Ckkr3VzpJk8ubgzziCM0r0GkFKmU/34R7dpFX6QpaEMrzV8IpoQhYVIp2Hprr1ZC\nPjNn1lY9haiNGlXcTpnnnKOkQaRYe+3l3/7cc9F/lgUN9Wr+QjAlDAmSSnmFmAqNq82dq7kKUcju\nlDl7dv7zzjlH1SFFCkmlYPXqlu2DBmmSdqVQwpAQ2WQhn06dvG+9yoCjNW5c4W7KCy9UT4NIPkG9\nC1FPdMyn2KJutUoJQwIUkyzstBO88YaGIOIyapT3/2QWfI56GkT8BfUubLppPJ9pM2b4ty9eHG0c\nlUYJQ8yKSRZmzlQRpiTo2xdWrMg/r0E9DSItBfUu+O0jEYWg1WcaGslPCUOM0mkYMCD/OTNnar5C\nkmTnNVxxRfA5mggp8oXGxuDeBf2CrixKGGK0zz75JzjOnq1kIakuuCB/vYZzztFulyLpdPDeN3H1\nLqRS/u29ekUbRyVSwhCTadPglVeC7585U/UVkm7y5Pw9DZMmqbiT1LZDD/Vvj7N3Ydgw//annoo2\njkqkhCEGDQ35izKpZ6FyFOppOOII7XQptWvJEv/2uHoXwNtrx4/miBWmhCFijY3594ZQz0LlmTw5\nf9IwejSsXBldPCJJ0NjoP+Tat6/mLlQqJQwRSqWCx/NAExwr2eTJ3k6WQUaOjC4Wkbjlm7vw6KPR\nxpIraDmlykEXRwlDhILGzsBbqqdkobItXBh837//rUmQUjvyzV2Is+s/aDmliuEVRwlDRGbMCB47\nM4t3TE/Ko2/f4BnY4E2C1HwGf2Z2vpnNN7OPzCxtZveZ2a5xxyWlS6f9t68Gfc5VOiUMEWhszL9N\n9fPPa8JNtejbN//eE5rPEGhf4HpgT+BAoAPwNzPrEmtUUrKg3oW45y40Nvq3z5wZZRSVTQlDyPKN\n5YE3dqYJQNVl3Lj8H0Kaz9CSc+6bzrnbnXPLnHPPA5OArwFD441MSpGvdyHOuQsAY8b4t2souHiR\nJAxmdpaZLTezNWb2tJkFjuab2UQz22hmGzJ/bjSzT6KIMwz77Rd838yZGjurVhMnBk+C/Pe/NTRR\nhC0AB3wQdyBSvKTOXYDCuwBLYaEnDGZ2LPBL4GJgMLAUmGNmW+W5bBXQM+fIU70/uRob4aWX/O/r\n1UuZbbXLNwlSQxPBzMyAa4EnnXMvxh2PFCfJcxeC5hZ16hRtHJVukwieYwrwG+fcbQBmdgbwLeBk\n4OcB1zjn3LsRxBaqAw4Ivk9Vxapf377ekNPo0f73jxoVnFDWuJuA3YG9C504ZcoUunXr1qStrq6O\nurq6kEKTIEms6pgVtEJt0aJo44hbfX099fX1TdpWrVpV9PWhJgxm1gFvDPLzTX+dc87MHgZG5Ll0\nMzN7Da8HZDFwQaV902hogI0b/e+bPTv+7jmJxqhR8PWvw7/+1fK+f/7TG5rQsNQXzOwG4JvAvs65\ndwqdP23aNIYMGRJ+YFJQEqs6ZgWtUIs7kYmaXzK9ePFihg4tbqpQ2EMSWwHtgXSz9jTeUIOff+L1\nPowFTsCL8Skz2zasIMstlQqu5rjbbqrkWGuefDL4vqDeh1qUSRbGAfs7596IOx4pXlBVxyT0LqSb\n//aRVotrlYThTWhqwTn3tHPuDufcc865J4DxwLvAaVEG2BZ77hl8nya71Z7u3fNXktNrAszsJrwv\nCMcDq82sR+bIUz9TkiCJO1Lm2mcf/3ZVdyxd2HMY3gM2AD2atXenZa+DL+fcZ2b2LLBLvvOSMpaZ\nSsEnAWs6NBRRu0aNgt69/ecsjB7tfehG/dpo63hmmZ2B9yWisVn7ScBtkUcjRQuau2AWf+8CBO8K\nrKHA0oWaMDjnPjWzRcABwJ/g8xnQBwB5tuv5gpm1A/oBf8l3XlLGMoN6Fzp31lBErZs7F3o0T50z\nxozJXyUyDG0dzywn55xqwlSooLkLQYWSoqThiPKK4k16DXCamX3HzHoDNwObAjMBzOw2M/t8UqSZ\n/djMDjKzncxsMHAn3rLK30YQa5vk613It8ROakP37sFVIF94IdpYRMohaO5Cly7J+AYfVAcnXzVW\nCRb6skrn3N2ZmguX4g1NLAEOzlk2uR3wWc4lWwLT8SZFfggsAkY45xK/AC2od6FLl2R0zUn8xo2D\ndu38V9BoxYRUmqCl4wsWRBtHkKBly+rtbZ0o6jDgnLsJb221331jmt2eCkyNIq5yyte7kJQ3jyTD\nY4/5r44YPVrV6KRyNDb6J75Jmbug4Yjy07hhmQT1LsS94Yokz6hRXi+DH62YkEoR1LuQhLkLELw6\nQsMRraeEoQzS6eDehbg3XJFkeuwx/3bVZZBKkEoFF6ZLyrBa0OoIDUe0nhKGMsi3nauWUYqffL0M\n2mNCki6oRzUptQ00HBEOJQxlELSsSL0Lkk9QL0PQNrwiSZCvRzUpvQtaHREOJQxtlEoFLytS74Lk\nE/ThqiWWkmQHHujfPmhQtHHko9UR4VDC0EZBXXNaGSHF6NfPv12THyWpggqMzZkTbRxBNBwRHiUM\nbeTXNZeUZUWSfI884t8e1KUqEqegFRBJ6lHV6ojwKGFog6BMOynLiiT5unf3dvRrTvUYJGnybTKV\npB5VrY4IjxKGNthrL//2pEz8kcoQtKOf5jJIkgTNXYDk9Kjqy1q4lDC0werVLdu6dIk+DqlsQR+2\nw4dHG4dIPkE9qklZSgnBxaSSFGMlU8LQSkFvniR1zUnl6Nq1ZVvQ0jWRqOXbSTVJPapJLyZV6ZQw\ntFLQcERSuuaksjzzjH+7hiUkCZJeqAmChyM6d440jKqmhKGV/IYj/CaviRQjKNEM+qAWiUq+jfWS\n9M09aDhi4cJo46hmShjKKGjymkgx/IYl/BJTkSjl21gvSYKGI5IWZyVTwtAKQYVB9MKUtggalhCJ\nS6VsrNfQ4N/eq1e0cVQ7JQytELTZlEhbBCWcmscgcQlaSpmkQk0AEyb4tz/1VLRxVDslDK3gt9mU\nX3eySKnMWrZpHoPEpRJWg+UrBZ2kpKYaKGFoBb8qfOpOlnLw28BH8xgkDvnKQCdp+DWojPpuu0Ua\nRk1QwlAmSXoDSeV66KG4IxDxBK06SFLvAgTvTKkN3MpPCUOJ8hUwEWkrdaFKEqRSlbHqIF8paL2X\nyk8JQ4lGjmzZ1rFj9HGIiIQlqCx50nZ8VCnoaClhKNGaNS3bBg6MPg6pLStXxh2B1Ip02v9zDpK3\n46NKQUdLCUOJ/F6gDz4YfRxSvTbbrGXbYYdFH4fUpkqZRBhUe0GloMOjhKFE7ds3vd2hg8bKpLye\nfrpl29Kl0cchtalSJhEG1V5QKejwKGEo0Wef5b8t0lZ9+7asx/Dpp/HEIrUl3wZOSfpilG/yeZIm\nZVYbJQwl6tAh/22RctDrTOJQKRs4VcqkzGqjhKFE+iCXKDRfeaOVOBK2dLoyllJC5UzKrDZKGEq0\n5Zb5b4uUwxZb5L8tUm5Bkx2TlizkGzaRcEWSMJjZWWa23MzWmNnTZjaswPlHm9myzPlLzSwx2z2t\nWpX/tkg51OLrrNTPCSmvoMmOSdqVEipn2KQahZ4wmNmxwC+Bi4HBwFJgjpltFXD+COAPwC3AIGA2\nMNvMdg871mJ8+cv5b4uUQ629zkr9nJDyqpSKiZU0bFKNouhhmAL8xjl3m3PuJeAM4BPg5IDzzwX+\n6py7xjn3T+fcxcBi4OwIYi1ou+3y3xYphxp8nZX6OSFlVCkVEyulRkS1CjVhMLMOwFDgkWybc84B\nDwMjAi4bkbk/15w850cmlfpiPbwZDB4cXDxEpC0uv/yLmh/t28OVV8YbT5ha+TkhZZJv34ikVUys\nlBoR1SrsHoatgPZA8x3L00DPgGt6lnh+ZEaOhI8/9v7uHLzySrK665Js1apVOL99wcXX2LGwYYP3\n9w0bqr7SY2s+J6RM9tzTvz1pXfyVMmxSzTaJ6XkNKOW3R8Hzp0yZQrdu3Zq01dXVUVdXV3p0AZov\n5Qla2iNfePfdd5k6dSqPPPII2267LX//+9/ZQlP+C4ritVZfX099fX2TtlXJml2ZiPd9tfvkE//2\nSpnsmLRhkyRr83veORfaAXQAPgXGNmufCdwXcM3rwDnN2i4Bng04fwjgFi1a5MK2+ebOeX0L3rH5\n5qE/ZcXauHGjmzlzpvvyl7/sAHfAAQc4wB122GFuw4YNcYeXeHG91hYtWuTwfkkPcSF+Nrim7+HW\nfE5E9r6vZo891vR1lj06d447sqZWrPCP0xu8krYo5T0f6pCEc+5TYBHweW5oZpa5/VTAZfNyz884\nKNMeq3nzYPPNYZNNvD/nxR5RMr3yyisceOCBTJo0ia222orGxkYefvhhzj//fB588EEuu+yyuENM\nvFp6rbXyc0LKoFKWKGqyYzJEsUriGuA0M/uOmfUGbgY2xfv2gJndZma5U7p+BRxqZlPNbDczuwRv\nQtQNEcSaV9++8K9/eWN+X/0qnH66th3O9emnn3LVVVfRv39/nnjiCS666CKWLl3K6NGjAbjssss4\n6KCDuOSSS3hQW3zmtdVWMGAAfO1r3p9f/WrcEYUu7+eElF++yY5Jm7+gyY4JUagLohwHcCbwGrAG\nr6dgj5z7HgVmNDt/AvBS5vzngIPzPHakXZN77920O2zvvSN52sSbN2+e69evnwPc3nvv7V544QXf\n89577z234447um7durmXX3454igrR1yvsziGJLJHvs8Jn3M1JNFGXbr4d/HvtlvckTUVNGyi4Yjy\nSMyQRJZz7ibn3I7OuS7OuRHOuYU5941xzp3c7PxZzrnemfMHOOfmRBFnMd58s+ntt96KJ46k+Oij\njzj77LMZOXIkb7zxBjfffDOPP/44u+/uX2frK1/5Cg0NDaxbt47x48fzcXbZiTTR/HVVC6+zfJ8T\nUn5BE2mT9q1dkx2TQ3tJlOjDD5ve/uCDeOJIgtmzZ7P77rtz4403MmHCBJYtW8bpp59Ou3b5X1aD\nBw9m+vTppFIpTjnllOw3RsnR/HVVy68zKb9K2cY6X2XHpNWIqAVKGErUbAVXi9u14D//+Q/jx4/n\nyCOPxMy4//77ueeee9hmm22Kfoxvf/vbnH322dx9991cc801IUZbmfQ6kzBV+mTHpM2xqBVKGErU\nvIeh+e1qtnHjRm666Sb69OnD7NmzOffcc3nxxRcZO3Zsqx7vl7/8Jfvssw/nnXcejyZt0XfM/u//\n8t8Waa1K2o+hUjbEqhVKGEr06af5b1erVCrFPvvsw1lnncVOO+3E008/zbXXXsvmm2/e6sfs2LEj\n99xzDz169ODYY4/ljTfeKGPElW39+vy3RVqrUpYoVsqwSS1RwlCiWvsgX7t2LT/60Y8YPHgwzz77\nLD/72c9YuHAhw4cPL8vj9+zZk1mzZrFq1SomTJjA2rVry/K4lSydbvm6qpXEVMJXKUsUK2XYpJYo\nYZBAjz32GAMGDOCKK65g//33J5VKcd5559GhQ4eyPs+IESP41a9+xcKFCznrrLNqfhKk3whPmX/k\nUqNSqeD7kvStvZJqRNQSJQwlMmvZVm3Fm95//31OOukkxowZw4cffsjtt9/OnDlz6NWrV2jPecYZ\nZzBp0iRmzJjB9OnTQ3ueSpDdETXXwIHRxyHVJ6hjMGlLFCtlQ6xao4ShRB07tmyrlp0EnXPceeed\n9O7dm5kzZzJx4kSWLVvGiSeeiPllSmVkZtx0000MHTqUyZMnM6+aayEX4DfMpcKYUg5BtReStEQx\nna6cDbFqjRKGEvl90/P7RlhpXn31VQ455BBOPPFEunXrxsMPP8zMmTPZaqutIouhS5cuzJo1iy99\n6UscddRRrFixIrLnThK/EZkkdRdLZco3iTBJDjzQv71LF70P4qaEoUQPPNCyrZInPn722Wf84he/\noF+/fjz66KOcf/75PP/88xwQNOMoZDvssAN//OMfWbFiBccccwyfarafSFmMGePfnrRJhEHzLBYs\niDYOaUkJQ4mqKcNdsGABw4YN47zzzmPAgAEsXryYK6+8ki5dusQa14EHHshPf/pTnnjiCb7//e/H\nGkvU0um4I5BqlE7791xBsuYFBPWCQLLirFVKGMrkhRfijqB4H3/8MVOmTGGvvfbi3//+NzfccAP/\n+Mc/6N8jlAB1AAAgAElEQVS/f9yhfe773/8+Rx99NNdddx133HFH3OFE5tBD445AqlGl1F7QvhHJ\npoShFfzm/5WpLEHo/vznP7P77rtz7bXXcvjhh/Piiy9y1lln0b59+7hDa8LMmDFjBrvvvjunnXYa\nS5YsiTukSPj9M7t2jT4OqS6VUHsh31LKJE3KrGVKGFph0KCWbUGzepNixYoVHHvssRx22GFs2LCB\nhoYGZs+ezXbbbRd3aIE222wz7rvvPjp06MD48eP5oAZ2YPLrNn7mmejjkOpRKbUXtJQy+ZQwtMJD\nD8UdQfE2btzI9OnT6d27N/fccw9nnnkmL774IkceeWTcoRVl11135fbbb2f58uUcf/zxbNiwIe6Q\nQhP0wa4PTGmLSqi9oKWUlUEJQysEZeVJm8ewbNkyRo8ezemnn852223Hk08+yY033ki3Ctv6cOzY\nsVx00UXMmTOHiy66KO5wQrPXXnFHINWoEmov5NuVMkm9ILVOCUMZJWUew7p167jkkksYOHAgCxYs\n4PLLL2fx4sWMHDky7tBa7eKLL+ab3/wmV155Jffdd1/c4YRi9eqWbX7DXyLFCuq16tQp2jjySae1\nK2WlUMLQSptt1rItCfMYHn/8cQYNGsRPfvIT9tlnH5577jkuvPBCOvqVqKwg7dq144477qBXr15M\nnDiRl4I+YSpU0Af7nDnRxiHVZdgw//ZFi6KNIx8VaqocShha6emn/dvjmnX84YcfctpppzF69GhW\nrlzJrbfeyiOPPMKuu+4aT0Ah2HLLLWloaGDDhg0ceeSRfPTRR3GHVDZBE770gSltEbT5a5LmxahQ\nU+VQwtBKQW+4oLG4sDjnuOuuu+jTpw+33HILJ5xwAsuWLWPSpEmh7/8QhwEDBvC73/2Ol156iUmT\nJlXNzpZ+vVObbhp9HFI9Ghr825M0HJGvXHWSkhrxKGFoA7/18VH+/nr99dc5/PDDOe644+jSpQsP\nPfQQd9xxB92r/Gvpcccdx9SpU7nvvvv46U9/Gnc4bRb0DWv+/GjjkOoyYYJ/e5KGI4IKNSWtXLV4\nlDC0QdD6+LCHJTZs2MC1115L3759eeihh/jBD35AKpXi4IMPDveJE+RnP/sZ++23HxdeeCF/+9vf\n4g6nTbT+XKKUlNdVvkJNSYlRmlLC0AZBL+rRo8N7zmeffZY999yTKVOm0Lt3bxYsWMDPf/5zutZY\nOcBNNtmEu+66i2233Za6ujqWL18ed0itkkr5D0fEvJ2HVLig4YgklYIOSpSTVB9CmlLC0EZBv6fL\nXZNh9erV/OAHP2DYsGG89NJLTJs2jWeeeYbBgweX94kqSPfu3Zk1axYff/wx48eP55MkLFMpUVDt\nBU34krYIGo5ISinooEQZklUfQppSwtBGQcMSQcuZWmPOnDn069ePq6++mkMOOYQXXniB733ve4nb\n/yEOw4cP58Ybb2TJkiWcccYZFTcJ0q/2AqhLVsKRlOlNGoarTEoY2qhvX2jn81NcswZWrmzbY69c\nuZITTjiBQw45hDVr1nDXXXfxwAMPsMMOO7TtgavMd7/7XU499VRuv/12brjhhrjDKVrQDHF9aEpb\nBA1HzJwZaRiBVAa6cilhKIPHHvNvHzOmdY/nnOPWW2+ld+/e/OEPf+C0005j2bJlHHPMMVW5VLIc\nrr/+eoYPH87UqVN54okn4g6nKEEzxPWhKW0RNBwxcWK0cQQJ2sJdZaCTTwlDGQSNub3wQulzGV5+\n+WXGjBnDySefTI8ePXj88cf5zW9+w5Zbbtn2QKtYp06dmDVrFl/+8pc5+uijefvtt+MOKa+GBv8Z\n4mb60JTqFrRTvRLl5As1YTCzLc3sTjNbZWYfmtlvzSzvdH4zazSzjTnHBjO7Kcw4yyGoyE6xcxnW\nr1/P5ZdfzoABA3jqqae45JJLWLJkCfvuu2/5gqxy2223HXfffTfvvfceRx11FOvXr487pEBB3wKD\nhilEijFjhn97UoYjGhv9a9WoDHRlCLuH4Q9AH+AA4FvAKOA3Ba5xwHSgB9AT2Bo4L8QYyyKoyM6a\nNf69DB988MHnf3/qqacYMmQIP/7xjxk+fDhLlizh4osvplOSSrJViNGjR3P11Vczb948vve978Ud\njq98SYFmiEtbnHKKf3tShiOChuG0KqgyhJYwmFlv4GDgFOfcQufcU8Bk4Dgz61ng8k+cc+8651Zm\njo/DirNc+vYN7mXYY4+mt9955x2OPPJIVq1axZlnnsk+++zDW2+9xfTp02lsbKRPnz7hB1zFzj33\nXI4//nh+/etfc+utt8YdThPpNOy/v/99Wn8u1SyoUJOZJvpWijB7GEYAHzrnns1pexivByFgUc3n\nTjCzd83seTO70swqooxNUC/D2rVfrH/++OOPOeyww3jqqafo06cPv/71rzn66KN56aWXOPXUU2nn\nt+RCSmJmTJ8+nQEDBvA///M/LExQndl8e43UYu+Cme2QGap81cw+MbN/mdklZtYh7tgqTdJXRwQt\npdQwXOUI87dTT6DJwkLn3Abgg8x9Qe4ETgT2A64Evg3cHk6I5dW3L/Tr53/f6NHw2Wefcdxxx7F4\n8WI+++wz3n33XRoaGrjrrrvo2bNQp4uUomvXrjQ0NNClSxfGjx/Pu+++G3dIpNMQtCt3Dfcu9AYM\nOBXYHZgCnAFcEWdQlSjJqyMaG1WoqRpsUuoFZnYV8MM8pzi8eQuBD5E5x/9i536bc/MFM1sBPGxm\nOznnAuv/TpkyhW7dujVpq6uro66uLk8o5ffII9Cjh989jsMPn85DD/3585bPPvuM6dOnc+CBB7L5\n5ptHFmOt6NWrF3/4wx/41re+xXHHHcecOXPYZJOSX/Jlk7Tehfr6eurr65u0rVq1KtIYnHNzgDk5\nTa+Z2dV4SUPi5y5JcYLmLtRwolyZnHMlHcBXgF0LHJsAJwHvN7u2PfApMK6E59sU2AgcFHD/EMAt\nWrTIJUXv3s55c4GbHxtdp07bu0MOOcRde+21btmyZW7jxo1xh1v1LrvsMge4H/zgB7HF8NhjQa8J\n5+bOjS2sFhYtWuTwEvohrsTPhnIdwOXA/ALnJO59H6eg11evXnFH5tzzz/vHZhZ3ZOJcae/5kr9u\nOefeB94vdJ6ZzQO2MLPB7ot5DAfg9TAEFFT2NTjzj3mn1FjjMnduUC+Dse22y/nrX1XSOUoXXHAB\nCxYs4Be/+AV77LEHxxxzTKTPn0oFT3Ts3FldsrnMbBfgbGBq3LFUkqAicU89FW0cfoL2S9HchcoT\n2hwG59xLeF2Nt5jZMDPbG7geqHfOrQAws23MbJmZ7ZG5vbOZ/cjMhmQmQ40Ffg/Mdc6lwoq13Lp3\nD+5qe/XV9tx/f7Tx1Lp27dpx2223seuuu3LyySeTSkX7Uho+PPi+BM3HLCszu6pZPZXmxwYz27XZ\nNdsCfwXucs4FVBQQP0FbqMRd2yCVCt4vRYly5Ql7QPd44Aa81REbgXuBc3Pu74A3hJFdkLgeODBz\nTlfgTeAeKnAC1KhR3h4TfsuIjjjCeyNpKVF0unXrxn333cfw4cMZP3488+fPZ4sttgj9eRsavFoc\nfnbbrapfA1cDhda0vpr9i5ltAzwKPOmcO73YJ0nK3KU4BeW/SSjjEtS7MGhQtHGIp83zlgqNWST9\nIMFjmXPnBo9bd+4cd3S16Z577nGAO+yww9yGDRtCfa6gsdvskU6H+vStEsccBmBb4J/AHYAVeU1i\n3/dR69zZ//WVSsUbV77XfxJf+7WqlPe8Fv2HaNSo4DXQa9fC738faTgCHHXUUfzwhz/kwQcf5PLL\nLw/1ufKVBZ87N/7u4iQws62BRuANvFUR3c2sh5n5zgKSltau9W+Pu/cqX++CXvuVSQlDyCZO9N/+\nGmDSpC8KOkl0Lr/8cg488EAuueQS/vznPxe+oBVmzAj+IN9tN43f5vgGsDMwBm8I8m28Cc7J3j0s\nIdLpuCPwl2/uwpw5/u2SfEoYIhC0/TV4BZ1K3dFS2maTTTahvr6er33ta5xwwgm88sorZX38hobg\nmv5mShJzOed+75xr3+xo55zTUqIi7LOPf3vc9Q3Uu1CdlDBEYNQomD07+P5+/ZQ0RG2rrbaioaGB\ndevWceSRR7I66OtQiRobgyvuATz/vD4wpXyCct04e7DUu1C9lDBEZNw4b819kKFDo4tFPEOGDOHm\nm28mlUpxyimnZCfTtVq+egvgzWeJe1xZJGzqXaheShgilG/N/bp1cP310cUinokTJ3LWWWdx1113\nMW3atFY/TjoNAwYE39+rVzJq+kv1CNpsKl9vZtiCehfatVPvQjVQwhChvn3zjy2ec46Shjhcc801\njBw5kvPOO4/H8k04CZBOw447BhfPgWRU3JPqEjT0NW5ctHHkCupdeO459S5UAyUMESs0n+Gcc7Tc\nMmodO3bknnvu4atf/SrHHnssb775ZtHXZpOFoBURoCWUUhvyzV3QUFx1UMIQg3Hj8u9RP2mSkoao\nbbPNNtx77718+OGHTJgwgbVr17Jy5cqC8xpGjCicLGgJpZRb0HLKHXaINo5cqupY/ZQwxGTiRLju\nuuD7lTREb++99+baa69lwYIFnH322Vx22WXMDRhDyvYsLA/ccN1LCpUsSBhGjPBvnz8/2jiyNHeh\nNoS9l4TkMXkyTJsW/Etn0iT46CPvPInGmWeeyfz58/nd735Hu3bteP/999lvv/2anJNOe9/k1q0L\nfpyZMzXJUcIT9JkR19CX5i7UBvUwxOzpp/Mvt9REyGjNmjWL5557DoCNGzcya9Ys3n//i93cUynY\nemslCyJZQb0Lm26quQvVRglDzLp3h9df9yoABjnnHC+pUHGn8I0fP55vf/vbdOzYEYD169dz2223\nAV5Rpv7986+GuO46JQsSrsZG//a4llMG9S7ENTwi4VHCkADdu3sVAPMlDevWqSJkFNq1a8fUqVNZ\nuHAh/fv3B+CWW26hocHlLcoEXrKg4SMJ25gx/u1xLKdU70JtUcKQEH37wooVhfew79cP7r8/mphq\nWf/+/Zk/fz7/+7//y7Jlw/KWewYlCxKdNhYkLaukTb6UcClhSJDu3eGNNwovjTriCM1riMIrr3Tm\n+uuvBmYCwd0/M2cqWZDak0rBxx+3bFfvQvVSwpAw3bvDa6/lX3IJ3ryGTp00RBGWhgZvvsL69VAo\nWdCcBYlKUDnoqHenTKdh4ED/+9S7UL2UMCTU5Mn5izuB98usXz/Vayi3Sy/Nv+Nk1ty5ShYkWkGv\ny6jrfRx6KGzc2LJ90CD1LlQzJQwJNnFicTOfJ03SKopyaGjwJp5efHHhc1XBUWpVOg3PPut/n4o0\nVTclDAk3blxx3Y3ZVRSa21C6bNXGYnoVdtrJO1/JgiRFZgVwZA491L9d21dXPyUMFWDUKG9m9BVX\nFD73nHO8b8laSVGcGTOgZ0+vFkYhM2fCq6/qQ1HiETR/YfHi6GII6l1QCejaoIShglxwQfHFWY44\nQsMU+WSHH045pfC5HTt6M8I1X0HiFNQDFtWcgXQadtnF/z6VgK4NShgqzLhx3i+vYrohs8MUO+8M\nK1eGH1slaGz0EoVihh/AW62ybp0mcomMHeu/jLJrV70/aoUShgrUt6/3S6zY3obly6FHj9pOHLKJ\nQqFqjVlm3twR1VeQJIty/sLSpf7tzzwTXQwSLyUMFWzcuC92TixGNnFo1w4efzzc2JJixozSEgXw\nehU2btTERkmOVMq/Par5C6mU/4Zrw4apd6GWKGGocNlCT6UUbnEORo/2fpFWYw2H7KqHYucoZGVX\nQKhXQZJmjz3826P4ZZ1OeysgmjODBx8M//klOZQwVInsSopCxZ6amzTJe+NXQ69Dtjeh2FUPWdnh\nB62AkKTKt5162MaOhQ0bWrbvsYfeL7VGCUOVmTjRSxwKlZZuLrfXoZKWZWaThFJ7E7Jmz9bwg0iQ\ndBoWLGjZ3r69ehdqUWgJg5ldYGb/MLPVZvZBCdddamZvm9knZvZ3MwtYyCP5TJ7cuh6HrCOO+OIX\ncZImS2YnL7YlSQAvUXAuni2BRcqh2EnPbTF2rP/umEuXqnehFoXZw9ABuBv4dbEXmNkPgbOB04Hh\nwGpgjplFXMusemR7HNqyOU12smT2l3SUwxe5PQilTl5srlMnb/KWEgWpJI2N/u1hv4aDehc00bF2\nhZYwOOd+4pz7FfB8CZedC1zmnHvAOZcCvgNsAxwRRoy1JDvHIZ2G7bZr22M1H77IPVrTG9G816Ac\nPQi5rrvOi3ntWn3QSeUZMyae550woWXvgiY61rZN4g4gy8x2AnoCj2TbnHMfmdkzwAi83gppo+7d\n4c03vb/ff7839FBO2d6IuHXqBIsWKUGQyuc3JBC2dBoWLmzZromOtS1Jkx57Ag5IN2tPZ+6TMhs3\nzvswauuQRVJ87WveB516E0RaL52Gr3+95cqMzTdX70KtK6mHwcyuAn6Y5xQH9HHOvdymqJo9beZx\n85oyZQrdunVr0lZXV0ddXV0ZQ6le2SGLrOuv9zaySrrZszUfobXq6+upr69v0rZq1aqYopFShJng\nT5gA//1v07ZOneCVV9S7UOtKHZK4Gri1wDmvtjKWFXjJQQ+a9jJ0BwJ2X//CtGnTGDJkSCufWpqb\nPLlpAaPHH/fmLcRJwwzl5ZdQL168mKFDh8YUkRQrrGXAGoqQfEoaknDOve+ce7nA8VlrAnHOLcdL\nGg7ItpnZl4A9gada85hSPtkeCL+j1JoPhWSXPDY/NMxQ3cyso5ktMbONZjYg7niSYNq0aJ9vwgT/\noYigrbWltoRZh2F7MxsI7AC0N7OBmaNrzjkvmVluh/K1wI/M7HAz6w/cBrwFVEgZodqUrflQrkND\nDDXr53jv9xim+SXT1KnRPt877zS9raEIyRXmKolL8ZZFZmW3SdkfyK7i/zrw+cQD59zPzWxT4DfA\nFsATwKHOufUhxikiMTOzQ4GDgAnAN2MOp2ZtvbVXIj1LQxGSK7SEwTl3EnBSgXPa+7RdAlwSTlQi\nkjRm1gOYDowF1sQcTuK1tY5Kc6kUjBwJa9ZA584weDCsWuUlDxqKkFyJqcMgIjXrVuAm59yzZlbk\nZu21a9Gi8j7eyJFfrIr4+GNvCOKjj8r7HFIdlDCISNkVuwQbOATYHPhZ9tJSnqcWl1OXe4hgzZr8\nt6V6tHUptRIGEQlDMUuwl+PNadoLWGfWJFdYaGZ3ZoY2A2k5ddt16dK07kKXLvHFIuFq61JqJQwi\nUnbOufeB9wudZ2aTgQtzmrYB5gDHAPPDiU7Aq7kwYQJ06waffOK1bbopzJsXb1ySXEoYRCQ2zrm3\ncm+b2Wq8YYlXnXNvxxNVMpx1VniPnS3/nNuzsPfe8OST4T2nVL4k7SUhIgKqwwDATTeF87h+yQK0\nrMEg0px6GEQkMZxzrwMtlltLeQQlC+AtoxTJRz0MIiIV4vzz23b94Yf7Jwsq/yzFUMIgIlIhrryy\n9demUrBgQcv2zTdX+WcpjhIGEZEql0pB//4t282ULEjxlDCIiFSxoGQBYNgwJQtSPCUMIiJVKp2G\ngQP972vfHh54INp4pLIpYRARqULpNPTqBRs3+t+/dKl6F6Q0ShhERCrEypXFnZdOw847w+rV/ven\nUtC3b/niktqghEFEpELsvnvhc7LJQrbcc3ODBytZkNZR4SYRkQrxfoHdOdJp2Gmn4B0n27WDhx4q\nf1xSG9TDICJSBVIp2Gab/NtTP/ec5i1I6ylhEBFJoO9+17/98cdbtjU0eEsngyY4Asydq6EIaRsl\nDCIiCXTLLf7to0c3vX3ppd421fnMnQujRpUnLqldmsMgIlJhRoyAQw+Fiy8ufO7s2UoWpDyUMIiI\nVJinn/aOQmbPhnHjwo9HaoOGJEREEqrQUEM+Shak3NTDICKSUPfe620QVSrNWZAwqIdBRCTBzj+/\n+HO33tqrxaBkQcKghEFEJMGuvBKuu67weXPnwttvq86ChEcJg4hIwk2eDM7lP9SrIGFTwiAiIiIF\nhZYwmNkFZvYPM1ttZh8Uec2tZrax2fGXsGIUERGR4oS5SqIDcDcwDzi5hOv+CkwCsnOD15U3LBER\nESlVaAmDc+4nAGY2scRL1znn3g0hJBEREWmlJM5h2M/M0mb2kpndZGZfjjsgERGRWpe0wk1/BWYB\ny4FewFXAX8xshHPOxRqZiIhIDSspYTCzq4Af5jnFAX2ccy+3Jhjn3N05N18ws+eBfwP7AY+15jFF\nRESk7UrtYbgauLXAOa+2MpYWnHPLzew9YBcKJAxTpkyhW7duTdrq6uqoq6srVzgiVaW+vp76+vom\nbatWrYopGhFJupISBufc+8D7IcXSgpltB3wFeKfQudOmTWPIkCHhByVSJfwS6sWLFzN06NCYIhKR\nJAuzDsP2ZjYQ2AFob2YDM0fXnHNeMrNxmb93NbOfm9meZraDmR0AzAZeBuaEFaeIiIgUFuakx0uB\n7+TcXpz5c3/g8czfvw5kxxE2AAMy12wBvI2XKFzknPs0xDhFRESkgDDrMJwEnFTgnPY5f18LHBJW\nPCIiItJ6SazDICI1xsy+ZWZPm9knZvaBmTXEHZOINJW0OgwiUmPMbAIwHfh/wKN4ZeX7xRqUiLSg\nhEFEYmNm7YFrgf91zs3MueuleCISkSAakhCROA0BtgEws8Vm9raZ/cXMdo85LhFpRgmDiMRpZ7yd\naS/GW1n1LeBDYK6ZbRFnYCLSlIYkRKTsii0jzxdfWi53zs3OXHsS8BZwNHBLvudRhVeR4rW1uqsS\nBhEJQ7Fl5LfJ/H1ZttE5t97MXgW+VuhJVOFVpHhtre6qhEFEyq7YMvJmtghYB+wGPJVp6wDsCLwe\nYogiUiIlDCISG+fcf83sZuAnZvYWXpJwHt6QxT2xBiciTShhEJG4fR/4FLgN6AI8A4xxzmnrTJEE\nUcIgIrFyzm3A61U4L+5YRCSYllWKiIhIQUoYREREpCAlDCIiIlKQEgYREREpSAmDiIiIFKSEQURE\nRApSwiAiIiIFKWEQERGRgpQwiIiISEFKGERERKQgJQwiIiJSkBIGERERKUgJg4iIiBSkhEFEREQK\nUsIgIiIiBSlhEBERkYKUMLRCfX193CEEUmytk9TYkhpXtaqUn7fiLL9KiTXOOENLGMxsBzP7rZm9\namafmNm/zOwSM+tQ4LpOZnajmb1nZv81s3vNrHtYcbZGkl9Yiq11khpbUuOqVpXy81ac5VcpsVZl\nwgD0Bgw4FdgdmAKcAVxR4LprgW8BE4BRwDbArPDCFBERkUI2CeuBnXNzgDk5Ta+Z2dV4ScN5fteY\n2ZeAk4HjnHNzM20nAcvMbLhzbn5Y8YqIiEiwqOcwbAF8kOf+oXhJzCPZBufcP4E3gBHhhiYiIiJB\nQuthaM7MdgHOBqbmOa0nsN4591Gz9nTmPj+dAZYtW9bmGIu1atUqFi9eHNnzlUKxtU5SY4s6rpz3\nUefInrR1QnnfJ/V10JziLL9KibXccZb0nnfOlXQAVwEb8xwbgF2bXbMt8C/gNwUeuw5Y49M+H7gy\n4JrjAadDh46yHseX+tkQ5YHe9zp0lPso+J5vTQ/D1cCtBc55NfsXM9sGeBR40jl3eoHrVgAdzexL\nzXoZuuP1MviZA5wAvAasLfD4IpJfZ2BHms4/SiK970XKo+j3vGWy9VCY2bZ4ycIC4NuuwJNlJj2+\nizfp8b5M267AS8BemvQoIiISj9ASBjPbGngc7xvARLyhCgCcc+nMOdvgTXD8tnNuYabtJuBQ4CTg\nv8B1wEbn3L6hBCoiIiIFhTnp8RvAzpnjzUyb4Y2VtM/c7gDsCmyac90UvOTiXqAT8BBwVohxioiI\nSAGhDkmIiIhIddBeEiIiIlKQEgYREREpSAlDEczsAjP7h5mtNrN8lSqbX3epmb2d2Xzr75niVeWO\nbUszu9PMVpnZh5kNv7oWuKbRzDbmHBsyk03bGstZZrbczNaY2dNmNqzA+Ueb2bLM+UvN7NC2xlCO\n2MxsYs7PJfsz+iSEmPY1sz+Z2X8yzzG2iGv2M7NFZrbWzF42s4nljku+YGYdzWxJ5v9nQNzxNNfa\nTf4iiq2kz4Oomdn5ZjbfzD4ys7SZ3ZdZlZdombg3mtk1UT+3EobidADuBn5d7AVm9kO8ypanA8OB\n1cAcM+tY5tj+APQBDsDbtGsU8JsC1zhgOtADr4Lm1gTs71EsMzsW+CVwMTAYWIr3790q4PwRmdhv\nAQYBs4HZZrZ7W+IoR2wZq/B+Ntljh3LHBXQFluBN6i04mcjMdgQexFtZNBD4FfBbMzsohNjE83Pg\nLYr4/4lJazf5C1Ur33NR2xe4HtgTOBDvc/5vZtYl1qjyyCRdp+L9PKMXd8W2Sjrwlod+UOS5bwNT\ncm5/CVgDHFPGeHrjVdccnNN2MPAZ0DPPdY8B15T5Z/M08Kuc24b3QXtewPl/BP7UrG0ecFMI/2+l\nxlb0/3MZY9wIjC1wzs+A55q11QN/iTLWWjnwlne/kPM+GxB3TEXG/X3glZhjKOk9l4QD2Crz/7xP\n3LEExLcZ8E9gTBif4cUc6mEIgZnthPetNHcTrY+AZyjvJlojgA+dc8/mtD2M921ozwLXnmBm75rZ\n82Z2ZVuy6kz351Ca/ntdJpagf++IzP255uQ5P8rYADYzs9fM7A0zC6XnoxX2IoKfmYCZ9cDrhTsR\nL9GvJIU2+QtVG95zcdsC77Mztp9dATcCDzjnHo0rgMg2n6oxPfFeeM3LWefbRKu1z7Myt8E5tyEz\nzyLf89wJvI7XCzIAr9t1V+CoVsaxFV5tDb9/724B1/QMOL+cPx9oXWz/xNtm/TmgG/AD4Ckz6+uc\n+0+Z4ytF0M/sS2bWyTm3LoaYqtWteL1dz5pZGMNRoShyk7+wteY9FyszM+BavC0MXow7nubM7Di8\nods94oyjZnsYzOyqZhP/mh8bQpgAky1cFXZseZ/HOfdb59zfnXMvOOfqge8AR2Z6RsqpqH9vG85v\ni4qxegUAAAONSURBVMDncs497Zy7wzn3nHPuCWA8Xsny0yKKrRSW+TOpY+yJUez7yszOATbHGwKC\nL37GiYu12TXbAn8F7nLOzYg65iJE+f4u1U14c0COizuQ5sxsO7xk5kTn3KdxxlLLPQwlbaJVohV4\nb44eNM2yuwPP+l7RVLGxrcg85ufMrD2wJcGbdfl5Bi/eXYDlJVyX9R5edc4ezdrzbRq2osTzW6s1\nsTXhnPvMzJ7F+/nEKehn9pFzbn0M8VSaYt5Xy4H98YZ/1nlfPD+30MzudM6dFFJ8ucLc5C9sbX7P\nRcnMbgC+CezrnHsn7nh8DAW+CiyyL16Q7YFRZnY20Ckz5BO6mk0YnHPvA++H9NjLzWwF3sqF5+Dz\njbX2xBuHKktsZjYP2MLMBufMYzgA75f/MyWEPBgv82/Vm8U596mZLco8958ysVnm9nUBl83zuf+g\nTHvZtDK2JsysHdAP+Es5Y2uFeXgT8XJ9gzL/zKpVCe+rycCFOU3b4M0VOQaIZAO8Uj6frOkmfyeH\nGVcxyvGei0omWRgHjHbOvRF3PAEeBvo3a5sJLAN+GlWyAGiVRDEHsD3eMraL8JbbDcwcXXPOeQkY\nl3P7PLw3/OF4/9mzgX8BHcsc21+AhcAwYG+88ffbc+7fBu+FtUfm9s7Aj4AheEsFxwKvAI+2MY5j\n8CaHfQdvVvlvMv/+r2buvw24Muf8EcB6vLHW3YBL8LYp3j2E/79SY/sxXvKyE14yVY+3LLZ3mePq\nmnkdDcKbnf29zO3tM/dfBfw+5/wdgY/xusp3A87M/AwPjPs9Us1H5n2SyFUSeEui/wX8PfNe75E9\nYo4r73suCQfeMMSHeMsre+QcneOOrYjYY1klEfs/vBIOvK7BDT7HqJxzNgDfaXbdJXgTCz/B+4ay\nSwixbQHcgZfIfIhX12DTnPt3yI0V2A5oxBuT/wQvwbgK2KwMsZyJtzvpGrxvvXvk3PcoMKPZ+RPw\nEq01eD0xB4f4f1h0bMA1eF3TazL/fw+E8csCGJ35RdT8dTUj53X3qM81izKx/Qtvp9fY3yPVfOS8\nh5KYMEz0ef1sBDYkILbA91wSjoD3XovP8SQemc+syBMGbT4lIiIiBdXsKgkREREpnhIGERERKUgJ\ng4iIiBSkhEFEREQKUsIgIiIiBSlhEBERkYKUMIiIiEhBShhERESkICUMIiIiUpASBhERESlICYOI\niIgU9P8B/EcwhVzEWEEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb76c53d690>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "arrow_dir = np.array([[0, 0, v[0][0], v[0][1]], [0, 0, v[1][0], v[1][1]]])\n",
    "X, Y, U, V = zip(*arrow_dir)\n",
    "ax = fig.add_subplot(1, 2, 1)\n",
    "plt.plot(x_values, y_values, '.')\n",
    "ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)\n",
    "ax.axis('equal')\n",
    "\n",
    "plt.xlim([-2, 2])\n",
    "plt.ylim([-2, 2])\n",
    "\n",
    "\n",
    "ax = fig.add_subplot(1, 2, 2)\n",
    "ax.axis('equal')\n",
    "plt.plot(trans[:, 0], trans[:, 1], '.')\n",
    "\n",
    "plt.xlim([-5, 5])\n",
    "plt.ylim([-5, 5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Singular Value Decomposition (SVD)\n",
    "\n",
    "**SVD** is another way of factorizing a matrix to give **singular values** and **singular vectors**. However, it is more generally applicable than eigendecomposition, e.g. eigendecomposition is not defined for a non-square matrix and we opt for SVD there. Here, we write $A$ as:\n",
    "\n",
    "$$A = UDV^T $$\n",
    "\n",
    "**Shapes**:\n",
    "- $A$: $m$ x $n$\n",
    "- $U$: $m$ x $m$\n",
    "- $D$: $m$ x $n$\n",
    "- $V$: $n$ x $n$\n",
    "\n",
    "**Properties**:\n",
    "- $U$ and $V$ are defined to be orthogonal matrices. \n",
    "- $D$ is a diagonal matrix (not necessarily square), diagonal elements of which are called **singular values**.\n",
    "- The columns of $U$ are known as **left-singular vectors** (eigenvectors of $AA^T$) and those of $V$ are called **right-singular vectors** (eigenvectors of $A^TA$).\n",
    "- Most useful feature is to extend matrix inversion to non-square matrices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. The Moore-Penrose Pseudoinverse\n",
    "Suppose we want a left-inverse $B$ of a matrix $A$ ($m$ x $n$)to solve a linear equation:\n",
    "$$ Ax = y \\Rightarrow x = By$$ \n",
    "\n",
    "We define the pseudoinverse of $A$ as:\n",
    "\n",
    "$$A^+ = \\lim\\limits_{\\alpha \\rightarrow 0} (A^TA + \\alpha I)^{-1}A^T$$\n",
    "\n",
    "However, for practical algorithms its defined as:\n",
    "\n",
    "$$ A^+ = VD^+U^T $$\n",
    "\n",
    "where $U$, $D$ and $V$ are the SVD of $A$ and $D^+$ is obtained by taking the reciprocal of all non-zero elements of D and then taking the transpose of the resulting matrix.\n",
    "\n",
    "**Case 1**: m <= n\n",
    "\n",
    "Using $A^+$, gives one of many possible solutions, with the minimal **Euclidean norm**:\n",
    "\n",
    "$$ x = A^{+}y $$\n",
    "\n",
    "**Case 2**: m > n\n",
    "\n",
    "It is possible for there to be no solution and $A^+$ gives the $x$ such that $Ax$ is as close to $y$ in terms of the **Euclidean norm** $||Ax - y||$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. The Trace Operator\n",
    "\n",
    "The trace operator gives the sum of all the diagonal elements.\n",
    "\n",
    "$$ Tr(A) = \\sum_{i}A_{i,i}$$\n",
    "\n",
    "Properties:\n",
    "\n",
    "- $||A||_F = \\sqrt{Tr(AA^T)} $ (**Frobenius Norm**)\n",
    "- $Tr(A) = Tr(A^T)$ (**Transpose Invariance**)\n",
    "- $Tr(ABC) = Tr(CAB) = Tr(BCA)$ (**Cyclical Invariance** given that the individual matrix multiplications are defined)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. The Determinant\n",
    "\n",
    "The determinant of a square matrix (denoted by $det(A)$) maps matrices to real scalars. It is equal to the product of all the eigenvalues of the matrix. It denotes how much multiplication by the matrix expands or contracts space. If the value is 0, then space is contracted completely atleast along one dimension causing it to lose all its volume. If the value is 1, then the transformation preserves volume."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}