KrylovBasedMethods.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#math and linear algebra stuff\n",
    "import numpy as np\n",
    "import numpy.linalg as la\n",
    "\n",
    "#Latex rendering\n",
    "from IPython.display import display, Math, Latex\n",
    "\n",
    "#plots\n",
    "import matplotlib as mpl\n",
    "mpl.rcParams['figure.figsize'] = (15.0, 15.0)\n",
    "#mpl.rc('text', usetex = True)\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Krylov space based methods\n",
    "\n",
    "This notebook was inspired by some previous work by [François Roux](https://www.ljll.math.upmc.fr/MathInter/telechar.php?filename=polycopies/fxroux6.pdf) and [Gilbert Strang](https://ocw.mit.edu/courses/mathematics/18-086-mathematical-methods-for-engineers-ii-spring-2006/readings/am64.pdf)\n",
    "\n",
    "## Taylor expansion: from scalar to matrices\n",
    "\t\n",
    "Taylor expansion can easily be applied over differentiable scalar functions of one variables, this leads for instance to the following expansion, for $x$ close to $0$:\n",
    "\n",
    "\\begin{align*}\n",
    "  \\frac{1}{1-x} &= 1 + x + x^2 + x^3 + ... \\\\\n",
    "  &= \\sum_{n=0}^{\\infty} x^n\n",
    "\\end{align*}\n",
    "\n",
    "It is interesting to notice that the right part of this equality can also be interpreted as the sum of all terms of a geometric series with a common ratio of $x$, which is valid when $|x| < 1$.\n",
    "\n",
    "For the sake of our argument, we will assume that, $M$ is symmetric, so that all its power can be simultaneously diagonalised in the same eigen space, so that we juste have to extrapolate the scalar case to each diagonal elements.\n",
    "The previous expansion extended to the symmetric matrices case over a real field then reads:\n",
    "\n",
    "\\begin{align*}\n",
    "  (Id - M)^{-1} &= Id + M + M^2 + M^3 + ... \\\\\n",
    "  &= \\sum_{n=0}^{\\infty} M^n\n",
    "\\end{align*}\n",
    "\n",
    "This expansion is valid for matrices whose operator norm $\\|M\\| = |\\lambda(M)_{max}|  < 1 $, and of course apply for nilpotent matrices, which have all their eigenvalues identically equal to zero, their characteristic polynomial being $x^n$.\n",
    "\n",
    "Using $|\\lambda(M)_{max}| =  c \\neq 0 $, excluding the already handled case of nilpotent matrices, we can extend the previous power series to all symmetric matrices M:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\left( Id - \\left( \\frac{M}{c}\\right) \\right)^{-1} &= Id + \\frac{M}{c} + \\left(\\frac{M}{c} \\right)^2 + \\left(\\frac{M}{c} \\right)^3 + \\; \\dots \\\\\n",
    "    (c Id - M)^{-1} &= \\frac{1}{c} \\left( Id + \\frac{M}{c} + \\left(\\frac{M}{c} \\right)^2 + \\left(\\frac{M}{c} \\right)^3 + ... \\right) \\\\\n",
    "    &= \\frac{1}{c} \\sum_{n=0}^{\\infty} \\left(\\frac{M}{c} \\right)^n\n",
    "\\end{align*}\n",
    "\n",
    "Withou loss of generality, a variable change $M = c Id - P$, where $P$ is also symmetric, gives us\n",
    "\n",
    "\\begin{equation}\n",
    "    P^{-1} = \\frac{1}{c} \\sum_{n=0}^{\\infty} \\left(\\frac{c Id - P}{c} \\right)^n\n",
    "\\end{equation}\n",
    "\n",
    "We will see how this Taylor expansion is linked to preconditionned iterations in the next section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x8eb5ad0>"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAEZCAYAAAA6xErnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4FUUXh98JHUKvoQapgoIiYEPBjgUsKArSBESwYP3s\nil0sFJUiKkUQRFAUUIr0Ioj03iH0Hlooaff3/TEbuEnuTe5NIUH3fZ77JLvTzs7uztmZPXuOkYSL\ni4uLi0tWEpLVAri4uLi4uLjKyMXFxcUly3GVkYuLi4tLluMqIxcXFxeXLMdVRi4uLi4uWY6rjFxc\nXFxcshxXGaWAMeakMSb8ArU10BjzxoVoK7thjClljJlrjDlujPksnXX9q/oxI/smDW1fsOs/EIwx\n3Ywx+40xJ4wxRS9QmzWMMbEXoq1gMcZsMcZcnUVtLzTGtM7IOnNmZGXBYIyZDdQBSkvKlidbUsHM\nqNcY0x7oLOkGr7a6ZUZbFwldgIOSCvtKNMYMBVoD0UAMsAx4RtLGpHn/hf2YYt9kFMaYWcAISUMS\n9mXW9Z8WjDE5gV5AQ0lrkqQ1AiYDwj5g5weiAOPsqyVpdzqaz5YfY0qqmtUyZCRZMjMyxlQCGgIH\ngeaZ2E6OzKo7nSTcJNkKX/0VbB+msc8rAetSyfOJpEJAeex1MzQN7WQKmXydBdI3/wXKAHmA9UkT\nJM2XVNC5Pmpj763CCfvSqYjSTGZdFxdyXMvotlKsT9IF/wFvAeOB14GJSdKGAgOBP4ETwCygole6\nB3gG2IodlD71SmsPzAd6A4eB97AD/5tABLAfGAYUdPK3BLYBoc72ncA+oLhXW5d4ydUfmAScBOZi\nb5C+wFHsgFHXS5ZXgC3OMawB7nP21wTOALFOPZFe9b/nVf5xYLNzHL8BYUn64AlgExAJ9Euhrw3w\nqiPLIWA0UMRJq+TU1RHYAcz2tc/J29w5jkhgJlDTq43twMvASufYQnzIcR3wj9NXi4BrvY47Bjvr\nOQHc7KNs0r65Czjh53jP5QUaA7uAF4ADwB6gg5PW0DnXxqvs/cDKNPZbHuAH53wlHGNJJ38h4Dtg\nryPP+97tpnDukvWNj75oDOxKci5edM7FUeBHILdX+r3AcuC4c33dDnwAxAGnnXa+9HH9FwKGY++5\n7cAbSe67ecBnzvWxFWjqld7B2XfC+dvKz/Hmxt5Pe4DdQB8gF1ANO9OJd+qYnkKfVXLyhSTZ3wWr\nyE5g75vHvNI2A7d4becBjgE1nF+MV1oF4A/gCLABaOeV9jEw0rlWjgOtfch3r5ccO4Cnk15/znmb\nA1zqlbbPOa9rgCivfdc5/4dgx9WEcXEEUCiFfnqK89f2z0Apr2P3AF2d9HXO/rs5P970AhZ6Hx92\nPNqAvf4nAmVTqs+nTKndEJnxc05+a+cii8G5ab1uwOPA9c6F2BeY55XuAWYAhbFPyRuBjl43RSzw\npHNy8mAHjE3ORZof+AUY7lXfCGAIUAx7E9zplRZPYmV0ELgCe9PMwCq4R7ED1/vATK+yLbBLkAAP\nYW+m0l5yzk1hEL3ZuUjqOn3wJTAnSR9MAApib46DwO1++vpZYAEQ5tQ1EBiVZFAdBuRz+svXvoTB\n4GYgB/A/5xzm9BoAlwFlgTw+ZCiKvYhbO+flEWe7aNJjT2FQTuibUOwNPzuAvI2d66GHI/edwCns\nUzMkH4TGAP8Lst/yOn3UBfuAlce5Hq7k/EPOr8AAJ28J4G/g8QDvlaTKx5cy2um1vd2pvzRQBPuQ\n1MVJa4gdZG92tsOA6s7/s3DuIz/X/3DnOPI7x78RZ0DHXs/R2HvNYAeePU5afuz9XNXZLo3XIJuk\nvfecPi/u/P4C3vXq83hSUeL4V0b34DzUYq/j0wlyYB9Wh3nlfRhY5PyfVBn9DXyOfcVxFVYpJTxY\nfYx9GLvD2fZ1LxwGrvK6L+o6/1+DHX+ucPqws9PHIU76PuwDTumEekmsjF7BKrDS2PFpCDDETx/d\nhX0wqu3kHQRMTZAZe21PxD6A5ME+dEdhFVIO7ENaLI4ycvprLVDFSX8PZyxMUl9BX31yTq5AboiM\n/AGNnAsh4UZdDjyb5GYb5bVdAPvUVs7Z9gC3eaV3A6Z53RQRSdqbDnT12q6OVYAJJ7kw9gllFTAg\nSdmkM6NBXmlPA2u9ti/DmeX4Oe7lQDMvOVNSRt8BPZP0QQznbyYPzg3gbP8EvOyn3XXATV7bYQnH\nz/kbt5KPm9l735vAaK9tg31yvdHZ3g60T+HY2wB/J9m3AOepksCU0RmsAtuLnSlWTiGvtzI6hdfA\nhJ0hNXT+fx8Y7PxfEHvDlU9jvz2GnZVfnkSeUsBZvG5CrDKe6e94/R2Pn21fyqiV1/YnCdc18DXQ\ny087vpSRB7jEOeZooIZXWhfODzjtgU1eafmcsqWwyigS+9SfN5Vj3YIzkDvbtwPbnf/D8aFkfNTh\nUxn5yDcZ54HAqfsY5wf5iTgzFryUEfah7HSSc9nbq38/Bqak0u5+p79Ck+wfAryWZF8E0MD5fx/w\ncJJ0b2W0jcRjQmXglB8ZfgDe8dou7PRZKc4rj6u90h8n8YN2CPY+SlBGM5Ncc7lwJhm+6vP3y4p3\nRu2APyVFOdtjsSfHm10J/0g6hb2Yy3qle68B70iStovElHXyeOfPiX2CQNJxR4ba2AsrJQ54/X/G\nx3ZowoYxpp0xZrkx5qgx5qhTf4lU6vcps9MHR4ByfmQ57d12EioBvxpjIo0xkdhBNhbn+B18ral7\n70sqj7D9XM5P/qQkPQc42+V85PXHZ5KKSSor6T5J2wMsd0SSx2vbu69GAfcbY3IBDwBLdf79QrD9\nNhyYCow2xuw2xvR01scrYW/OfU5dR7FKIdBrIS34uzYqYJdxgqUE9p7Z6bUv6fnbn/CPpDPOv6GS\nTmOfnLth+2CiMaaGn3bK+mgjLKHaNMh9DmNMc2PMImPMEecc3IRzDiRFYB8W7zPGlMDOnEb7qCYM\nOCQpOomM3v2QdPxJyr3Ag8BOY8x0Y8xVzv5KwOsJ15sjYwkCv8cqAJO8rtdlznEX85E36f18HLts\n6K+tsiQekz3YWVwClYCvvdo+iFVG5QOUHbjABgzGmLzY9zQ3G2P2GWMS1kHrGmMu98pawatMKOeX\n0JKlAxWxT8sJJL1o92I7K4FK2EHlgFP/FdjlhR+Br9JwWMkwxlQEvgGelFRUUlHsNNb4kTEpiWQ2\nxhTALluk5UXsTuzSYzHnV1RSAUn7vPL4ksd7X9I+BHsOvOVJ6Zj2Yp8+valI4nN6wZG0HntT3gW0\nwiqnBILqN0nxkt6XVBv7fqwZ9sFrF3ZmVNyrniKS6qRR7FPY2UYCYf4y+mAXdinFFymdv8PYeybp\nfRTQ+ZM0TdLt2OWejcC3frLu8dHGXj95A8YYkx+7BPsuUMK5H2dx/n4E+zDRFjtrnSHpsI+q9gIl\njTF5vPYlvY5TvLclLZLUDDsLmcb5a24X8HaS6y1U0m8B1r0bu/ya9HqN9HMc3uNLEeySnL/7eZ9z\nnAn5DckVcAcfsi8PUHbgwlvT3Y9dcrsU+z6krvP/fOyNm8BdxpjrjDG5sUspCyV5X5T/M8YUMcZU\nwK7t+3qKSeBH4HljTLij2D7ELjl5HOU4ArsG2hEoa4xJj2lwwsVdADs1PWyMCTHGPIZdxkvgAFDe\neSL3J/Njxpg6zoX/EXaZK7WnLl8MAj5yFCTGmJLGGG8LRuOjTNJ9Y4C7jTE3GWNyGmNewg6wCwOU\nYRJQzRjziDEmhzHmYex5/z2oI8kcRmGvoRuwM+QEguo3Y0wTY8xlxpgQ7HJfLBAvaT/WGKePMaag\nsVxijLnRKVfJGONJaCcAVmDvj6LGmDKO7IEyGHtd3eTIUdZrlnIAuySXDOdJeAzwoTEm1LGGfR57\n76SIsd9JNXcUQiznDRF8MRp40xhTwpmhvJWkDV/Xqs9mk2znw87sDjkyNQeaJMkzFvsKoStWMSWr\nT9IWYDXwgTEmtzGmHnbcSrUfnHbzG2MeNsYUxPZBFHacAPvw+kzCTMnp52bOGBUIg4BPjDHlnfKl\njDH3+Mn7I/C4MaaWU39PrAI+5Cf/BKC+MeYuY03sX8a+70rga+AtY0x1p+2ixpgHApT7HBdaGbXD\nvlTbI+lgwg/oBzzq3MhgB4h3sEtTV2LfOXgzHliKnYpOxK63+mMI9mKZi12iOA10d9I+AnZI+kZS\nDPbJ6H1jTMLTY7BLA4JzT9y9sC8792OX6OZ75ZuJnSntN8YcTFaJNAN7I47DPnVVxj6xJWonhW1v\nvsD215/GmOPYdzUNUymbaJ+kTdhz0A97Q9+Nff8VF0D7OE9n9wAvYZ+yXwLu9npqS62f07VEk0pd\no4EbsTej91NksP1WBmuVdBx7bmdh1+bBXve5sUt9kdiBr4yTVhH7bsDfLCNpOyOw7zcjgCkkfxDz\n21eSFmPfbfV15JzN+SfeL4CHnGWsvj7q6o69d7Zh76UfJA3115ZX2RCsNeMe7Lm/Ebtk54sPgCXO\n8a10/v8wkGPz07bdkI5gr7nfHRmaYy3ivPNEOellsYOvv/oewt7P+7GD+kuSAn0oA/vQG4G1mGuD\n8xAuaQG2jwc5S3QbsLP1hLZTu08/xc60ZjrX63zs2Jm8kPQH9v3WROxsqBR27PNVL84D1SPYa+Qg\n9l3QEq/00dhVpXHGmGPYcflWf/X5wzgvnLINxn7guEvS237SPVjLnG0XVjIXl4zHWG8RByX5W7py\nuUAYYz7Amjh3yWpZ/otkmQcGFxcXkPRh6rlcMhtjTEns91D3ZbEo/1myo2+6C7lk4+Li8h/HGPMU\n1iR+jKQlqeV3yRyy3TKdi4uLi8t/j+w4M3JxcXFx+Y/xr3tnZIxxp3ouLi4uaUBSoObzGc6/cmY0\n9W0xsqr4uaHYMkbExwbnskgSPXr0CLpMZtWTnWT5Nx5TdpLFPSZXlqyqJ6v5VyqjtuNg2p1QoTOs\n6gujqsOqLyE2KvWyLi4uLi4Xnn+lMvp4ApQtC3f2gQV3QMOBsG8u/BAOf78Op/alWoWLi4uLywUk\nVWVkjCltjBlsjJnsbNcyxnTKfNHSTp/jsPJemDIVok7B9S/A8sZw52yIPQmja8HMx+DIGv91NGnS\nJENkyYh6spMsGVWPK0vm1pOdZMmoelxZMr+erCRV025HCQ3FBtOq6/gmWi7p8hQLZhHGGC2K38+f\ne0vTdx/0KA+3noK3P4GFS+GdF+GRW2Hjd7CmHxS/Aq54EcrdAibLXt25uLi4ZC3GGJTNDRhKSBqD\n49BP1h+ZP0eH2YK7Q36hZPmVzLtMjD0Cj52BHl/AuMEwchw0eBB2Xgmtt0GVh2B+dxh7JWz6AeJj\ns1p6FxeXjCA8PBxjjPtL8gsPD8/qU+OTQGZGs7FRS6dJqmeMuQb4RFLjCyBf0BhjtFFHeJCJXE4J\nBuo2fjyYmzd3QtfS8Ho5mDMHXv0I8uWFT96AG66GnVNgZS84tgku7w61ukCewll9NC4uLmnFedLP\najGyHf765WKYGb2A9WJbxRjzF9a9+jOZKlU6mcm3LKQV+cjJ1WYk15c+zMq6sO4MXLkKCtSHZX/C\n049B+2ehWXs4URmaz4A7x8ORFTCysp0xHduc1Ufj4uLi8u8nIHdAznuiGti4HhslZcliljHmXmz4\ngoLYUBTTfOTRHbqM67iFN+nNCNbzEnP4nMa0pzbjjsAz26FZMfikIuSNh0Ej4KMvoelN8O5LUKkC\nRO2GNQNg/bdQ+lqo8yyUu9l9r+TicrHgzox8k11nRn6VUWrBkSSNyxSJAsDYyISfSXrcR5qOKZLH\nuZcwyvMZw9jECR5iItdRlq+4mZi4XLy6E34/Cl+Fw/3F4cRJ+Hwg9B8GHVrC692heDGIPQ2bR9rv\nlUwI1HkOqrWGnPku+GG7uLgEgauMfHMxKqMUA2dJ6pjmRo0ZjA22dkBe4ZeNMU2xgb9CgMGSPvFT\n/nNscK8VPtIUrT14KMqztOYUUXzNOCAPTzCN1RxmLM2oQTHmnoAuW6FWfuhXGcrmhv0H4b3eMGYC\nPNMJnu8ChQqCBLunw+ov4OBi+06pdjcoUDatveDi4pKZuMrIN9lVGWWIK4o0uJ1oBFwBrPLaFwJs\nwcZmz4UNr1zTSWsL9MZGYeyJjfXur26t1eWK1VHFKU6vqYvuUT0d1H555NHXWqES6q/RWi9JOhMv\nvblDKvGP9PV+Kd4jSdLWCKnt01Kpy6TPBkinT+sckRukuU9Jg4tK0x6VDiyWi4tLNsMOb9mXfv36\nqX79+sqTJ48ee+yxFPP27t1bZcqUUeHChdWpUyfFxMScS4uMjNR9992nAgUKKDw8XKNGjUqxLn/9\n4uzPEp0gKeB3RndjQ+2ei8cu6b30KEFjTCVgopyZkWOl10PSnc72q07nfOJV5hlsmN7FwApJ3/io\nVzv1LKdZSjX+xJCXL3iX3/iB75lKJaqwnAM8xO80JZxeNCYPOVl9Ch7fBrkNfFMFajrLcOs2wtuf\n2W+U3ngWOreG3LltWvRRWD/Yfq9UoLxdwqt8H4T869zPurhcfGT3mdFvv/1GSEgIU6dO5cyZMwwZ\nMsRnvqlTp9KhQwdmzZpFWFgY9913H9deey0fffQRAK1atQJgyJAhLFu2jLvvvpuFCxdy6aWX+qzv\nop0ZAV9jLeh2AT2A1dgltPTOjiqReGbUAvjGa7sN8GUa6tWNjW/UA+3D9UyP6po5a7ok6QcNVEOF\naY2WSZKO6awe0HhdpRHapmOSpDiP9MVeqfg/0rs7pbPx558alqyQmraSwhtIQ0dLsbHn0+JjpS0/\nS+MaScMrSss+lc5G+nz4cHFxuUCQzWdGCbz55pspzoxat26tN95449z2zJkzVaZMGUnSqVOnlDt3\nbm3ZsuVcert27fTaa6/5rS+hX2bNmqUePXqoffv2aty4cZbPjAIZ3Fcl+RsKzEt3w5mojCQpXtHa\nrDsUocfkkV17m6SfdZVK6i/NkCR55FFfLVVJ9ddv2nzuZO04KzVfL9VYJs08lvhEzvtbuvE+qWYj\nacwEKT4+cfqBxdL0NnYJb043KXK9XFxcsoB/izKqW7euxowZc277yJEjCgkJUWRkpJYvX64CBQok\nyt+rVy81b97cb33++iWrlVEgC0pnnL+njTFlgSNAWLAzsADYA1T02i7v7AseiRCTm8r8wmZuYS+v\nUY6e3EkLilKCp3mId+jHPbTkWepxNWV4hD+Yx24+5gYq5snB+JowPhLab4EmheDzcCiVCxpdDbPH\nwbQ58EZPaxL+wStwl+NOqFR9uGWEdca6diCMbwIl6sJlz0Clu6xFnouLSzbB1MqYerQuY+rxQVRU\nFIULn/8Cv1ChQkji5MmTREVFUahQoUT5CxUqxMmTJzNNnswikKHx9wRTamAZEAH8mAFtG+eXwGKg\nqjGmkjEmN/AI9mPb4FnSD4AcFKAqf3CM8RygNwDX0JjhTONDXmAYXzn7yrKUNqwnksb8xC5OAHBv\nMVh3hVVCl62Abw+AR1bp3N4E/pkMbz8Pr3wAje6F2QvOi1AgDBq+B213QLU2sOQdG8piZV+IPp6m\no3JxcclotC5jfplIaGgoJ06cOLd9/PhxjDEULFgwWVpCesGCBTNVpswgVWUk6X1JxyT9gl1aqynp\nrfQ0aowZBSwAqhtjdhpjHpMUj/Xs8CewFhgtaX2aGpj/HhxcDUBOilONqRykL0cYAUAt6jKGeXzP\nV3zGGwhRnHxM5H7upSoNGMlktgMQmsPOiqbVgiEH4Ya1sPpUwnHA/XfByhnwZHvo/CLc9jD8s/y8\nKDnyQI220GKxnTEd/Md6d5j7FBxN29G5uLj8h6hduzYrV648t71ixQpKly5N0aJFqV69OnFxcWzd\nuvVc+sqVK6ldu3ZWiJo+UlvHA54CinhtFwWezMq1xVTklVYMlQbVlmLO22Of1lqtVCkd0x/n9h3W\nQTVXA72sjorVeYuEOdql8vpa/9NsRSvu3P54jzX/LvmP9L8IKep8kiQpJkb6ZoRUvp7UvL20cq18\nErVHWvSWNLS0NOE2aftEyRPvO6+Li0vaIJu/M4qLi9OZM2f02muvqW3btjp79qzi4uKS5ZsyZYrC\nwsK0bt06RUZGqkmTJnr99dfPpbdq1UqtW7fWqVOnNG/ePBUpUkTr1q3z266/fuEiMGBY4WPf8qwU\nOhV5JY9HGvewNOXpRJ19Ugu0UiV0UgvO7YvSSbXTHeqkZjqtU+f2H9Ip3a1xaqgftFVHE9WzP1p6\ndJNUaak04YiSceaM1GeQVPpy6eEnpPWbkueRpLiz0obh0tj60g9VpBW9pbNHfed1cXEJjuyujN55\n5x0ZYxQSEnLu9+6772rnzp0KDQ3Vrl27zuXt06ePSpcunep3RpUqVdLo0aNTbDe7KqNAvHavBuo4\nwmKMyYG1gsuW80BjjBX17DH47gq4oz9Uu/tc+nEms4MOVGMm+bCHEEMMr9CJnWxlML9ThGIACPEF\ny/iIRfTnFh6iRqK2ZhyHbtvgsvzwRThUyJNYlqhT0G8I9B5k/d69/QJUrZxcZgkOLILVX8KuKVC1\nFVz+NBT1/ZmAi4tLAGT374yyiuz6nVEgBgxTgJ+MMbcYY27BGi9MyVyxMoC8RaD5CPijM0TtP7e7\nMHdSjl5s4U5i2AlAbnLTi++px3U8RCP2sgsAg+E5rmISD/Aa83mCaZzmvI/YWwrDqrpwRX7rDbz3\nXojzOsehBeDVZ2DLQquErrnbvlfasSuxqMZAmWvgtlHw8BrIWwLG3wQTb4eI38GTraNHubi4uKSf\nQGZGIUAX4FZn1zTgO1mDg2zHuZlRAnPehr2L4JHJieyqD9CHwwyiBvPJSYlz+7+lF8P4gqFMpjrn\nJ38niKYr01nFIcbQjFoUT9Tu5jPw5HY4FAuDLoGrfRizHD1mZ0kDvoeWzaxHh/J+fNvFR8OWMXa2\nFB0JtZ+ESztCnqJp6xcXl/8a7szIN9l1ZhSQO6BzmY0pBpSXtCrzREofyZSRJw6G3wCXtoSrn0+U\ndw+vcZKZVGMGOQg9t/9XfuBDXqQfY7iG8zEEhRjKGl5hHj25gY5chvGyTpdg9BF4McKahX9cEYr4\n+JLr8BH4bCB8NwraPACvdYcypXwfT8IS3tr+dpZU5SG7hFe8ju/8Li4uFlcZ+eaiVUZOpNfmQE5g\nKXAQWCDp+ZTKZRXJlBHA0W0w7GpoNQ3KXHFutxA76UwMu6nCRELIfS5tPtN5jtb04Eua8Uii6tZz\nhIf5ndoUZxC3UYjEL4uOxcHrO+G3SPisErQu4TsO0oFD0PMrGP4zdHwEXn4SSpZIni+B0wdsfKW1\nX0OhKlYphd8HOXIF3j8uLv8VXGXkm4tZGS2XdKUxpjNQQVIPY8wqeYV+yE74VEYAq3+Avz6ETksh\nV/5zu0Uc23iQEPITzg8Yr9do61lFJ+6mPd3pwkuJZkFniOVF5vAnOxjN3dSnTLImF52EbtuhcA4Y\nUBkuzZ8sCwB79llPDqPHwxNt4KVuUCyF5bj4WIgYD6u/ghNboXZXuPRxyF869f5xcfmv4Coj32RX\nZRSIAUNOY0wY0BL4PZPlyTwubwNhV8H0FxPtNuSkMj8Sy2528zzi/Em6lDr8wkJ+ZTg9eIZ4zr8m\ny0cuBnArPbmBuxhHb5bgIfEJvrogLL4cWhSDG9fCqzvglI83beXCoP/HsPxPOBwJ1a+Hdz6H4yeS\n5wU7E6ryINw3B+6eBFG74MeaML0N7P/bLu25uLi4XEwEoozeA6YCWyQtNsZcAmzOXLEyiTv6w7ap\nsHF8ot0h5OMSJhDFbA7wcaK0MMozhvlsZT3daMEZTidKf5Dq/MOjjGUTzfiVQ0nScxh4OgxW14U9\nMVBrhV2+86UwKpaHbz63boYidkHVa+HDvnAyyv8hFa8DjQdBm21Qsh7MeBR+aQgbvoe4s8F1j4uL\ni0tWEZQBw8WA32W6BHYvhJ/vh07LoGBiU7ZY9rGR6ynDa5QgcUTzhG+RItjMd0ykOCWTlI3nLf5i\nJOsZwV00oYLP5mcdh6e2Q+U88FVluCSvz2wAbNxio85OnwcvdoWnHoMCfpb6EpAHdk6xMZYOLYWa\nnewyXsGKKZdzcfm34S7T+Sa7LtMF8s7oSx+7jwNLJI33kZalpKqMAOa9BzvnQus/k7nRPstmNtOY\n8vSjKA8kShPic95kEmMYymTCqZqs6j+JoANTeJzLeYtryelj8hnjgT774LO98GwYvFwW8qQwR127\n0SqlOQvhf09Ct3aQPxWlBHBsM6wdABuHQ9nGcNnTUO4m38YULi7/Nlxl5JvsqowCWabLiw0Rvtn5\n1cGGd+hkjOmbibJlHte/bj/k+btXsqS8VKMKv7OLrpxgeqI0g+F/fMjj/I+W3MBy/k5W/nbCWUZb\n/mIvtzCW3SR35Z47BF4pB0vrwLIouHwl/HnMv7i1a8BPg2DaT7BwCVS5FvoMgjNn/JcBKFINru9j\nPYdXuB3+ehZ+qg1r+kOMn/dRLi4uF4b+/fvToEED8ubNS8eOHVPM26dPH8LCwihSpAidO3cmNvb8\nx/dt27YlLCyMwoULU6VKFT788MPMFj1zSM1fEPA3kMNrOyewEMgBrMtKX0Z+5FVAHIuQepeU9i7x\nmXxCc7RSJRWlhT7TZ+h31VMJTdVvPtPjFK+P9LdKaYB+lR/ndA6/R0qVl0oPbZR2n01d9BVrpPs6\nSGF1pS++tb7wAsHjkfbMlqY+5AT/e1I64seZq4vLxU7AY0EW8euvv2r8+PF68sknUwyuN2XKFJUp\nU0br16/XsWPH1KRJk0SRXNeuXaszziCwceNGlS5dWlOmTPFbn79+4SJwlLoRKOy1XRjY6Pyf7Rym\nBnUBrvlRGlBdio7ymXxMf2ilSum0VvlMX6nFaqgwfa9+fptYqD2qrG/VTdN0WjF+852Ok97aYUOe\n99ojxQTgxXvZKusdvNyVUr8h0tkAFFkCJ3dLi96WhoVJv91kw6bHx6ZezsXlYiG7K6ME0hN2PCkb\nNmxQ+fLltXTpUr/1ZVdlFMgy3afACmPMUGPMMGA58JkxpgAkWce62Kj9CJS/FqY95zO5MHdRni/Y\nQlPOsiW6r9jRAAAgAElEQVRZeh3qM5b5DONLPuZlPHiS5bmGsiynLceIpgEjWcNhn23lywHvVYQF\nl8HkY3DVapifylLalZfD+GHw2xCYPBOqXgcDv4fo6FSPnNBy0PBdaBMBtZ+AVX3hh8qw9AP7ca2L\ni0v2YO3atdStW/fcdt26dTl48CBHjx49t++pp56iQIECXHbZZbzxxhvUq1cvK0RNFwFZ0znfGTV0\nNhdL2pupUqWDgAwYvIk+CYOvhJs/gZotfGY5zDfspyfVmUduyiVLP8oRunAvYVTgM4aRJ4lHBrDG\nD8NZx0vM4V2uoxt1E31EmyivYOwReGEH3FYYPq0EJQPwsvDPcujxGazbZP3edXgYcudOvdy541xp\n3ydtHQsV77QeHkpf6xo8uFycBGTA8GEGXdxvpN1Q4q233mLPnj0MGTLEZ3rVqlUZMGAAt99+OwBx\ncXHkzp2biIgIKlY8byYriblz59KiRQsmT55MgwYNfNaXXQ0YAl36ag587vyaZeVULgBZ/UxOU2D3\nIqlPKen4Tr9Z9qmn1upSxeqQz/QzOq2uaqGWulHHFOm3no06onoarnv1qw7rtN98knQiTnphuw3m\nN2CfFOcJ6Gi0YLF0W0spvIH03Ugb9C8Yzh6VVvSRRlaVxlwhrftOijmVejkXl+xEmsaCLCC1Zbq6\ndetq7Nix57YPHz6skJAQRUb6Hme6du2q559/3m99/vqF7L5MZ4zpCTwLrHN+3Y0xH2WOaswiyjWE\nBs/ChHZ+4zWU4RUKcy9buJN4kq+f5SUf/RnD5VzFg1zPbnb4rKc6xVhIa6pShCsYzmx2+cwHUDAH\n9AqH6bVg1GG4ejX8k9w4LxnX1oc/f4If+sGPv0HNG2DoaIiLS70sQJ4iUPc5aLURru4J23+DHyrC\nghfhePLVShcXl0wkpbDjvoiLiyN/IN9+ZDdS01bAKiDEazshuF6Wz4L8yOv3iSBF4uOk4Y2l+R/5\nzeKRRzvUVRvVWPEpzGoGq4+uUTmt0bIUm5yi7QrTQL2heYpR8nDDidr2SMMPSmGLpce3SIeCmO3M\nWSA1vl+qco007CcpNg2GCse3SQteloaUkCY2taHS41MW2cUlS0nzWHCByIiw4wcPHtTo0aMVFRWl\n+Ph4TZkyRYULF9Y///zjt11//cJFYE23CijmtV0sK5URkB9YDNzlJ93vSUiV4zvtct3uRX6zeBSn\nbXpEm3WPPClYx03Sz6qnEpqtySk2uV9Raqqfda1GaruOpSri0Vip+zap1GJp0P7Al+4kadZf0o33\nSVWvlb5Po1KKPS2tHyb93EAaES4t6ymd9r1y6eKSpWR3ZZQRYccPHTqkxo0bq2jRoipSpIgaNGig\nCRMmpNhudlVGgXhgaAX0BGYBBrgReFXSTxk1OwsGY8y7wEnsN06TfKQrtWNKkfU/w6xXodNyyOMj\nQh4gYtnK/eSgMOGMSOTp25ulLKAbLXiOd2lNF79NehB9WUpP/uErbuZhaqYq5spTNphfrAf6XwIN\nQlMtco7ZC6yhw94D8NZz0PoByOkj7lJqHFwMawbYZbzw5nDZk1CqoWvw4JI9cD0w+Ca7GjAEY02X\nYJrxj6T9KeUPoL7BwD3AAXmFojDGNAX6Yj1DDJb0SZJytwLFsV4hDkv6w0fd6VNGAL93BsVBs2F+\ns3g4wxaakpfLqEA/v5ZxEWzhMe6iKQ/wPz4iJIXXdMs4QCv+oBHl+JKbKUDKJnQewYhD8OpOG8zv\nwwpQPMDYRtJ5pbT/kFVKre5Pm1I6ewQ2DIW1A20k2tpPQrVWkDNf8HW5uGQUrjLyzUWnjIwxKRqq\nS1qW5kaNaQREAcMTlJET3nwTcAuwF7sU94ikDcaYtkA9oBDWL15t4LSk+33UnX5lFHMKhlwF179p\nQ0/4IZ4TbOImCtGUcvh3wRHJYZ7gPkpTjl58Tx78e0eNIobuzOQv9vIjd1OP1IMUHY2Dt3dZc/AP\nK8BjpSAkwEtKgll/WaV08IhVSo/clzallOCkde0AG522RnvrpLVwchd+Li6ZjquMfHMxKqNZKZST\npJvT1bAxlYCJXsroGqCHpDud7Veddj7xUbYddmaU8ct0CRxYCaNuhfYLoFg1v9liOcQmbqQEnSjN\nS37zRXOWF2nPAfYwiN8oRgohXYHRbKA7M3mNq3mWeoT4mXl5s/wUPLkNhA3mVy+IpTtfSqnV/ZAj\nR+B1eHNim41Iu2EolKxvl/Aq3gUhaazPxSVYXGXkm4tOGWV6w8mVUQvgDkldnO02QENJ3YOsV40b\nNyY8PJzw8HCaNGlCkyZN0ibk4n6wahh0WAA5/H89GsNuNnEDZXg9WegJbzx4+IzXmcI4hjLJp9dv\nb7ZznNb8QRHyMIymlKZAqiJ7BMMO2bDnLYrBBxWhaBCzHAlmzrdK6VAkvP28nSmlVSnFnYEtY+xs\n6cxBO1Oq2RHylUy9rItLenCVkW8S+mX27NnMnj2biIgIIiIimDNnTpYqoyyznAAq4WWVB7QAvvHa\nbgN8mYZ6lWF4PNKYe6VpL6Sa9Yw2aZXCFKmfUs07UoPUQGW0RH+lmjdGcXpT8xWmgZqkbQGJLUlH\nYqRuW6XSi6UhB6T4IKzuJHvo0+dK1zeTalwv/fCz5MPqNCgOLJZmdJC+KyJNbyvtW2jbcXHJDDJ0\nLPgX4a9fyO7WdJmFn2W6dyQ1dbb9LtOlUq8y9JjORMJ3V8CdX0PVu1LMeppVbOE2KvE9hWmaYt45\nTOFF2vEu/bmbh1IVYy67acsk7qcaPbmBvAQ23VkaZa3ucgADLoErUp9cJUKCGfOgx+dwJBLefgEe\nvjftMyXwMnj4GnIXcgweWkOui/A7PZfsizsz8o27TJe0YWPCscrocmc7B9ZD+C3APuAfoJWk9UHW\nm7HKCGDnPBjXEjotTRYdNilRLGQbzbmEXwmlUYp517GSztxDe7rThZf8WuQlcJSzdGEamzjKKO6i\ndirvnRLwCAYfhDd3wUPF4P0gl+7AKqXpc+GdXlYpvZXO5TuwBg+7/rTm4fv/ghrtoHY3KFI97XW6\nuCTgKiPfZFdllJbltTAgT3qmY8AorMVcNLATeMzZfydWIW3GfsuUlrp9TkHTzdx3pRE3BeR24Lj+\n1EqV1KlUPDBI0l7tUlPV0Rvqqlil/hWqRx4N1iqVUH8N0HJ5FPg615EYqauzdDc4DUt3kl1Wmzbn\n/PLdiLFp+3g2Kce3Swtfk4aWkibcKm0d54a0cEkfmTYWXOT46xcutmU6Y8x0oArwiyT/5mNZRKbM\njMD6rBt5C1S+DRq9kWr2o/zCLp6mOrPJS40U857kBE/TkhBC+IqfCMX3x7bebCKS1kyiHKEM5nZK\nEPga19IoeGq7/b9/ZbgqCKu7BOQYOrzzefpNwr2Jj4atv1iDh5M7oFYXuLQzFAhLX70u/z3cmZFv\n/jUzI+cgDFA7K7VoCrL5fhzICI7vkvqUlnbODyj7YQ3VKlXQWW1PNW+MYvSKOusuXaF92h1Q/dGK\n08uao3L6WtMUEVCZBOI9dnZUerGdLR0J0rN3Ah6PNGOedMO9UrXrpOFjMmamJEmHVkizn7AGD1Nb\n2ii1rsGDS6Bk6liQAfTr10/169dXnjx5UvTaLUm9e/dWmTJlkrkDkqTQ0FAVLFhQBQsWVGhoqHLk\nyKHu3bv7rctfv5DdfdNZGSkA1ARqAAWyUuAAZPV7EjKEjROkLytKp/2HifDmgL7UGlVRjPammtcj\njwboY12nClqnlQGLNF0RKqev9ZJmKzoVh6tJiYyVntpqfd19sz9tS3eSVRIz56ff950vzh6TVn0l\n/Xip9GNtaXV/Kfp4xtTt8u8luyujjAo77k1UVJQKFiyo+fP9PzBfdMoICAVewBoSbAUWAn8D27He\nEZ4HQrNSeD9y+z0JGcbUZ6WxDwT8mL5PH2qtavmNhZSUCfpRV6mk5mhqwCId0indq19VT8O1QUcC\nLpfA8ijputVSg5XSPyeDLp6IWX9ZL+FVr027l3BfeDzS7pnSlAelwUWlOd2kw74jwru4ZHtllEBG\nhh0fNmyYqlSpkmJ72VUZpRTPaDxwAhtMr4qkayVdI6ky1q9cFPBb+hcKL0Ju/gSObYdlXweUvQyv\nU5jmbKEp8RxPNX8zHmEAv/Ai7fiJwQG1UYL8/Mq9PE4dGjGa71iFCHy9/IoCMK82PFUGmm+ALlvh\ncGzAxRPR5DqYPQ6+/RyG/WTjKQ37KfB4Sv4wBsrdBHeMhZarIV8p+KMp/HoDbP7Rvm9ycfm3EUjY\n8QSGDx9Ou3btLqR4GUaWmXZnFplmwJCUI5tg+PXw6EwodXmq2YXYxdOcYRXVmEpIAAYH29hER+7i\nbh7mRd5P0cmqN+s4Qmv+oCpF+IbbKEZwHkuPxUGPXfDjYXivAjxeGnKk47XmnAXwbm/YsRvefA7a\ntIBcATp0TY34WIiYYJ20Rq6x3h1qPwEFK2VM/S4XL4EYMCwLwM1WINQL4sEvKRkVdnzHjh1UrVqV\nLVu2UKmS/xsguxowBDS6GWPqGGOaG2MeSPhltmDZnuLV4ZZe8OvD1rFqKhgMFfiKPFzCVu7HQ+qP\n8ZdQnV9YyN/M4jkeJZqzAYlWi+L8TWsqUJArGJFiNFlfFMkJX1S2EWZHHoaGq+HvACLM+qPxdTDz\nZxjaB374xc6UhvwIsWmceXmTIxdUaQHNp8O9syHuNIytB5Oaw47J9lsmFxd/1EMZ8stMQkNDOXHi\nfHTp48ePY4yhYMHEVrcjRoygUaNGKSqi7EwgYceHAEOw7nqaOb97Mlmui4M67aDMVTDtuYCyG0Ko\nxGByEMp2WiFSX7cqTklGMgMPHh7lFo5wKKC28pKTPtzEN9xGa/7gDeYTi++Q6v6oUwDm1obnw+CB\njdBxCxxMhwK58VqYMRaG9bXh0Gs0gu9GQkxM2uv0pmhNaNQX2u6EyvfCP2/CqGqw/DM4czhj2nBx\nudAEGnZ8xIgRdOjQ4QJLl4Gk9lIJG8Quyw0TAv1xoV9anj0h9a8qrR0dcJF4ndVmNdV2tZVH8QGW\nidenek2NVUVbtCEoEfcrSnfqFzXQD9qswKwAk3I8Vnp+u1TiH+mrvVJsBphYz18k3dZSqlRf+vp7\nKTo6/XV64/FI+xdJM9pL3xV2/eH917jgY0GQZETY8QT++usvhYaGKioqKtV2/fUL2dWa7lwGGAbU\nykohgzqgrLgA9y6RepeUjgbuyDRep7RRN2iHugXlReEnDVZ9ldbfmhOUiB559KWWqoT6a4hWB9Wm\nN2tOSTetkequkOZlkHn1gsXSHY9IFa+SBgyTzp7NmHq9OXNYWv659EMVacwV0tpvpJjU71uXi5js\nrowyIux4Ak888YTat28fULvZVRkFEna8CTAB6y8uGvvBq+QVoTU7ccEMGJKyqA+sGw3t5tsXGQEQ\nzwk2cwsFuZmy9EzVN10C85nOc7TmdXrxAG2DEnMNh2nFH1xKMQZxG0VTCPTnDwnGHIGXdkCTQvBp\nJQjzH2EjYBYtg3d7weoN8OrT0KkV5A1evBSRB3ZNswYP++ZB9UetP7yil2ZsOy5Zj+uBwTfZ1YAh\nEGW0Bfu90Wrg3OtgSTsyV7S0kWXKSIIxzaBkbWv6HSBxHGETjSlGa8rwesDlNrGWTtxDC9rzLD0C\nVmQAZ4njFebyK1sYzp00oULAZb2JiocPd8O3B+G1ctC9DOQKzOAvRRavgPd6w/I18MpT8PijGa+U\nAE7uhHXfwIbBUKSmVUqV70sxdJXLRYSrjHxzMSujhZKuvUDypJssU0YApw7B4CvhniFwye0BF4tl\nH5u4kZI8QykCjyV4iAM8TnMqU52efEce8gQl7mS204mptKMW73E9uUmbC+5NZ6D7dtgZA19VhlsK\np6maZCxdCe/1gSUr4eUnoUsbyBeclXpAxMfA9l/tbOnYJri0k/WJF5o2He2STXCVkW8uZmU0ACgC\nTITz9siSxmWuaGkjS5URQMQsGP8odFoGoWUCLhbNDjZzI2G8Q3EeC7jcGU7zAm2J5DCD+JUiFAtK\n3IOcphNT2ccpRnEX1YMsn4AEE47CcxHQIBQ+rwQVg9ONflm+2s6UFi2H/3WDJ9pC/kyKfRS51sZZ\n2jwSwm60s6UKt4HJgBmfy4XFVUa+uZiV0VAfuyWpY+aIlD6yXBkBzH4L9i6CVlOCGsXOsonNNKE8\nX1A0gIB7CXjw0JNXmM4EhvBHquHMkyLEQFbSgwV8TCM6cXlQy37enImHT/bCV/vhxTB4sSzkyaCB\nfMUaeL8PLFgCL3WDru2gQCYppdgo69Vh7UCIOeGES38M8hbPnPZcMh5XGfnmolRGTsC77pL6XDiR\n0ocxRvJ4rO+YrMITByOaQPXmcO3LQRU9Hy12KIVJObJsUkbyNX15h4H8Qn2uD6oswFoO05pJ5zw3\nFA/Sc4M3287CCxGw9gx8EQ53FU2tROCsWmeV0rxF8GJX6NYeQoOMYBsoEhz8xwYAjJgA4c1sZNrS\nV2ftJeaSOq4y8s1FqYwAjDH/SGp4geRJN8YYaXMEVM3ir5CP74ShDeCh8VDumqCKnmIRW2lGZcZQ\nkCZBlZ3DVF6kLT34kmY8ElRZgGjieI35jGUjw2jKLaSvHycfhWcjoGY+6BsOl2SgIcKaDfBBX5j1\nFzzfBZ56DAqmITZToJw9AhuG2dlS7oJ2Ca9aa8iViW26pB1XGfnmYlZGfYBcwE/AOb83kpZlrmhp\nwxgjDf4ZOrbIalFgw68w/QX7/ihfcFODk8xmOy2pwkQKcHVQZdezis40oxVdeIrX07Tk9icRPMZU\nHqUmH9AozcYNANEe6LMPPt9rHbG+UhbypyNceVLWbYQPv4Bpc+HZzvB0RyhcKOPqT4o8sHu6VUp7\n51qFVLsbFKuVeW26BI+rjHxzMSujWT52S9LNmSNS+jDGSO1fg2EfZbUolqnPwMm90OLnoNd1jvMH\nO+hIVaaRn+A+6zrAXjrTjJrU4UMGkZvg7ZUPcZrO/MkuTjKKu6hJ+l6Y7Iq23yYtioI+leC+Yhm7\n1LVxi1VKk2fCM52geycokkGWff6I2gXrvoX130HhalYpXfKAax6eHXCVkW8uWmV0sWGMkcJvhe3T\nsloUS9xZGHYdXNEZ6j8ZdPGjjGE3z1GN2eSlelBlT3OKZ2nNaaIYyC8UokjQ7QsxiFW8yXw+pBFd\nqJNm44YEZh6HZ7ZDudzwZWW7hJeRbN4GH30JE/+EJzvAc49DsQx8Z+WL+FiIGO94D1973jzc9R6e\ndbjKyDfZVRkF4ii1sDGmtzFmifPrZYzJ5OdNv7IYY8wHxpgvjTH+XQ9EnYadey+gZCmQMy888BPM\n6wH7VwRdvCgtKctHbOZWookIqmx+CvA146jB5bTgOnaxPej2DYau1GUej/A1K7mf8RzmdND1eHNz\nYVhRxxo13LAG/hcBJ4Pz4Zoi1S6BoX3hn8mw9wBUux7e+BiORGZcG0nJkQuqPAjNZ8C9c7y8h98D\nO/4ATwYen8u/g/79+9OgQQPy5s1Lx47+jZPXrl1L06ZNKVmyJDlyJF7fjomJoXPnzoSHh1O4cGHq\n1avHlClTMlv0TCEQo9shwEmgpfM7Afgy974Q3AuUB2KA3X5z3Vgf5i29UDKlTrFqcNsXTriJqKCL\nF6cDZXiFzdxCDHuCKpuDHLxNX9rwJA9yPctYGHT7AJc6YSmqUZS6jODPIBVjUnKFwHNhsKYuHI6D\nmsvhh0PWei2juKQSfNcLlk6Fw5FQvRG8+iEcymQP3kVrwPV9oO0uqPwALH4HRlWFZR/D6YOZ27bL\nxUO5cuV466236NSpU4r5cuXKxcMPP+wz3lFcXBwVK1Zk3rx5HD9+nPfff5+WLVuyc+fOzBI780jN\neR2wIpB9wfyAwcABYFWS/U2BDcAm4BUf5V4BHnf+H+unbumL4dLjbyvbMbGjNL5tmovv0ydao5qK\n0cE0lZ+h31VPJTRRP6VZBkmargiV19d6TjN1RhkTU3zBCemqlVKj1TYEemawY5fU7RWpaE3ppXel\n/WnrxjRxYLE0s6P0XRHpz0ekPXNc7+GZDdncUWoCqYUdT2DLli0KCQlJNV+dOnU0btw4v+n++oVs\nHHY8gTPGmEYJG8aY64Ez6dSBQ4E7vHcYY0KAfs7+2kArY0xNJ62tMaY3sBdIiLXrf+Hjxvowd0k6\nRcwEbv8S9i2BVd+nqXgZXqYoD7GF24gjecjh1LiZuxnBdD7mJfrzUVBhyb25hUqspB27OElDRrI6\nwBhLKXFtQVh0ObQtCXesh6e2QWQGBN/zpmJ5GNATVs2As9Fw6Y3wQg/YdyBj2/FFqfpw02Bosw1K\nXwNznoCfLofV/SA69Uj0Li4BceDAATZv3kzt2rWzWpSgCUQZdQX6G2MijDE7sAqja3oalTQfko2m\nDYHNknZIigVGY5flkDRC0gvAz0BTY8wXwBy/DVxeHQ4cgQPZLKJa7gJw/xiY8RIcXp+mKsJ4l4Lc\nzBbuJJ7gw6/Woi7j+Jsp/MLLdCSGtEW2K0Y+xtKM57mKmxnLlyxLs3JLIIeBLqVhfV27felK+OYA\nxGfwO+jyZeGrD2HNLPB4oHYT6P4m7NmXse34Ik9RqPMsPLIObvgK9s2FH8Jhdhc4tDzz23dJTGVM\nhvyyA3FxcbRp04YOHTpQvXpwxk7ZgYCt6YwxhQAknUgtb4D1VQImyglFYYxpAdwhqYuz3QZoKClw\nz6G2nBo3bkz4ln2EX9uAJk91pkmTJhkhcsax/FtY8hV0WAS5gjclE2IX3TjLBqoyiRCC94mTYGkX\nxQkG8kvQPu282cJR2jCZouRhKE0pQ8a4Q1hxCp7eDmc91gHrtQVTL5MW9h+EzwfCkNHQ6j7rKbxi\n+cxpyxen9lnP4eu+gfxlrXl41ZaQMxOcwv6XuFis6d566y327Nnj852QN1u3bqV69erExydfFJJE\nq1atiIqKYvz48ckMHbxJ6JfZs2cze/ZsIiIiiIiIYM6cOdnbtNsYkwcbcjwcyJmwX9J76Wo4E5WR\nJPhsMOzcB1+9mR4xMwcJfmsNeYvAnQPTVgUedtCeOA5zCb8REqTHboB44vmYl5nFHwzm96B92nkT\nSzzv8zffsppB3ErzdNTljQSjDsPLO+D2ItCzIpTOpG94Dh6G3oPg25Hw4N3wWncIv4Ceuz1xsGOS\nNQ8/tARqtINaXaFItQsnw7+J/5Iy6tixIzt37mTSpEnkzp3yDXLRmnYD47HLZXFYDwwJv4xmD1DR\na7u8sy9t3Fgf5mYjizpvjIG7BtlvodaPTVsVhFCJoYSQn+20QsQFXUcOcvAmvXiMZ3mIRixmfppk\nAchFDt7jen6mGc8yi65M4xTpf+ljDDxaEjZcCSVzwWUroc9eiPWkXjZYSpWAnm/AxnlQohhcdQd0\negG2RmR8W74IyQmVm8M9k6HFIgjJBb81ggm3wtZf7LdMLv8e4uPjOXv2LPHx8cTFxREdHe1T0QBE\nR0cTHR2NJKKjo4mJOb+83rVrVzZs2MCECRNSVUTZmtQsHIA1mWE5gZ1prfbazgFsASoBuYEVwKVp\nqNeahsTESAXrS0eO+rQcyRbsWWzDlUduTXMV8YrWZt2pbXpUHsWnuZ7ZmqKrVFK/aWSa60jgmM6q\nrSaphgZrifanuz5v1p+Wblsr1VouzTiWoVUn40ik9PanUvFLpXbPSBu3ZG57vog7K20cKY1rJA0L\nkxa9JZ3clXo5l+xvTRdo2PGIiIhE+Ywxqly5siRpx44dMsYoX758Cg0NVWhoqAoWLKhRo0b5bddf\nv5DF1nSBDO7fAJdnaKMwCmsZFw3sBB5z9t8JbAQ2A6+mse7zvXtbJ2n8DL8nJVuwqK80uL4UF53m\nKuJ1WhvVRDvURR6l3V54vVbpelVSX72TrnoS+FHrVVL91VOLFJcORZkUj0cad1gKXyo9tFHacTbD\nqvbJ0WPSe72kErWkR5+U1m3M3Pb8cXi1NPcpaXBRadK90o4pkifjuvVfR3ZXRllFdlVGgbwzWgdU\nBbY7ysM4QgfnLO0CkSie0Qdfw7ET8HlwYRwuKBKMvQ+KVYVbe6W5mnhOsoXbKcA1lKN3ml32HGI/\nnWnOJdRIU/TYpOzkBG2ZTAiG4TSlAhnnwfR0PHy6F/rttx/QvlQW8mZiELwTJ6H/UOjzDdzcCN58\nDi6rmXnt+SM2CjaPOh9rqdYTULMj5Ctx4WXJzlws74wuNNn1nVEgysindy1JOzJFonSSSBnNWwIv\nfAqLx2StUKlxJhK+uxKa9odq96S5mjiOspmbKcw9lOX9tIvDaV6gHYc5wCB+pRjpG+Xi8fAZi+nD\nMr7iZlpSI131JSXiLLywA1aegj7h0Kxo5sYaOhkFA7+3xg6NGsJbz0PdLPisQ06spbUDYftvUOke\na4lX5jo31hK4ysgfF60yuthIpIzORkOJ6+zHHAUzKfpaRrHrL/ilBXRcAoXSblccyyE205hitKMM\nr6a5Hg8ePuU1pjKOwfzBJUE6afXFEvbzKJO4lrJ8yU0USuesKynTjkH3CAjPYwP6Vc9k0+hTp2HQ\ncPhsIFx9pVVKV9XN3Db9cTYSNn5vQ6bnyGMj01ZvA7kzMZRGdsdVRr7JrsooExc1sgF588BVtWHB\nRfA1YYXrocGz1uTbE7xlXAK5KElVpnGYbzlIvzTXE0IIr/IJXXiZltzA3yl8Yxwo9SnDMtqShxxc\nyQgWpMNY0he3FYGVdeDWwnDdGnhlR8Y6YE1KgfzwQlfY9jfcdD007wD3tIVFWRDpK28xqPs8tNpg\n/eLtmQkjKllPD4eD98/r4nLB+XcrI4DGDWDO4qyWIjCue8V6+Z6Xrk+4+D975x0WxdXF4fdSLBQ7\nKjawxhJLxN7AroktYmI3dpMYayxpJhqNxi+Jxl5i710Te4miscfeFQt2xV5RgT3fHwMGcYEFdncG\n5X2eeWSH2Xt+rLBn595zfycF2cnP3wTzC7eJfe9CXDSnE78zly/4iKXMStRYAK44M4majMAPf1by\nA6IZbEcAACAASURBVDsIw3p12ikc4MtsmgHrzVAodAjmWtmANTqpU0OPTnBuF7xfHT7qBHWaw469\ntosZE0pBjupQe4nm8uCWE9Y2gKXl4NRMCEuskVcyydgIS9aMXIEQETEppQoABYG1oln2GI5XpukA\nNu2EgeNh+xz9RMWHxzdhakloMAtyV0/UUM84QyB+ZGcEGRLQgjwqgZygA/VoSEt6MQgHK3yOucET\n2rGOezxnDnXJh/WbDu16pPVOSu2guTiUsMNs7YsXMHOR1lMpjxd83wt8K9g+bkxEbqY9MVFbYyrQ\nRit6SG/dpTvDkTxNZx6jTtNZkoz2A5WB9MAO4F/ghYi0tL28+PNaMnryFLJUhls7IHUq/YTFhwub\n4K9PtHblblkSNVQIRwmkJrmYTDoaJGqs2wTTmYZkx4tfmE4qEr8oIwhjOciP7OZnKtOedxPdvC86\n4QLTgmHAZWicAYbkhAzOVg1hltBQmLNUS0rZssD3vbUqPD2LCx6e1zrTnpoGGd7VCh68G2r9mN40\nkpOReYyajCz5eKtE5CnQGBgvIh+huWonDVxdNOPU3Yf1VmI5uWtA8Xawsg1I4qawUlOUvKzkEh15\nSOK632YiM/PYDEALqnGLxNtdKxTdKEkAHzOagzRhJXcSbQr/Ko4KOmWBkyW0rwsdhok3rG/AGh1n\nZ2jXDE5ug04toevXULEBrNts22nD2EiTB8oNg9aXoFAnODoG5njB3gHwKAm2wEnmzcGiZKSUKg+0\nBFZHnIvZhc+IGLWlRGxUGQihT2HX/xI9lCulycMygmjBI7YlaqxUpGYU86hETRpTjjMcT7Q+gCJk\nYi8tyENaijOLjYls3meO9E7aVN3GQjD/NpQ+CjusYvsbO05O0KoJHN8K3TtAnx+h7PuwaqN+Sckx\nJeRvBo22Qv1N8OIBLH4P1jTQpvSSO9MmY3fi2hUL+AJ/EdHsDsgDjNZzp24ceuU1VgWIVGv7+nmj\n8+CyyMgsIpd3WGc42SiHxUMeyx6rjLdEZoqPeMhWWW+V8SL5Wy5avXlfdEwmkfm3RHLsE2l1RuRa\nwg0w4k14uMiSlSLFqom8V0Nk+RrtnN68eCxyYorIYh+R2d4i+4eKPLGum5NdMfteYCDGjh0rpUqV\nkpQpU8baXO/YsWNSu3ZtyZQpk9nmeleuXJH69etLhgwZxNPTU7744gsJj+UXKqbXBaM7MCQ1Xlsz\nAnjwCHJUhTs7IakZCQaugnVdoeNBSJ3wNg+RPGA1F2lPPtbjQolEj7eHbXzBx/RkIC0T1+bqFe4S\nwqds4hR3mcv7FMXDamNH5XE4DL2q9U3qnw16eGoVefbAZIKVG+DHERAWrjk6+H8ADgaocQ3ep22m\nPb8MctXW1pY8qyStzbRGXzNasWIFDg4OrF+/npCQkBhdu8+cOcOOHTvIlCkTjRo1es1M1d/fnzRp\n0jB58mTu3btHjRo16Ny5M1988YXZ8Yy6ZmTJncYWYHP0Q88MGode8x8H3msssuOA+e8ZnY29RRY2\nsFqf6ruyWA5LVnkqx60y3gUJlKpSQAZLLwmTMKuMKSJiEpPMkGOSScbJSNkn4Vbwy4uJwKci9U6K\nFDggsvauzcKYxWQSWb1JpOz7IoWriMxdKhJmvZcxUTy7J3JktMj8QiLzCooc/l3kmZ1fn4QS43uB\nwUhs2/ECBQrI2rVrXz7u27evfPrppzGOE9PrQhJoO94H6BtxDEBz005iCzBo60ZJZb9RdKoOgyc3\n4N9RVhkuPU3Izv84Sy2ecTbR43mTj2Xs4gSH+JTGPOGxFVRqxQ2fUIQ9tGARp6nDUq5Zaezo5EsN\nKwtqdkLdgqDhKTj3zCahXkMpbX/SrlUwchCMnwGFfWHWIghL+P5nq5AyHRTtBk2Pg+8kuLkH5uSG\nLe3h5l791ryS+Y86deowb948QkJCuHr1KmvXrqVu3bp6y4o3CZqmU0rtFZEyNtCTaMxO0wEs3wST\nF8HayfYXZQ3uX4AZ5eDjVZCttFWGvM1kbjCUAmwjxSutpBLGC14wgM84xgGmsBJPrNcuNQwTQ9nD\neA4xnho0xnYd556b4Pfr8Ms1+DQLfJ0dXO1YsiMCATu16btLV+HrbtDmI+PMMD8NhtPT4fgkrY16\nkU8hf3NwdtNb2atYMk2nSLg5cVSELxP83MQ217t37x7Vq1fn6NGjmEwmPvnkk1jHSsrTdBmiHJmA\n2sBpPW/n4tBr/t40+I5ImtIiobZZELcLJ5eKjM0tEmK9Hk03ZaQck3zyQq5ZZTyTmGSC/CzlJYcc\nlf1WGTMqu+Sq5JUp0l7WySOxbdXBlWciLc6I5NwnsuCW1WZJ48W2XSI1PxbJ5SMyfobIMxu3y4gP\npnCRi2u1dhZT02vtLW4f1VvVf8T4XmAwEjtNV7p0aRk2bJiEhobK3bt3pWHDhtKvX78Yx4npdSEJ\nTNPtR5uW2w/sAr4EOlg7KdocjwyQIwscOqW3koRTsDHk+wBWd7Ta/EhmepKRdgRSg1BuJXo8heJT\n+jOA3/mE2mzgTyuo/I9yZOMgrQEowWx2c82q40cle0qYm187hl2FaifgqC16HMdC5XKwYSEsnKSV\nguctD6OnQIgBbH2UA+SqA3VXwMeHIWVGWF0blleCM3MhzE7TnG8zt2/fZt++fXTt2hUnJyfSp09P\nu3btWLt2rd7S4k2cyUhEcotInoh/84tILRFJeH9qPfEtnfT2G0Wn+q/alN3+8VYbMivfkI4POUst\nwrhnlTHr4s801vA9nzOFEQjWW1xwJwVTqc3/qEIj/mQQO63qbxedymlgfzH4OCNUP6HZC92z81pO\nOR9YPQf+nA6bt2tJ6beJmsGIEXDLCWUGQcsgKN5bcxCfnQt29YMHiV+WfCOxRtvxTJkykS1bNiZO\nnEh4eDj3799n5syZFC+uk318YojplgmoFvFvY3OHnrdzsR3Edms+f7VIw64xfz+pcCdQa1d+3XrV\ngSYxyWXpKSelrITJQ6uNe0UuSm0pKt9IF3khL6w2biRX5ZHUlMVSTubKWbF9i/nbL0Q+OyeS5V+R\nSTdEwnSYuhMROXRMpElHkczvigwbLfLwkT46YuN+oMjOviLTPET+qilybqlImPV/BWIk1vcCA2CN\ntuMiInv27JFKlSpJunTpxMPDQ5o2bSrBwcExxo3pdcGo+4yUUoNE5Ael1HTzOUzaWzsxWoMYCxgA\nrgVD0YaaT50RNnMkhuMLYOsA6LAfUlqnaY0gXOYznnGSfKzFARerjPuIh3SnGWGEMY5FpCGdVcaN\nxBThbzeY3QynMu1s4G8XnYNPoPsFeGrSXB0quNs0XIwcPw0/jYKNEe4O3TpAurT6aImJsGdwfqnW\na+nReSjYEQp31O6mbInR9xnphVELGN6OTa9RyV8Hlo3W/OqSOmu6wPOH0Gie1XYjCiYu0pZQbpKX\nv3CwUgO8MMIYQm928jdTWUVOcltl3Kgc4zYtWUM+0jGZmmS0gpFrbIhotkL9LkG1NDDcCzx1qng7\nfVYzZF29CT5vCz07QQbrm6AnmjtH4cQkrW26Z2Uo/Km2qVbZ4LNhcjIyT5JNRkqplIA/4A04RZ4X\nkcQ13UkASqnswBjgLhAoIsPNXBN7MurwHZQsDF1b2Eyn3QgN0cq9S30B73Wy2rBCGBdogfCcPCxB\nYT1L55mMZTxDmcBSSlLeauNG8pwwvmU7CzjNNGpTC2+rx4jOo3D46QpMCba/i0N0zgXBsDGwfC10\nbgm9u4BH4rrG24TQxxC4QHN5eH5Xa2lRsB24JM6k/hWSk5F5jJqMLPmT+RNoCIQBT6IcelAMWCIi\nHSGBXjZJefNrdJxTw4cLIeAbuHnEasMqnPBmDoKJC7RCsJ5r5id8wc9MoTMNWckCq40bSUqc+BU/\nZlKHDqynB5sJwbatt9wd4Wcv2PUubH0IRQ/DOuvUgcSbvN4w5Tc4sB7uP4R3KsOXA+F64g3WrYqz\nmzZV99F+qLUIHgTC/HdgYzO4GpC8mfZtxJI7o2Mi8q5Vgyo1FagH3BSRYlHO1wF+R0uSU6Pf+Sil\n0gArgVBgtojMNDN27HdGQVehXDO4vi1pGW3FxtHZsGMotP8XUlhv56GJZ5yjPs5kx4tpKCs2Bj7F\nUTpSn49pTzcG2GSN5y4hfMYmjnOHubxPcTJbPYY5Vt+DnkFQKDWM8NLcHfTiyjX4ZTzMXgqt/KHf\n55Ajm356YuP5fTgzW1tbEpO2mfadNtrG2oSQfGdknqR8Z7RTKVXUynGno22efYlSygEYG3G+CNBc\nKVUw4nutlVIjga7AtyJSAy2ZxR+vbJDCGQIvJkK+wSjaGrKX1wxVrYgDqcjDCp5zjst8YdXy7IIU\nZTm72cxqetGa51h/U0oGUrOAevSjNDVYwm/sw2TFnyEmPkivtT2v6A7ljsG3lzRDVj3IkQ1GDYET\nW7Vf+2LV4dN+EHRZHz2x8dJ66Bj4Tv7PemhzO+3r5LzyZmNJMqoE7FdKnVZKHVFKHVVKJWpOSLR9\nStEnMsqgrQNdFK2l+QK06UFEZLaI9ELrp9RTKTUBuJCg4Epp+43elKm6SGqPgev74PAMqw7riCv5\nWM1T9nGVvlZNSB5kZQEBhBFKS6pzxwqbbqOjULShCHtpyXICqcFiLmP7JkYpHaB/djhSHC49h0KH\ntGIHvd5Qs2aGX3+A0/9AhnTgUxs69IazCfsrsilKQbbKUHMeND8D6QvBphawxAdOTNbWm5J587Bk\nms7L3HkRSdStRcS4KyOn6ZRS/kBtEekc8bgVUEZEusdzXPH19cXb2xtvb2/8/Pzw8/N79aI/Fmub\nX2e/Vv+QtLl1HOb4Qaut4FHYqkOHcZdAqpKWhmTDurUrJkyM5Af+ZC5TWUV+rKs9kjBMDGcvozjA\nWKrzMe/YJI45djzUDFjdHGB0bijharfQZrl7D0ZNgXHTNZPWb7pDQdvZ/SUaMcGVTdoU3rUAyNdM\nm8bLWCzm5yRP05kn8nUJCAggICCAoKAggoKC2Lp1q67TdJZuJK0EtIv42gPIndgNToAXcCTKY39g\ncpTHrUhAEz8s2eh26rxIrmpxX5cUOThVZFIRkRdPrD70C7kpx6SgXJdhVh9b5L9mfdtkg03Gj2Sv\nXJcCMlVayxq5L/YzewszaRtlM/+rbZy9bccNoDFx/4HI4BEiHkVEmnYROXpSb0Vx8+iKyL+DRGbm\nEFlaXuTUTJHQp69fZ9F7wVtITK8LRvemU0r9APQHvo445QzMsVYyjMJVeMU6OkfEOetTwBuev9CK\nGd40ireDLCVgfbxuKC3Cmczk52/uMIVgrNPOIir+tGE8S+lNa+Yy0erjR1KarBygNS44UYJZbOeK\nzWJFxVFB5yxwsrj2daFDMP4GhOv44T1tGviuF5zbDSWLQo2Pwb8DHDyqn6a4cMsOpb6HVhfgva/g\n7AKYnRN29IZ7p/VWl0yCiStbofUvUsDBKOeOJDYLou1bOhrlsSNwFu2OKUVE3EIJGNeyjwcf9RSZ\nucKya5Mazx6KjC8gcnSObYaXIDkqXnJLJtlk/AsSKNXkHflRelq1WZ85/pKzklUmyDfyjzy3cazo\nHH4s4ntMpPghka0P7Bo6Rh4/ERkxUcSzuEi91iJ7D+qtyDIenBfZ9bXI9CwiK6qKBC40/p2RpW3H\nZ86cKT4+PpImTRrJmTOn9OvX77W24vPnz5dChQqJq6ur5MuXT7Zv3x7jeDG9Luh8Z2TJm/veiH8P\nRPzrmthkBMwDrgHPgUv8NwVYFzgNBAJfJXDsGP8TXmHMHJEO31l2bVLkxiGREZlEbp+2yfAhEihH\nJIfclpk2Gf++3JUWUk3aSz15ZEWvPHPckMfygSwTH5ktJ+W2TWNFx2QSWXhLa1PR7LTIZYO0iAgJ\nERkzVSRHSZHazUS279FbkWWEPdcS0Yqqxk9Gy5cvlz///FM+//zzWJPRxIkTZfv27RIaGirXrl0T\nHx8fGT58+Mvvb9iwQby9vWXv3r0iInLt2jW5di3mljBJORn1ASYB54FOaG0kuukpOg69Mf4nvMKR\n0yL5alt2bVJl3wSRycVFQkNsMvxTOSFHxFPuykKbjP9CXshX0knqSnG5KpdsEiMSrQ/TIckoY2W8\nHBSTDVucm+NxmMiAiyIZ94oMuSwSEh73c+zBs2cik2aJeJcWqeovsnm7Pn2dEoLRk1EklvYzimTE\niBHSoEGDl48rVKgg06ZNs/j5Rk1GlrSQ+BVYAiwFCgDfi8gYC2cBjUuRfHDnvmae+qZSsgtkfAc2\n9rbJ8KkpRF7Wcpnu3Ocvq4/vjDNDmcSHtMaf8hzGduX4Wh+m4uygOVM5Rj2Wc9OORiOujvBjLvi3\nKOx/AkUOwV939d9bkzIldG4NZ3Zo3Wa79IPKDWH9Fv21va1s27aNIkWKAGAymdi3bx/BwcHkz5+f\nXLly0a1bN54/f66zyvhj6Zb6o8A/wLaIr5M+Dg5Q2Qf+2a+3EtuhFLw/GS5sgBOLbBLCheLkZRWX\n6MhD1lt9fIWiE18yiHG0533WsczqMaLyDhnYRXPeIzMlmM1f2LcZT+5UsOwdmJgHvroEdU/CKQM0\n0nN2hrZNtc2zn7eFXj9AuQ9g5YaknZTULusc9mLatGns37+fPn36AHDz5k1CQ0NZunQpO3bs4NCh\nQxw8eJAhQ4bYT5S1iOvWCeiItq4zA5gJBAHt9bydi0NvHDepUfhtushngyy/PqlybZ+2fnT3rM1C\nPJLtclgyyUPZYrMYR2W/lJccMl6G2WUa7R+5LN4yWTrLBnlsg15McfEiXGTEVZFMe0V6XxC5H2p3\nCTESHi6yeKVIsWoiJWqILFmpnTMS8Xov0BFLp+mWL18uWbNmlePHj788d+/ePVFKyezZs1+eW7p0\nqZQsWTLGcWJ6XTD6NB3QF3hPRNqKyCeAD1qpd9KnSqmk3/nVEjx9oNIAWNYUwmxz++5GRXKzkAt8\nxGNs81HxXUqyjN2sZhH96cALXtgkTiSVyMFh2vCccN5jNnu5btN40XF2gF7ZNGuh+2FQ8BBMDwaT\nAe5EHBygST04uBEGfqk5hRerBgtWQAzNSpNJBOvWraNLly6sWrWKwoX/2xSeLl06cuTI8cq1Kol6\nblqSjO4Aj6I8fhRxLulToiBcvgG3dbJYtielukHaXPB3X5uFcKcaXszmPA15im2mP7OSnUX8wwPu\n0YZa3LPxr2IaUjKDOgylEvVZwY/ssmmLc3NkSQFT88FfBWHSTSh/DPY+ivt59sDBARrWgX/XwS8D\nYPRUKOILsxZBmJ1bsyc1LG07vnnzZlq1asXSpUvx8fF57fvt2rVjzJgx3Lp1i3v37jFy5Ejq169v\njx/BusR0ywT0jjhmAQeBgcAPwAFghp63c7EdxPfWvE4nkWUb4/ecpErIPZGxuUVOLrVpmHuyXA5L\nFnkqR2wWI1zCZaj0FT/JL+fENuXr0bkiD6WGLLJbi3NzhJtEZgaLeP4r0i5Q5MZzXWTEiMkksmmb\niO+HInnKivwxR+S5Thrj/V5gZyxtO161alVxdnYWd3d3cXNzE3d3d3n//fdfjhMaGiqff/65pEuX\nTjw9PaVnz57yPJYXPabXBQO3Hf8hjiQ2yDrp0LrE2UIiOsMmQ/BdGPmV7UQZiat7YVE9aLsb0uex\nWZi7LOAqvcnPZlJR0GZxFjKVX/iGMSygPFVtFicSE8IYDjLEji3OzfEwDIZc1abtvs4OX2TVr6Ff\nTPyzGwaPhNPnoP8X0L4ZpEplv/jJ3nTmMWoLCYvbjiul3ABExNCeufFORjsOQPehsH+J7UQZjb2/\nw7F58Ml2cLRdn+w7zOAaAyhAACnJa7M4u9hCN5rRl6E0pYPN4kQlaovzSdQgEy52iRud0yFa76Sg\n5zDKG2ql00VGrOzeD0N+h4PHoO9n0LkVuNjh5UpORuZJsslIKfUuMBvIEHHqNtBGRI7bWFuCiHcy\nevECMlaAK1sgrbvthBkJEVjyIaT1hlq/2zTULSZwk/9RgK2keMV60Lqc5wwd+IBafEh/fsbBio0A\nY+I5YXzHDuZziqnUprYdWpybQwRW34eeF+BdFxjhDXnseAdiKQeOaElp5z6tHfpnn4C79XpBvkZy\nMjKPUZORJX+xk4HeIuIlIl7Al8AftpVlR1KkgNJFtTuktwWloN50OPMnnF5h01AefIYH3QikOqE2\nrEbLQwGWsZvD7OUz/Hlqhw2rKXHiF3yZRV06sYHudmhxbg6loF56OF4CyrlDmaNaQ78nBqtqK1kM\nlk2DjQvhwFHIUxYGj4D7D/RWlowRsCQZuYrIlsgHIhKA5k/35lDFB7a9wZtfzZE6PXy4ANZ2gftB\nNg2Vhd5koC2B1CDUBg30IklPRmaxgbSk52Mqc8NGpu/RqUYuDtOGWzzFhzkc5KZd4kYnpQN8lR0O\nF9Om7Qrq3NAvJooWggUTYfufcDYI8pWHAcPhzl29lSWjJ5Yko/NKqQFKKe+I4zs0n7o3hzex86sl\nZC8L5fvD8qYQbts9O558S1oacZaahL3W5Nd6pCAFw5lKPZrSmHIctVGJeXTSk4r51OM7ylGbpQxn\nL+F2LgGPJHtKmJsf5ueHX65BleNw0H7ORhbzTj6YORr2rIEbwVCgEvQfAsG39VaWjB5YsmaUHhiE\n1mBP0GyBBomIITfnxHvNCOBpCHhUguB/wFWfhWjdEIHFDSFDfqjxm21DIVzlSx6zg/xsxJE0No23\njmV8SxeGMpnafGjTWFG5xEPasBYTwizq4k1au8WOTrjAtGD47jJ8mAGG5IRMzrrJiZVLV2D4OJi/\nAj75CPp+DtmyJny85DUj8yTJNSOllCPwrYh0F5GSIuIjIj2NmogSjEtqbQPs7sN6K7E/SkH9GXBq\nKZyxvtnpK6FQZOc3XCjJWT4g3MbrOnVozAzWMYjuTOR/CPZ5Y8pFGv7mI+qTlzLMZTYn7BY7Oo4K\nOmWBUyUgpYLCh2DsdQgz4Ht0rhwwbhgci1gUeLcqdP1aS1LJvPnEmoxEJBztjujNx7cUbH0LrIHM\nkToDNJoPazrBg4s2DaVQ5GQcKcnHeRpgwrYuoEXxYRm7WcUCu1gIReKIA30pzQaaMJy9NGMVd238\ns8ZGeicYlRs2F4Hld+G9w7DFoIUD2bLCyB/h1D/g5golakLHL+FckN7KkrEllqwZHVRK/aWUaq2U\nahx52FyZvXlbfOpiIkd5KNsHljeDcNtWhCkc8GIKTmTmPP6YsK3dfVays5Bt3OeuXSyEolKCzPxL\nSzxxoziz2IRtk31cvOsCmwrDoJzQ7hx8dBouGrTbQOZMMPw7CNwBnpmh7PvQphucCtRbWTK2wJJk\nlArNi64aUD/iqGdLUbpQ4T3YdwyeGfQv0x6U+xJSZ4SAb2weSuGIN7NQpOICzRAbl0S74sZEllGc\nMjSmPOc5Y9N4UUmNM79TlWnUpi3r6E0Az9DPuE0paJwRThaHoi5Q8ggMvAxPDVYKHknGDDC4P5zd\nBQXyQJUPodmncPSk3soSx7hx4yhdujSpUqWiffv2MV63cOFCChYsSNq0afHw8MDf359r1669/L67\nuztp0qQhTZo0uLu74+TkRI8ePezxI1gVix0YkgoJKmCIpPTHMKIfVC5lXVFJiad3YOp7UGc85Lf9\nZw4TLzjPhzjijjdzUTjaPKa9LYSicocQPmUTp7jLHOpSnMx2jW+OS8+h70XY8xh+9QL/DFrCMiqP\nHsOEmTBiEpT3gQG9tD1M0TF6AcOKFStwcHBg/fr1hISEMG3aNLPXXblyhRQpUpA5c2aePn1K586d\nCQ8PZ/78+a9d++TJEzw9PVm7di0VK1Y0O16SK2BQSm2I8vXX9pHzSvzcSqkpSqlFUc65KKVmKKUm\nKaVaWD1oFZ+3e6oOwCWjtn60ugM8vGzzcA6kIA9LCeM2F2mP2KEcuikdGMMCutGMhUy1ebyoZCQ1\ni6hHX0pRgyX8yr+YdCpuiCRXSlhYAGbkhR+vQPUTcNSApeCRuLtBv65wfjf4VYAGbeGDVprtUFKi\nUaNGNGjQgAwZMsR6XY4cOcicWfvQYjKZcHR0xNPT0+y1S5YsIXPmzDEmIiMT2zSdR5SvP7K1kOiI\nyAUR6RjtdGNgsYh0ARpYPWiVt7iIISo5K0KZ3nZZPwJwIBV5+JMXXOAyn9ul8qw8VVnEP0zkZ4bR\nD5Md9wQpFG0owl5asoKz1GAxl3lot/gx4ZcWDhQD/4xaQup2Ae7a31DCYlxcoEcnOLsT6tWEpl2g\nZlPYZsfOq/Zix44dpEuXjrRp03L58mWGDx9u9rpZs2bRpk0bO6uzDrElI6u8IyilpiqlbiqljkQ7\nX0cpdUopdUYpZWmzvhxA5Md1689wV/bRyrtDDfwXaC/K94WUaWHrd3YJ54greVnFUw5xhV52SUhR\nLYQ+pTFPsK8HcG7SspWm1MQLH+YwH/0XQZwUdM0KJ0to5d+FDsPEG9p+JaOSKpXmcxe4E5o3gva9\noUojy56rPK1z2JqKFSty//59rly5gpOT08u241G5ePEi27Zt45NPPrG9IBsQWzLKE1FFtzLK1y+P\neMSYDtSOekIp5QCMjThfBGiulCoY8b3WSqkRSr38L446h3kZLSFFP28dMqQD7+xwUP83Bd1RDtBg\nlubufXaNXUI6koZ8rOUxW7nGN3ZJSJEWQunJyMdU5jr23dTiiANfU5a1+PMju2nBau7zzK4azJHR\nGSbkgfWFNEuhUkfgH/1v3mIlRQpo31wrCe/cyrLnyHXrHPbC09OTwYMHM3v27Ne+N3v2bCpVqoSX\nl5f9BFmR2JJRQ+A34NcoX0c9LEJEtsNr/i9lgEARuSgiocCCiBiIyGwR6Q08V0pNAEpEuXNaDjRR\nSo0DVlqqIV5UKfV2WgOZwyUTNJoHq9rDQ/u8STuRnvxs5AErucFgu8RMQQp+ZgoNaEFjynEE+0/V\n+pCF/bQiA6koziy2cMnuGsxRwhUCimiedy0DofkZuGzwglMnJ2jVRG8VtiM0NBQXMz04Zs+eeInJ\npAAAIABJREFUTdu2be0vyErEmIxEZGtsRyLjZue/6TaAKxHnosa/KyKfiUh+ERkece6piLQXka4i\n8nopiTXwLfX2mabGRq7KULoHrGgOJvuUIzuRifz8zV3mcYP/2SWmQtGFvgxiLO2oy1qW2iVuVFxw\nZizVmUhNWrGWPgTwXMcS8EiUgqaZtKm7/KmgxBEYcgWe6WO998ZgadvxefPmcfmy9nZ58eJFvvvu\nO/z9/V+5ZufOnVy7do0mTZJuFo6tmm61UuojpdRrKTiiqq2pUmq1beUlDD8/P9q2bcvAgQMJCAiI\n35Mr+8D2A2Dml+KtpUJ/cHaFrd/bLaQzWcjPJm4ziWBG2y1uLRoxk/UMpifjGaaLjU9dcnOY1pzn\nAWWYx1EbOp3HB1dH+DEX7CuqGa8WPgTL7xjPFTypMGTIEFxcXBg+fDhz587FxcWFn376icuXL+Pu\n7s6VK9psxIkTJ6hQoQLu7u5UrVqV8uXLv1bAMGvWLPz9/XF1tbyhQkBAAAMHDqRt27b4+flZ80dL\nELG1HfcAvkCrpAsDrqOt03gCjsBCYLyIxPmXopTyAlaKSLGIx+WAgSJSJ+LxV2j9182XiMTnB0rM\nPqNI3nkfFv4GJQolVs6bw5NgmFoSPpgCeevYLexzggjEl6x8SyY62y3uDa7SmYbkpwhDmUxKUtot\ndiSCMIPj9GMbX1OGnvjgoEOL85j4+wH0uACeKbQus4UN5jFs9H1GemHUfUYWbXpVSmUBIlfFLopI\nvBq2KKW80ZJR0YjHjsBpoDpaktsLNBeRRFcOWCUZdf4B3s0H3VsnVs6bxaVtsKwptN8HabLHfb2V\neEYggVQlG8PIiP3+T0J4Sm9ac5tgJrGcDGSyW+yonOc+rVlLKpyYSR1yYJyOxKEmmHATBl+Blplg\nYE5I56S3Ko3kZGQepRT3TgvpCrx+3pCbXqMiIjdFZG/EEd9ENA/YCRRQSl1SSrWLMGDtBmwAjgML\nrJGIrEYVn+T9RubIVQVKfWHX9SOAVOQnHxu4Sj/usSjuJ1iJ1LgwjsWUoTIfUpazOpVe5yEdW2lK\nNXJSktks4JQuOszh7ADdPeFECQgxaQ39/rhp7FLwZGBFZfirOpxbbPNWZhaTbAdkjsvXoWQTCN5u\nbF8UPRATzK8DnqWh6k92Df2Uw5ylFrmYTDqt+NJuLGUmw+jLSOZSmZp2jR2VfdygFWsoRVbGUo10\npNJNizkOPIbuQfDUBGO8oaJtW1bFSvKdkXmUUoQ9Ey6sgOMT4d5JKNgeyg9LAndGbx05PcHdFU6c\n1VuJ8VAO0HAOHJ0J59bbNbQLxcnLai7RiQess2tsfz5hHEvoTWvmMMGusaNSiqwcoDVpSUFxZhGA\n7S2b4kNJN/inCPTNBs0CoVUgXDV4KfjbiGNKyNcUGm6BhgEQrv/Wtrib6yml5tpLjKF4m/sbxYVr\nZmg4F1a2hYdX7RuaUuRhBRdpzSM22zV2WaqwmO1MZxQ/0pNwG5iAWIILzoyjBhOoQQtW04+thigB\nj0QpaB5RCu6dEoofgaHJpeCGJX1BqDhCbxWWNdfzUkqlsJMe4+BbOtk0NTa8fHVZPwJwowK5WcwF\nmvKYHXaN7U0+lrGLMxynEw14pKOn3Pvk4TBtCOQ+ZZnHMW7rpsUcbo4wJBfsLQr/PoYih+DPu8ml\n4MmYJ841I6XULKAQ8Bf81ydaRAyQS1/HKmtGAOcvQ8WWcG1r8rpRTOi4fgTwkPUE0Zq8rMKVMnaN\nHUooA+nGfnYwhZXkwNuu8aMiCNM5Rn/+4VvK0p2ShioBj2TjfegRBDlTwO/eUMjGpeDJa0bmMWpp\ntyVrRueAVRHXukc53mxy59B8RQL17cxpaHRcPwJIQ21yMZVz1Ocph+wa2xlnhjCBpnTEnwocQD+r\naIWiPUXZTQsWcZpaLOEKj3TTExM108HhYlA3PVQ5Dr2C4L5xZheT0RmLq+mUUm4AImJfa+N4YrU7\nI4BW/bTpuk5276CRtLi4VWs3Yef9R5HcYymX+YL8bCI1RewefzOr6UtbfmA0DWhu9/hRCcPEz+xl\nDAcZTVWaUlBXPTERHArfXoKV92BITmiXGRyt/Jk8+c7IPEn2zkgp9a5S6iDafqDjSqn9Sin7/8Xr\ngW/pZNNUS9Bx/QggPf7k4DfOUotndmwnHkk1PmAOf/M/vuZ3BupiIRSJEw58RzlW8yE/sJNWrDGE\nC3h0MjvDH3lhdUGYfgvKHIUdBncFtzaWth2fOXMmTk5OL9uKp0mThm3btsV7HKNjyTTdZKC3iHiJ\niBfwJfCHbWUZBN/SWkVd8qeruKn4NTilgq0/6BI+Ay3wZDCBVOc55+0evxDFWM4etrGeHrTgGSF2\n1xAVo5eAR+LjBtuLQG9PaPqWlYJnz56dAQMG0KFDhzivrVChAg8fPuTRo0c8fPiQKlWqJGgcI2NJ\nMnIVkS2RD0QkALDcjS8pk98LwsLggn173CRJdF4/AshEe7LyDYFU54UOLRg8yMK8iHLz5lTlFjfs\nriEqkSXgE6lJS9bQ12Al4JEoBS094FQJrQV68SMw7OqbXwpuadtxe42jN5Yko/NKqQFKKe+I4zvQ\n4aOnHiiVPFUXH3TcfxSJB5/hQXcCqcYLrtk9fipSM4p5+FGXRpTlJEfifpKNiXQBP8d9Q7mAR8fN\nEYbmgj1FYfcjrRT8r+RScAAOHjxI5syZKViwIEOGDMFkevMytSWWhu2BQcAytFbk/0ScezuInKpr\n11hvJUmDqOtHrTaDg/1dM7PQC+E5gVSnAFtxJrNd4ysUPfiBPLxDK6rzP6ZTnXp21RCdTLiwlAbM\n4DjVWGxIF/BI8qaCPwvChohS8HE3bFcKPsFKP/5nNkyYvr6+HDt2DC8vL44fP87HH3+Ms7Mz/fv3\nj/vJSYhY3yki3LW/FZHudtJjPHxLwf+m6q0iaVHxa7i0VVs/0mH/EUBWvsJECIHUoABbcCKj3TXU\npxk58OZTGtOJPnSgF0rHN3+Foh3v4ksO2rCW1VxgBrXJiY4GcrFQKx0cKQZjb0Dl49DaA37IYV1X\ncFsmEWvh7e398usiRYrw/fff8+uvv75xycgSB4ZKdtJiTArlhSchcMn+Uz5JFgOsHwF4MpC01CWQ\nWoRxXxcN71GOpexiCTP4hi6EEqqLjqhEuoBXJxc+zDGUC3h0nB2gVzbNFfxxuOYKPiXZFfyNLFm3\nZM3ooFLqL6VUa6VU48jD5sqMglJQJdmnLt4YYP1IocjGz7hRiXPUJVynjaA58GIJOwjmGm2pwwPu\n6aIjKo448A1lWYs/g9hFS1YbsgQ8kqil4NOC34xScEvbjq9bt47g4GAATp06xZAhQ2jUqFG8xzE8\nIhLrAUw3c0yL63l6HdqPZGVGzxbp8J31x30b+GeIyMzKIuGhukkwiUkuShc5LZUlTB7rpiNMwmSw\n9JaqUkDOyxnddETnibyQrrJJcskk2SwX9ZYTJyaTyJxgkez7RFqeEbnyzPx1NnkvsCIDBw4UpZQ4\nODi8PAYNGiSXLl0SNzc3uXz5soiI9OnTR7JkySJubm6SN29eGThwoISFhcU5TkzE9LpEnNftvTtW\nB4aINaPuIjLStinReljVgSGSI6ehSU84s9a6474N6Oxf91IGJi7SgVAuk5eVOJBaNy3zmMwIBjCW\nhZTDTzcd0VnHBTqwgea8w09UIqVF9U368ThcKwGfeBO+9ITe2SBVlLmeZAcG8yRJBwbR1oz09Tcx\nAu/mhzv34boxS2INjUHWjxQOeDEFJzw4TxNM6NfesgWdGcU8vqApC5iim47o1IkoAb/AQ0oz17Al\n4JG4OcJPEa7ge5NdwZM8lrh2jwScgYW86tp9wLbSEoZN7owAGnaF5h9As/etP/bbgM7+dZEIoZyn\nKQB5WIjCWTct5zhNR+pRk0b052cccdRNS1QEYSbH6cs2vqIMvQxaAh6dDfehZxDkSAGjvKGwa/Kd\nkTmMemdkSTLaYua0iEg120hKHDZLRiNmaA7eE/Sxu3kj2P4TnF+v2/6jSEy84DyNccQNb+aidEwC\n97jD5zTBjTT8zlxccdNNS3Qu8IDWrCEFjsygDrkMWgIelVATjLsJP12B22WSk5E5kmwy0hOlVG7g\nWyCNiHwcca4h8AFaG4tpIrIx2nNsk4z2H4fW/eHEKuuP/bZgkPUjABPPOEd9nMmGF9NRFhWW2oYX\nvGAAn3GU/UxhJdnIqZuW6IRj4hf+ZQT7+Z2qtKCQ3pIs4lYoZE6RnIzMkWSTkVIqCzAUyCYidZVS\nhYHyImK3naBKqUWRySjKuXTALyLSKdp52ySj8HDIWAHOrIHM9t9A+cbwJBimloQPpkDeOrpKMfGU\ns9QlFQXJyURdN6QKwh/8xjRGMpHllLBzs8C4OMhNWrKG4ngwnhqkJ5XekuIkuYDBPEZNRpZ8HJwB\nrAeyRTw+A/SMTxCl1FSl1E2l1JFo5+sopU4ppc4opeK7nfg7YFw8n5NwHB2h4nvJrcgTi2tmaDQP\nVraDh/oa0DrgQl5WEcJRrtBT19YPCkVn+jCYCXTgA1axSDct5niPLOynFZlxoRgz+ZvkppPJWBdL\nklEmEVkEmABEJAyI746q6UDtqCeUUg7A2IjzRYDmSqmCEd9rrZQaoZTyjLw82nN/BtaIiH3beyab\nplqHXFWgdDfd+h9FxRF38rGWJ+zkKv11TUgANWnAbDYyjD6MZrDueqKSGmdGUY2p1OYT1tGLLYQY\nwFEiJry8vFBKJR/RDi8vL73/a8xiSTJ6opTKiGaSilKqHPAgPkFEZDu8tu28DBAoIhdFJBRYADSM\nuH62iPQGniulJgAlIu+clFLdgOpAE6VU5/joSDS+yU4MVqPCV+DsClsH6K0ER9KSj/U8Yj3X0b9A\npTAlWM4e/mYlvWjFc4M5I9TCmyO04SqPKcVcDhGstySzBAUFmd1ceS5E+PCUkHu/sOy2YDLpv1nf\nmse9+8KPvwmZCgvNPhWOnHj1+0FBQXr/15jFkjWjksAY4F3gGOABNBGReHnjK6W8gJUiUizisT9Q\nW0Q6RzxuBZSRRJqyKqXE19cXb29vvL298fPzw8/PLzFD/kdoqLZudGEjZExnnTHfZp7c0taP3p8M\n+erqrYZQbhGIH+lpgSff6i2HZ4TQh7Zc5zITWY4HWfSW9AqCMIeT9CaAPpSiD6Vw1LEQJL5siigF\nzxpRCl7EBq7gevLoMUyYCSMmQXkf+K4n+BT/7/sBAQEEBAQQFBREUFAQW7du1XXNyFKLHSe0qbR3\nAeeEZGvACzgS5bE/MDnK41bA6MR+KsDWFiC1Ooos32jbGG8TF7eJjMwi8uCS3kpEROSFXJNjUkBu\nyK96SxERkXAJlxHyvVQULzkpR/SWY5YgeSC+skAqy3y5IPf1lhMvQk0io6+JeOwV6X5e5K5+rlU2\n48kTkVF/iGR/T6RuC5Ede81fh852QBZ9jBGRMBE5LiLHRJtSswZXgVxRHueIOGdskqfqrEuuylCm\np7YhNlz/9QdnPMnP39xiPLfsWB8TEw440ItB9GUoLanGZlbrLek1vEjD33xEffJSmrnM4rih1rpi\nw0lBN0/NFfyFaK7gk94wV3AXF+jeEc7tgoa1oWVXqNYEtuwwlluF3fYZKaW80abpikY8dgROo63/\nXAf2As1F5GQi44hNf6YdB6DbT3Bgqe1ivG2ICRbWg8xFodpwvdUA8JwgAvElKwPIREe95QBwgF18\nhj+d6Ut7eupaih4ThwmmJWsoSAYmUZOMOnoAJoRDT6D7BXgUDqNzQ2Xj7/ONN6GhMG85DB0NmTLA\ndz2gTjVwcDD4PiOrBFFqHuAHZARuAj+IyHSlVF3gd7RCiqki8rMVYtk2Gb14oa0bXdkCad1tF+dt\n4+ltbf2ozgTI/4HeagB4xlkC8SMbQ8lIG73lAHCFi3SkHiWpwCDG4qyjnVFMPCOMb9jOIk4zldrU\nxltvSfFCBBbdgb4XoYI7/OIFOVPqrcr6hIfD4pUw5HdIlRL2b0gCyUgplR1tzeelh4uIbLOhrgRj\n82QEUL0d9G4LH/jaNs7bxuXtsNQf2v0LaXPFfb0deMYpAqlGdkaSIcLTTm8e84geNOcZIYxnCWlJ\nr7cks/zNRdqxnobkZThVcDFg4oyNp+Ew/JrWabZHVuibDVIbwz7QqphM8Oc6aPyBwZORUmo40BQ4\nwX/7i0REGthYW4KwSzIaNE7r/vq/PraN8zayczic+RNabwVHY7x5hXCUQGqSiwmk40O95QAQTjjD\n6MtmVjOVVeQmv96SzHKPZ3zOJg5xizm8j4/BKgItIeiZdpf07xP41Qv8M2g9N9809HZgsCQZnQaK\nichz+0hKHHZJRgF7of9vsGehbeO8jYgJFtYHj8JQ/Re91bzkKQc4S128mE5ajOPcPo/JjOR7xrDA\nUL2RojOfk/RgCz0oSX/K4JSESsAj2fIAegRBJietFLyoq96KrIveyciS34jzkMTur21N2WJw/Cw8\nehL3tcnED+UADWbCiYVwZqXeal7iQkny8hcXactDNukt5yVG7Y0UneYUYj+t2MxlfFnIOe7rLSne\nVE0LB4pBk4xQ/QR0PQ939C8AfWOwJBk9BQ4ppSYppUZHHrYWZmhSp4KShWHnQb2VvJm4ZIIPF8Ca\njvDAOB5orpQlD8sIogWP2Kq3nJdUoBqL+IdJ/I+f6EN4vN267ENO0rCRJjShAOWYx1SOJpkS8Eic\nFHyeFU6W0DzKCh2CcTcgLGn9GIbEkmm6T8ydF5GZNlGUSOwyTQcwYLRWjjK0l+1jva3s+gVOLYU2\n28Axhd5qXvKIzVygGXlYgRsV9Jbzkvvc5XOakBpXRjEPN4xb7XmM27RkDblJwx/UwoOkaX9w5Ik2\ndXcnTJu6q5pWb0UJx/DTdBFJZz6wP+KYZ9REZFeSN7/annJfgosHbPlabyWv4E41vJjFeRrxBOP8\nDqQjAzNYhwdZ+YhKXOWS3pJi5F0ysZcWvEMGijOL1ZzXW1KCKOYKmwvD9zmg3TlocloreEgm/sSZ\njJRSfkAgWruG8cAZpVQVG+syPuVLwKFT8DREbyVvLsoB6s+Ak0vg9J96q3mFtNQhF1M5Rz2eclhv\nOS9JQQqGMRl/PsGf8hxkt96SYiQlTgynCvP5gK78zads5ImBXcBjQiltHelkcSjmAj5H4ftLWml4\nMpZjyZrRb0AtEfEVkSpoLR9G2lZWEsDVBYq/A7vs28XircMlo7Z+tLYz3A/SW80rpKM+ORnHWeoQ\nwgm95bxEoehIb4YwkY7UZyUL9JYUK77k5DBtCCGMEsxiD9f1lpQgUjvC9znhYDE480yzFlpw21iW\nO0bGkmTkLCKnIx+IyBmSq+s0kqfq7EOO8lCuHyxvCuEv9FbzCunxJwe/cZaaPOOM3nJeoQb1mcPf\nDOcrRvKDoYsF0pKSmdRlGJVpwAoGspNQgxZixEWulLCgAMzJDz9fBd/jcDC58DZOLElG+5RSU5RS\nfhHHH2CgiXI9SW62Zz/K9gbXLLA5vg2BbU8GWpCNIQRSg+cGW/soRDGWs4d/2ED3CNcGI9OEAhyk\nNbu5TkUWcIa7ektKMFXSwP5i0NID6pyELufgVtKbhbQbliSjz9DcF7pHHCciziVTsSTsPw7PksR+\n4KSNUtr60enlcHqF3mpeIyPtyMo3BFKd5wZrye1BFuazBQccaIYfwQafBsuGG2tpTBsKU4H5TOSw\noe/qYsNRQZcscKoEpHKAwodg1HUINemtzHjYzbXbXtittDuSMk3h175QpZT9Yr7NXN0Di+pDuz2Q\nLrfeal4jmFHcYgz52UoKsust5xUEYQyDWchU/uBPClNCb0lxcoo7tGItWXBhKrXJStK2PTj+VGvo\nd+0F/O4NNQ3Uo9Owpd1KqUUR/x5VSh2JfthPosHxLZU8VWdPspeFCl/DMuOtHwFkpgcZ6Uwg1Qnl\nht5yXkGh6M73fM0vtKYmGzBWhaI5CpKRnTTnPTJTglmsIFBvSYmiiAtsKAQ/5YIu56HRKTiXXAoO\nxHJnpJTyFJHrEe3CX0NEjDUXEYHd74xWBcDvs2DTNPvFfNsRgSUfQlovqDVKbzVmuc5g7rGQ/GzB\nGQ+95bzGIfbyKR/Slh50oa8heyNFZydXac1afMnBKKrhjnE2QieEZyYYcQ1+u65N5X2THdx0dAU3\n7J2RiEROLH8uIhejHsDn9pGXBKhUEvYc0focJWMflIJ60+HMX5pDgwHxZADpaMRZahFmwEX4EpRh\nGbv5i3n0pwMvMP7vbwWyc4g2OOJAcWaxnSt6S0oUqRzgmxxwpBhceq6Vgs+59faWgltSwFDTzLm6\n1haSZEmXBvLlgn3H9VbydpE6PTReBGs/g3vn9FZjFk8G4051zlKbcB7oLec1spGTxWznPndpTU3u\ncltvSXHiTgr+oBYj8eMjVvEN//AiiZaAR5I9pVYGvqgA/H4dKh6DfY/1VmV/Ylsz+kwpdRR4J9p6\n0QUgec0oKskl3vqQrTRUGgBLP4Iw4028KxTZ+QVXynGWuoTzSG9Jr+GKGxNZxnuUozHlOMtJvSVZ\nREPycYjWHOU25ZjHCe7oLSnRVHCHvUWhQ2aofwo6nIWbxr9htRqx3RnNA+oDf0X8G3n4iEgrO2hL\nOviWTt78qhelvoD0eWFjb72VmEWhyMEoUvEu56iHiad6S3oNBxz4iuF05Vua4cs2NugtySKy4Mpf\nNOJTilOFBYzmAKYkWgIeiYOCDhGl4OmcoMhh+O0avHgLSsEtLu1WSmUGUkU+FhFDujDavYAB4PY9\nyFML7u4CJ6e4r0/Gujx7ANN8wHcwFGmutxqzCCYu0p5QrpKXlTj896dkKPawjS/4mO58T+sktDQc\nyD1as5Y0pGA6tcluYMfy+HAqBHoFwYVnMNIb6tqww7xhCxgiUUrVV0oFAheArUAQsNbGulBK5Y5w\nflgU7byLUupfpZRx2m1mSg9e2eCAcfzJ3ipSpYXGi2FDd7hzOu7rdUDhgBdTcSIT5/HHhDE3Spel\nCkvYwUzG8APdCCNMb0kWkZ/0bKcZlchOSeawCGP+HsSXgqlhTUH41Ru6B0G9kxBobBONBGNJAcMQ\noBxwRkRyA9XB9lbAInJBRDqa+VZ/wHj9vpPXjfQl63vg+xMs+whCjTcVBqBwxJtZOJCKCzRFDOpQ\n7UVelrGL85ymA/V4aMDiC3M44cD3lGcljfiOHbRmDfcx3lpifFEK6qWHY8U1i6Hyx6DfRXiYND4n\nWIwlyShURO4ADkopBxHZAlhsN6CUmqqUuhl9o6xSqo5S6pRS6oxSyiLDMaVUDTQ7oltgsI0Ryaap\n+vNeJ/AoCuu76a0kRhTOeDMfIZwLtEQMeueRhnRMZw25yIs/5bmIMSsWzVEGTw7SGndSUJxZBHBZ\nb0lWIaUD9MuuJaVboVop+IxgMCXtZbKXWJKM7iul3IBtwFyl1CggPh6009HaTrxEKeUAjI04XwRo\nrpQqGPG91kqpEUopz8jLozzVDygLtADM3TXpR5VSsP2A1v01GX1QCt6fBFd2whHj9n90IAV5WEw4\nDwiiLWLQ0mQnnBjMOFrTlSZUZA/b9JZkMa44M54aTKQmLVlDHwJ4btDEH1+ypoDp+WDFOzDxpnan\ntMd4hZrxxpJk1BAIAXoB64BzaFV1FiEi24F70U6XAQIjNtGGAgsi4iAis0WkN/BcKTUBKBF55yQi\n30V8by7wh6Ua7EKWTODpAYffjLnqJEsKN2396O8+EHxMbzUx4kAq8rKcUK5xic4Ixi2XakNXfmMW\nXWnCYqbrLSde1CU3h2nNeR5Qmrkc5ZbekqxGGXfY+S50zQqNT0Pbs3A9CZeCx6eaLg3wslRMRCze\nVh5hKbRSRIpFPPYHaotI54jHrYAyItI9HtpjiiW+vr54e3vj7e2Nn58ffn5+iR3WMj4dCAXzQM82\n9omXTMwcngG7hkP7f7UEZVDCecxZ6pCaYuRknKFtec5yko7UpzaN6ccwHNHRuyaeCMIMjtOPbXxF\nGXrhg4OBX+v48igchlyBqcHQPxv08IQUcdxqBAQEEBAQQFBQEEFBQWzduvX/7Z13uFTV1YffH51L\nR7GBNAE1dkARxXAtKNGIKNhQsZuoUYxdPxNNjIqNSGyxBcWKiAoqKLYLFpReVKRIU5Ai0lQEgfX9\nsffgcLll5sy9nJnrfp9nntlnn3PWWaeuXdeKdTRdqcZI0p+AfwA/A5twzWZmZi1TPsg2NkaxeSJ/\n/g148U145f54jh/YktfOg03rodvTrgkvS9nIambRhdocQmP6ZbVBWsFyLqYHdajHfTxLLbLX0BfF\nXFZxFsOpRmWeoiu7UjdulcqUWX4o+Ew/FPy4NIaCZ/3QbuBqYG8za25mLc2sRTqGqBgWAk2Tlpv4\nvNzm9+3hgwmwKXubXH5TdH0AlkyByY/HrUmJVKYurXiTNYxiETdkdeyeBmzHQEbSkEb05FAWkpXT\nDYulBfUYxakcRTPa8QzP54jHiVRpXRNe39OFp7hyHhw7HWbkyFDwVIzRHMh42rjYciDCOKCVpGaS\nqgGn4Tw95DaNd4T6deDz2XFrEgComuf6jwpuhMWT49amRKrQgNa8zSqGs4ibstogVaMafXmMk+jN\nSRzMpPKf6VGmVKYSN9KBEfTgn3xCL96oEEPAkzm2AUzbD46o53zdXTMv+4eCp2KMbgDGSHpE0n8S\nv1QPIOk54GOgjaQFks41s43AZcBI4HPgBTOrGEWUMN8ou9h+D+jS380/Wrc6bm1KpArb0Zp3Wc3r\nLOLGrDZIQlzIVdzOo1xIN4bxfNwqpU07dmQCZ9KQGuzHQN7PsVpeaVSrBFfv4oaCL9+Q/UPBU+kz\nGgd8AEyDX4f8mFlWjp2Ntc8IYOBQeO19GHxffDoEtmbExbB2OZw4KKv7jwA28B2z6EJdjmYX+mZ1\nHxLAdKZyId3owdn04WYqpVTGzS7eZC7nM5LT2Z3b6ER1Kp5br7FrnBcHA/7THDoU8pinNQ8WAAAY\n2klEQVQUd59RKsZokpkdsI30yZjYjdH8hXDQabB4dNZ/9H5TbPgZnjwE9j8f2l8atzalsoHlzKIL\ndTiCxtyd9QZpGUv4MyeyM7tyNwOoSV7cKqXNd/zERbzNV6ziGf7APlkYFDFTNhk88x1cPx+Org93\nNIWdfYzCuI1RKkWYEZIukrSzpIaJX7lrlqs0aww1q8OXc+LWJJBMlRou/tEH/4BF2d+M6prs3mEN\n77OQq7K6yQ6gETvyHO9RhaqcRj5LWBS3SmmzPXkMoRtX0JYjGEw/xue8F/DCVBL0buS8gu9YFfaZ\nAncvzA6v4KnUjOYWkZ3W0O5tSew1I4Czb4CO+8GfT4tXj8DWfDkE3r0GzpvgAvRlORtYwWy6UItO\nNOHfWV9DMoyHuINneZhHGcretI1bpUjMYSVnMYIaVOFJjqlwQ8ATzFzrRt3N/Blmtc3yZrpcIyuM\n0f+GwNtj4Pl74tUjUDQj+8Cq+dDzlZxoSt3ASmZzNLU4mCb0z3qDBDCCIdzEn7mNR+jKSXGrE4kN\nbOJOxtKfifyHIziNPeJWqdwYvgKOa5ilxkjSEWb2nqQinyQze7lcNYtIVhijrxbAYWfBwoKc+Nj9\n5ti4HgYeBr87FTpkZ1C+wmxkFbM4hjzasSsP5IRBmsYE/kR3zuQSLub6nNC5KMazmDMZTjt25EGO\npH6WxqLKlGzuM+rs/48v4vfHctYrt2m5K1SqBLPnx61JoCgqV3Oj6sbcCd+MiVublKhMPVrzFmuZ\nyNdcmtW+7BLsQzte5hNG8ioX04PVrIxbpUi0Zycmchb1qV6hvIBnG6n0GbUws7ml5WULWVEzAjjj\nGji8A1zQM25NAsUxc5gLN3H+BMjbPm5tUmIjq5N82T2EcmAY9TrWcTtXM4oRPMhg9iJnBuduxXDm\ncAEjOYM9+SeHUJOqcatUZmRzzSjBkCLyXiprRSocnQ+E0SG+UVbTpptrqht6hhv6nQM410FvsZbP\nWMCfcqKGVJ3q/IP7uZrbOJtjeI5Hs350YHEcS0um0JsFrGYfBvIuofWjrCipz2gPXKyhu4BrklbV\nBa4xs73KX730yZqa0Yy5cPQFMP/duDUJlMTGX2DYWbByLvR8FersXPo+WcBG1jCbY6lBG5ryWE7U\nkADmMJNL6Mme7Me/eDjnHK0m8zpfcSnvcjhNuZfObEfNuFXKiGyuGe2O6xuqz5b9RW2BC8tftRyn\nTXNYtx7GZ29MnQBQuSp0fx5ad4MBB8Gi3KjNVqYOrRjBOmYzn/OzNkBfYVrShlf4hKpUpTsHMYsv\n4lYpMn9kNz7jHOpRjb15iueYnrM1vmwglT6jjmaWG728ZFHNCFxIictvh39fB2d2i1ubQGnMeBWG\nXwRH94e9To9bm5TYyI98xR+pRlOa8T+UQzGGBjOAvlzL37iP7pwRtzoZMZZvuYCRNKY2D3MUzakX\nt0ppE3fNKBVjdBfwL1y01zeBfYG/mtkz5a9e+mSVMQKYOgN6XgGHHwT9b4Qa1ePWKFASS6bC4BNg\nr16Qfyso+5u/nEE6nmo0phlP5pRBms5ULqEnHTmcm+lP9RweNv0LG7mH8dzLeG6gA31oS5UcaT6F\n+I1RKlfqaDNbjWuymwe0Yss+pEBJ7Ls7jB8MK1bDIb3cHKRA9rLjvnDuWPjmQxh8IqxbE7dGpVKZ\nWrTidX7hW+bRGyPLYwUksSf7MozxrGIFPTiE+XwVt0qRqUplbqADY+jFcOZyMM8xiSVxq5UzpGKM\nEmMXjwMGm9mqctSnYlK3NgzqB+eeCB17wSvvxK1RoCRqNYJeb0PtneCpjrAi+/0MViKP3XiNDSxj\nHmfllEGqQ10eYBAncx496MhbvBK3ShnRmga8Q0/+wv505WWuZRQ/8UvcamU9qTTT9QW645rpDsIN\naHjdzDqUv3rpk3XNdIX5dAqcehX06AJ9r4SqFWeeQoXDDMY/CB/9C058AZrlx61RqWxiLV/RHSEa\n0Ye6HIVyaC7MZMbyF06hKz24jr5UzSHdi2IpP3EF7/Mp3/JfjqILzeNWqVjibqZLyTed99K9ysw2\nSqoF1DGzxeWuXQSy3hgBLF8Jva+HlWtg0L3QZKe4NQqUxNx3YWgvOOwWaHdx3NqUyiZ+5jseYQUv\n8DOzqM+JNOA06tAZ5UCcnpV8z1X0ZhUruJ9B7EyTuFXKmBHM5WLe4fc0oR+d2T4LQ2zEbYyKbaaT\ndG3S4pE+Oitm9iNweXkrVqHZrj689hD8sTO0PxlGfhS3RoGSaHEk9P4Ixt8PIy5xc5OymErUYAf6\nsDtj2IMJ1KANi7iOaTRhAZeyhtFZPVm2Pg15jGEcyfGcwIGMZmTcKmXMH2jBZ5xNI2qyN0/xNF+E\nYeCFKGnS60Qza1s4XdRyNpETNaNkCsbCGdfCBT3g75dA5dwZCfWb4+dV8Gov2LAWThoMedvFrVFa\n/MxsVvIi3/MCG1lOfU6mIaeRR4esdWL6CaO4gl6cwvn04WYq59BIweIYz2IuZCSNyONGOrAnDdmB\nvNjvQdw1o5KM0eYIr4WjvWZz9NecM0YAi5fB6ddAlcrw7F2wQ2595H5TbNoIBTfC9JfglGHQKCsd\nkZTKWqazgkGsYBDGWupzCg05lZq0jf2jWJhlLKYPvVjEAtqwN81oRQta04xWNKc1O9Mk50Kd/8JG\n+jORIcxiJivYwCZa04A2NKA19WlDQ//fYJt5Cc9mYxRrzUhSC+D/gLpmdorPE3ArziXRODN7uoj9\ncs8YAWzYADc/AAOHwfN3Q6d2cWsUKIlpz8A7V8JxT0Cb4+PWJjKGsZZp3jC9gKhEA06lAadSg72z\nxjBtZCMz+Iz5zGY+s5nLLOYzm3nMYiXf05SWm41Tc//fjFbswq45YaiWs5ZZrGAWK5nJCmby/eZ0\nHlWSDJX7b0MDWlGfvDIc4JHNxmgj8CMgoCbwU2IVUMPMtskwF0kvJhmj7riRfd8Bb5jZ+0VsXybG\nqKCggPz8/G0vZ/goOPcmuPocuPo8kOLTpRzlVAhdFn4KQ3pA+79Ax+soGDUqp8/JMH5iAisYxEpe\npBK1mFywAwfn75hklOR/W6ZLW/9JwWI65mfm929MwbdFyljLLyxiDQtZzTesZmFSejXr2JnaNKYu\nTahLY+qyoOBHds+vv1nDLf+3/BZri7Q2500vWMGe+Q0zOh82yyk+4rABq6jOYmqxhFos9r8l1GIZ\nedRhPTvwE+sL5lI7vynC/FW3pKu/ZR5FrK8EfKT7YjVGxQ6tMbMyaZyV9ARuwuwSM9s3Kb8rcB9u\nEMUTZnZnCuJ2Bz4ys8ckDQa2MkZlRWwfy2M7w7hBcMqV8OFEePL2rPpwl5WcCqFL4w5wzqfwUndY\nOo2C8buS36mj89pQqTKgSMEV4zonIWrRnlq0pzF38hNjeargHrrmn+i3MP9zaUtKF7U+kTaMiQVD\nOSq/U8SzcRQnoxawPc41TGF+Zh3fsIyvWco3LGUeSxldsIAf8rd2ampJ+ibOIHlt8roJBUtZlp/5\nyMSJBUtYlp/Kp3YZAnb2P4BNiB/JYzW1mVUwmx3yf/Q6qph/2NIkuXWJ/LjZFuM8BwD3AwMTGZIq\nAQ8ARwKLgHGShprZl5LOAg4A7jazb9nyKn0NrPfp3PAMGYWmu8DogXDNPdDuZOi0K3w+K3O5S5dn\nj5yKpEv7/8Hkv8GYe+Dufm5+km0CNuEMUiWgkjNMquyXS8gf8wP0eySzcxqzJiMZwn3ka45ZQ8N+\nmY/2zBuzhu37jY1FRhPg4KTlW8as4ZZ+mX0+nIzMW2DKVs7qjGTEbY7K3RiZ2YeSmhXKPgiYZWbz\nASS9AJwAfOn7gZ6W1FDSw8D+kq7zNadXgPslHQaMKm/dY6VaNefL7rB2cGkfmFAG4bGXzoBR87JD\nToXUpSUs3r1QpoHMv+lJafmaQyKdnL9wDnzfMjNdykJGtskJupSznHinjqY06TXjgzhj9FqimU5S\nD+AYM7vIL58JHGRmGc9fkpSDoxcCgUAgfrKyzyhXifNiBgKBQCAacY15XAg0TVpu4vMCgUAg8Btk\nWxmj5PGeAOOAVpKaSaoGnAYM20a6BAKBQCDLKHdjJOk54GOgjaQFks71fu4uA0YCnwMvmNn0MjhW\nV0lfSpop6bqIMp6QtETS1Az0aCLpPUmfS5omKVJfmKTqkj6VNMnLuj0DnSpJmigpstGXNE/SFK9P\n5KFRkupJGixpuj+vtDzAS2rjdZjo/1dlcI1v8DpMlfSsLxylK6OPv89p3euinjVJDSSNlDRD0luS\nSg0ZWoycnpI+k7RRUkoT1IuRc5e/T5MlDZFUN4KMf/rnZrKkdySV6vm0pPdQ0lWSNnkHzlHO6WZJ\n3/jnZ6KfZpK2LpIu89dmmlx0gyi6vJCkx1xJEyPIOFDS2MR7Kal9RF32lfSxv1dDJdUuTU6ZYmYV\n4oczrLOBZrgYTJOBPSLI6QTsD0zNQJedgP19ujYwI4oufv88/18Z+AQ4NKKcvwLPAMMyOK85QIMy\nuFdPAuf6dBWcl41M7vsiYNcI+zbz51TNLw8CeqcpYy9gKlDd36ORQMuozxpwJ3CtT18H9I0oZ3eg\nNfAe0DYDfY4CKvl0X+COCDJqJ6UvAx6PoovPb4KLOD0XaBjxnG4GrkzjHhclI9/f6yp+efuo55S0\n/h7gpgi6vI8LggrwB+D9iOc0Fujk0+cA/0znXcj0l/1+MlJn83BxM/sFSAwXTwsz+xBYkYkiZrbY\nzCb79A/AdKBxRFkJzxfVcR/etHXzJdFjgcej6JAsigxr075kfZiZDQAwsw3mIglH5SjgKzP7OsK+\nq3Hz1mpJqgLk4QxbOuwJfGpm68zV+EcDJ6WyYzHP2gnAUz79FM7jSNpyzGyGmc0ijekjxch5x8wS\nLr4/gZLjORQj44ekxVo4Dypp6+L5N2lEmi5BTkbXBbgYV1DY4LfJ5JwSnAI8H0HGt0CiBl2fFPrf\ni5HT2ucDvAP0KE1OWVKRjFFj3KTYBN8Q0QCUJZKa40ogn0bcv5KkSbhJAAVm9kUEMYkXONNh7wa8\nLWmcpAsjymgBfCdpgG+aeFTS1tPhU+dUSnmBi8PMVgD3AgtwL/BKM0s3DO9nwGG+eS0PZ/R3jaKP\nZwczW+L1WwzskIGssuY8YESUHSX9S9ICXIn7jogyugFfm9m0KPsX4i++2fDxVJpCi6AN8HtJn0h6\nP5WmsZKQmzu52MyixF2/Hujnr+9dwA0R1fjcX2NwhnGbBpKqSMYo6/Btri8BfQqVDlPGzDaZ85De\nBPfwd05Th+Nwrpgms/VAknQ51JyD3GOBSyVF8e9SBWgLPOhl/YR7mdJGUlWgGzA44v4tcc2XzYBd\ngNqSeqUjw8y+xDWtvQ0MByZRtt5BsmLenKT/A34xs+ei7G9mN5lZU5xHlvsiHL8mcCOuiW1zdhRd\ngIdwTan74wp5/SLIqIJrsj4YuBZ4MaIuCU4nYqEKeAK4zF/fvwL/iyjnPNx7PQ5Xg11fyvZlSkUy\nRlk1XNw3+7wEPG1mQzOV55uy3gDSLYEdCnSTNAf3sB8uaWAp+xSnw7f+fxnOG8ZBEcR8gyvdjvfL\nL+GMUxT+AEzw+kShPc7X4fe+ie1l4JB0hZjZADNrb2b5wEpgZkR9AJZI2hFA0k7A0gxklQmSzsEV\nQNIy1MXwHOk/wwC7Ac2BKZLm4t7vCZLSrjma2TLzHSPAY8CBEfT5Gve8YGbjgE2SIsV+kVQZ17Q7\nKMr+QAcze9Xr8hLR3kvMbKaZHWNmB+K6OaLU0iJTkYxRWQ4Xz7QGAa508oWZ9Y8qQNL2iSYEXzLs\nghuYkTJmdqOZNTWzlrhr8p6Z9Y6gS15idI1c6PmjcU1UaeGboL6W1MZnHQlEaXqEzEqT4AaWHCyp\nhiR5XdIe1Smpkf9vCpyI++CmvDtbPmvDcE1ZAGcDqRZkSnpm03mWt5DjR5pdA3Qzs3URZbRKWted\n1J/hzXLM7DMz28nMWppZC1yh5gAzS8VYF9Znp6R1J5Hac1z4+r4KHOHltQGqmtnyCHLAvdfTzSzV\n/srCMmYlWkwkHUnqhaHC1yXxHFcCbgL+m6KcsmFbjpYo7x/QFfeBmQVcH1HGc7hO7HW4voRzI8g4\nFNdUMxnXbDMR6BpBzj5+30nAFODqDK9PZyKOpsP19STOZ1rU6+tl7YcrPEzGlS7rRZCRBywD6mR4\nTa7BTS+YihswUDWCjNG4D9okID+TZw1ogOs8noEbrVU/opzuuNL7WlwH94iIcmYB8/1zOBF4KIKM\nl/wzMwkYgusXS1uXQuvnkNpouqL0Gejv92ScUdkxgowqwNP+vMYDnaOeE67p8qIMnpl2uD7pScAY\nnJGOIudy/9x9CdyeyXsV5bdNfNMFAoFAIFASFamZLhAIBAI5SjBGgUAgEIidYIwCgUAgEDvBGAUC\ngUAgdoIxCgQCgUDsBGMUCAQCgdgJxiiwBd41/91Jy1dJ+nsZyR4gKSUnohkep6ekLyS9Wyi/maTT\ny/v4JSHpw9K32mL7s5MnafowA6WGTihCzt76NezGcklz/PLICLJ28/4SY0HSJXHfx0DZE4xRoDDr\ngJOifPDKE+8yJVXOBy4wsyML5bcgTZc2aR63VMwsXX9+57Clw99IEwPNeTA4wJw/wKG4CdQHmNnR\nUeSlo0dZXkNJlc3sITPLxPNGIAsJxihQmA3Ao8CVhVcUrtlIWuP/O0sqkPSqpNmS+ko60wf6miKp\nRZKYLt7r95feiWvCM/ldcoEEJyc8gnu5oyUNxXlJKKzP6XJB8aZKusPn/Q0Xq+UJSXcW2uUOoJOv\nHfRJ9bi+RjXdn/8MuSB8XSR95JfbJ+2XqH1M8G6TCuucfM3e169BBp8uYtseOD9uz3iZNXDuWy73\n8qck3Cp5d01PyHmRniDp+CLu7WbRhY5TR9K7ksb763Csz79N0qVJ2/WVdHGhfWtIetLfg/Fy3qeR\ndL6kVyS9h4s/lLxPbUnD/bWamnimJLX3z9E4SW/oV/c0H0jqJxfQ8VJJt8oHMZTUStKbfp8Cbel6\nKJBLbGuXD+GX3T9cjJ/auOBldYCrgL/7dQOAk5K39f+dge9x4Q6q4RzU3uLXXQ70S9p/uE+3wrmr\nqQZcCNzo86vhXAU183LXAE2L0HNnnJuahrhC1bs4/2nggo1t5RKFQu6QUj2uz1sP/M4vjwee8Olu\nwMs+PQzo6NN5+KB0ha9vki4r/HkIFw35kCK2fy/5XPx9ucSnLwYe9enbgF4+XQ/n1qVmMfe48H2s\njA+ABzQCZvr0bsBYn66Ec5xZz+dP9PnXAv/16d8B83Cucs73um4VOBEXnuDBpOU6/vp/hHfxg6vB\nPuLTHwD3JW1/K3B50vVp4dOHAG/F/Q6FX7RfFQKBQpjZD5KeAvrgfJulwjjzTislzQbe8vnTcFEx\nE7zojzFb0lfAHjinq/tIOtlvUxcXpfQX3MdwQRHHOxAX0fJ7f8xngd/zq3PcVJyDpnPcufZrLKnP\ncf7jEufX3Kc/Av7tdXnZzErzGj/WvCd0SZO9nI8LbVOUY81X/P8EnGPWxLkcLykReK4azov9jFJ0\nAGdo7pQLCbIJaCKpoZl9JWm1pL28bp+a2SpJ2yft2wkXQwcz+0LSQlxBA2CkFR04cSpwh6TbgdfN\n7GNJ++Gi5r4jKRHEMTk+2VYereWcCB8MDPH7JM4lkIMEYxQojv44x5gDkvI24F92//JXS1qX7NF5\nU9LyJrZ8zpL7GuSXhYvH8nayAnKeiH8sQcdMPaunc9xSz8/M7pT0OnAc8JGko82sJA/KyTI3kvr7\nmNgveR8BPcxFd02X3jhDvL+ZmaSvgRp+3RM4J5rNSc2Lc/I9KfLemdmXvmnzWJxRGoFryptiZsXF\n6ypKloBl5vrBAjlOKEUECpNw2b8CV4s5P2ndPH6NRXMCUDWC/JPl2A03oGAGrhZ1iVwMKCS1loua\nWhJjccEGG8p1kJ8OFJSyzxpck1CCdI5bquGT1NLMPjezu3BNfntEkVOI1ThDURpv4ZpEE7rsn8Yx\n6gFLvSHqggs0mOBl4HhgPys6Cu4HwBn+mHsCOwGzSzqYpF2AH83sWVxgu7a4MCKNJR3ot6kq6Xcl\nyTGzlcC3krr7fSRp31LPNpCVhJpRoDDJNZd7gUuT8h4DhsoN632L4mstJY20WoAzJHWAP5nZekmP\n40reE32NaykuBELxSpotlnQ9vxqg183s9VKOPxUXBG0S8KSZ9ZcLC5/Kca2YdDJXSDocV2P5nKJD\ndBe3b3H5TwH/lfQTrk+kuO1uBe6TNBVn8Obi+rNSOdbTwGuSpuDuzebalZmtkzQaF4aiKO4HHvHH\nXQ+cZWYbfm01K5L9gL6SEjXMP/vnoCdwv6S6uILyvTgjVdLzdDrwsKRbcIWjZ3D3OZBjhBASgUCg\nWOQCrU0CTjCzeTGrE6jAhGa6QCBQJJL2xjW5DQ+GKFDehJpRIBAIBGIn1IwCgUAgEDvBGAUCgUAg\ndoIxCgQCgUDsBGMUCAQCgdgJxigQCAQCsROMUSAQCARi5/8Br2V0OYoc2LIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x8c76d50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Let see if this method work, on a well conditionned matrix:\n",
    "to do so, we directly generates random symmetric matrix,\n",
    "and then tune its spetrum to an arbitrarily well conditionned\n",
    "matrix\n",
    "\"\"\"\n",
    "size = 20\n",
    "P = np.random.rand(size,size)\n",
    "P = np.dot(P,P.T)\n",
    "[v,Q] = np.linalg.eigh(P)\n",
    "\n",
    "#generate arbitrary good conditionning\n",
    "NB_SAMPLE = 8\n",
    "nb_taylor = 20\n",
    "condstep = np.linspace(1+1e-5, 1+1e-1,NB_SAMPLE)\n",
    "cm = plt.get_cmap('gist_rainbow')\n",
    "colors = [cm(1.*i/NB_SAMPLE) for i in range(NB_SAMPLE)]\n",
    "plt.figure()\n",
    "plt.title( \"Approximation error of P inverse, functions of Taylor serie order\")\n",
    "plt.xticks(range(nb_taylor))\n",
    "plt.xlabel(\"Number of terms in the Taylor serie\")\n",
    "plt.ylabel(\"Estimation error (Frobenius norm), logscale\")\n",
    "plt.yscale('log')\n",
    "\n",
    "for cond,color in zip(condstep,colors):\n",
    "    P = np.dot(Q,np.dot(np.diag(10+np.cumprod(cond*np.ones(size))),Q.T))\n",
    "    c=np.max(np.linalg.eigvals(P))\n",
    "    condNum = c/np.min(np.linalg.eigvals(P))\n",
    "    realP1 = np.linalg.inv(P)\n",
    "    ratio = (np.identity(size)-P/c)\n",
    "    Pn = np.identity(size)\n",
    "    P1 = Pn.copy()\n",
    "    frobError = [np.linalg.norm(realP1-P1/c)]\n",
    "    #Taylor serie sum\n",
    "    for i in range(nb_taylor-1):\n",
    "        Pn = np.dot(Pn,ratio)\n",
    "        P1 += Pn\n",
    "        frobError.append(np.linalg.norm(realP1-P1/c))\n",
    "\n",
    "    plt.plot(range(nb_taylor),frobError,\n",
    "             color=color,label=\"{0:.2f}\".format(condNum))\n",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Krylov Subspaces methods\n",
    "\n",
    "### Introduction: preconditionned iterations\n",
    "\n",
    "#### Yet another fixed point iteration method\n",
    "\n",
    "The approach seen in the notebook called FixedPointIterationsMethods, with a-priori decomposition of the matrix $M$ that lead to a fixed point iterations can also be generalized as follows:\n",
    "\n",
    "\\begin{align*}\n",
    "  M \\vec{x} - \\vec{y} &= 0\\\\\n",
    "  (P-(P-M)) \\vec{x} - \\vec{y} &= 0\\\\\n",
    "  P \\vec{x} &= (P-M) \\vec{x} + \\vec{y} \\\\\n",
    "  P^{-1} P \\vec{x} &= P^{-1}(P-M) \\vec{x} + P^{-1} \\vec{y} \\\\\n",
    "  \\vec{x} &= \\vec{x} + P^{-1} ( \\vec{y} - M \\vec{x} )\n",
    "\\end{align*}\n",
    "\n",
    "This problem can simply be recasted as a fixed point search problem, that can be solved using the following iterations:\n",
    "\n",
    "\\begin{align*}\n",
    "  \\vec{x^{k+1}} = \\vec{x^k} + P^{-1} ( \\vec{y} - M \\vec{x^k} ) \\\\\n",
    "  \\vec{x^{k+1}} = \\vec{x^k} + P^{-1} \\vec{r^k}\n",
    "\\end{align*}\n",
    "\n",
    "or\n",
    "\n",
    "\\begin{equation}\n",
    "  \\vec{x^{k+1}} = (Id-P^{-1}M)\\vec{x^k} + P^{-1}\\vec{y}\n",
    "\\end{equation}\n",
    "\n",
    "Where $ \\vec{r^k} = \\vec{y} - M \\vec{x^k} $ is called the residual and $P$ is called the preconditionner.\n",
    "\n",
    "As seen in the notebook FixedPointIterationsMethods, according to the Picard fixed point theorem, the preconditioned iterations algorithm converges if the operator $T : \\vec{x} \\rightarrow ( Id . + P^{-1} ( \\vec{y} - M . ) ) \\vec{x}$ is a strict contraction, i.e if there exists $\\rho \\in [0,1[$ such that\n",
    "\n",
    "\\begin{align*}\n",
    "  \\forall (x,x') \\in \\mathbb{R}^n \\times \\mathbb{R}^n \\; &\\|Tx - Tx'\\| \\leq \\rho \\|x-x'\\| \\\\\n",
    "  &\\|(Id-P^{-1}M)(x-x')+P^{-1}\\vec{y}-P^{-1}\\vec{y}\\| \\leq \\rho \\|x-x'\\| \\\\\n",
    "  &\\|(Id-P^{-1}M)(x-x')\\| \\leq \\rho \\|x-x'\\| \\\\\n",
    "\\end{align*}\n",
    "\n",
    "This condition reduces to the constraint that the maximal singular value of $(Id-P^{-1}M) .$ should lie inside $[0,1[$.\n",
    "Using the property of commuting matrices implying that they are diagonalizable in the same eigenspace, we can say that the eigen values of the sum $Id-P^{-1}M$ are the sum of the eigen values of $Id$, which are respectively identically $1$ and $-P^{-1}M$.\n",
    "\n",
    "This general property of commuting matrices can be trivially obtained for the specific case of the Identity:\n",
    "\n",
    "\\begin{align*}\n",
    "  (Id-A) \\vec{x} &= x - \\lambda \\vec{x} \\quad \\forall \\lambda \\in \\lambda(A)_i, i \\in [0,\\dots,n] \\\\\n",
    "  (Id-A) \\vec{x} &= (1-\\lambda) \\vec{x} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "This help us to derive the initial constraint over $Id-P^{-1}M$ as a constraint only over $P^{-1}M$:\n",
    "\t\n",
    "\\begin{equation}\n",
    "  0 \\le \\lambda(P^{-1}M)_{max} \\leq 1  \n",
    "\\end{equation}\n",
    "\n",
    "In addition, the preconditionner could feature some useful properties, like the fact that $P^{-1}$ is easy to compute, and eventually $P$ has a low condition number, or $P$ may even carry some a-priori informations over the problem.\n",
    "\n",
    "We can notice that, if we choose the preconditionner such that $P^{-1} = \\frac{1}{\\lambda(M^\\intercal M)_{max}} M^\\intercal $ we get an instance of the gradient descent algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are Krylov subspaces, and how are they linked to the solution of a linear system\n",
    "  \n",
    "Let reconsider the equation of preconditionned iterations, where we will incorporate the preconditionner $P$ directly to $M$ and $y$ such that we have our new $M$ equivalent to $P^{-1}M$ and our new $y$ equivalent to $P^{-1}\\vec{y}$. The preconditionned iterations now reads\n",
    " \n",
    "\\begin{equation}\n",
    "  \\vec{x^{k+1}} = (Id-M)\\vec{x^k} + \\vec{y}\n",
    "\\end{equation}\n",
    "\n",
    "assuming $\\vec{x^{0}} = \\vec{y}$, we can write\n",
    "\n",
    "\\begin{align*}\n",
    "  \\vec{x^{0}} &= \\vec{y} \\\\\n",
    "  \\vec{x^{1}} &= (Id-M)\\vec{y} + \\vec{y} \\\\\n",
    "  \\vec{x^{2}} &= (Id-M)^2\\vec{y} + (Id-M)\\vec{y} + \\vec{y} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Now, using the matrix version of the Taylor expansion exposed in the previous section, we can give another proof of the convergence of the $x^k$ series:\n",
    "\n",
    "\\begin{align*}\n",
    "  \\vec{x^{k}} &= \\sum_{n=0}^{k} (Id-M)^n \\vec{y} \\\\\n",
    "  \\underset{k\\rightarrow \\infty}{lim} \\; \\vec{x^{k}} &= (Id-(Id-M))^{-1} \\vec{y} \\\\\n",
    "  \\underset{k\\rightarrow \\infty}{lim} \\; \\vec{x^{k}} &= M^{-1} \\vec{y} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "The fact that the inverse of $M$ in $\\vec{y}$ can be expressed in a basis made of vectors of the form $M^n \\vec{y}, n \\in 0,1,\\dots$ was first discovered by Krylov, using a more general approach based on the Cayley-Hamilton theorem, which do not impose condition over the spectral radius, and where the polynomial in $M$ was the characteristic polynomial of the matrix $M$.\n",
    "\n",
    "In the general case, the space $\\mathcal{K}_r$ spanned by the $r$ first vectors : $M^n \\vec{y}, n \\in 0,1,\\dots, r-1 $ is known as the order-r Krylov subspace, and it can be shown that its basis, made of the vectors $M^n \\vec{y}, n \\in 0,1,\\dots, r-1$ is free while $r \\leq r_{max}$.\n",
    "\n",
    "First, proving that the basis of $\\mathcal{K}_1$ is free is direct, because there is only one vector, then, as linear combinations of Krylov basis generates a vector in $\\mathbb{R}^n$ for the $n \\times n$ matrix $M$ Krylov subspace $\\mathcal{K}_r$ dimension cannot exceed $n$ in any case, so a free basis of $\\mathcal{K}_r$ cannot contain more than $n$ vectors.\n",
    "\n",
    "Now we have to show that if only one more vector $M^{k+1} \\vec{y}$ makes the basis $M^n \\vec{y}, n \\in 0,1,\\dots,k, k+1$ linearly dependant, then $k+1=r_{max}$ and all following basis vectors will also be linearly dependant.\n",
    "\n",
    "We will give the intuition of the proof using a reccurence strategy showing that if $M^{k+1} \\vec{y} \\in \\mathcal{K}_{r_{max}}$ then $ \\forall q > 0, M^{k+q} \\in \\mathcal{K}_{r_{max}}$:\n",
    "\n",
    "We begin assuming that the property is true for one $q_0$ : $\\exists q_0 \\geq 0 \\backslash M^{r+q_0} \\in \\mathcal{K}_r$ and see what happen for $q_0+1$:\n",
    "\n",
    "\\begin{align*}\n",
    "  M^{r+q_0} \\vec{y} &\\in \\mathcal{K}_r \\\\\n",
    "  M^{r+q_0} \\vec{y} &= \\sum_{n=0}^{r-1} \\alpha_n M^{n} \\vec{y} \\\\\n",
    "  M^{r+q_0+1} \\vec{y} &= M \\sum_{n=0}^{r-1} \\alpha_n M^{n} \\vec{y} \\\\\n",
    "  &= \\sum_{n=0}^{r-1} \\alpha_n M^{n+1} \\vec{y} \\\\\n",
    "  &= \\left( \\sum_{n=1}^{r-1} \\alpha_{n-1} M^{n} \\vec{y} \\right) + \\alpha_{r-1} M^{r} \\vec{y} \\; \\text{ and as } r-1 < r \\leq r+q_0 \\\\\n",
    "  &= \\left(\\sum_{n=1}^{r-1} \\alpha_{n-1} M^{n} \\vec{y} \\right) + \\alpha_{r-1} \\left(\\sum_{n=0}^{r-1} \\beta_n M^{n} \\vec{y} \\right)\\\\\n",
    "  &= \\sum_{n=0}^{r-1} \\gamma_{n} M^{n} \\vec{y}\\\\\n",
    "  M^{r+q_0+1} &\\in \\mathcal{K}_r\n",
    "\\end{align*}\n",
    "\n",
    "This property shows us that, in $\\mathbb{R}^n$, the $\\mathcal{K}_{r}$ subspace is possibly of lower dimension than $n$, and that a solution of a linear set of equations may be expressed or approximated within this framework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw Krylov basis and numerical stability\n",
    "\n",
    "From the previous section, we can derive the Krylov matrix $K$:\n",
    "\n",
    "\\begin{equation}\n",
    "  \\begin{pmatrix} \\vec{y} \\; M\\vec{y} \\; M^2\\vec{y} \\; \\dots \\; M^{n-1}\\vec{y} \\end{pmatrix}\n",
    "\\end{equation}\n",
    "\n",
    "Unfortunately, in many cases, this basis cannot be used as is to express the solution for numerical reasons: assuming $M$ can be diagonalized and has $n$ different eigenvalues $\\lambda(M)_0, \\lambda(M)_1, \\dots \\lambda(M)_{n-1}$ associated with the eigenvectors $ \\vec{v_0},  \\vec{v_1}, \\dots  \\vec{v_{n-1}}$ we can write\n",
    "\n",
    "\\begin{align*}\n",
    "  \\vec{y} = \\sum_{i=0}^{n-1} \\alpha_i  \\vec{v_i} \\\\\n",
    "  M^n \\vec{y} = \\sum_{i=0}^{n-1} \\alpha_i \\lambda(M)_i^n  \\vec{v_1}\n",
    "\\end{align*}\n",
    "\n",
    "Unfortunately, when the ratio $ \\left( \\frac{\\alpha_{max}}{\\alpha_i} \\right) \\left( \\frac{\\lambda(M)_{max}}{\\lambda(M)_i} \\right)^n $ where $max$ stands for the index of the maximum eigenvalue, exceeds $2^n$, the accuracy of any krylov subspace method would collapse. n being the number of significand bits in the floating point representation of a computer.\n",
    "\n",
    "In this case, all numerical approximation of the Krylov basis vectors exceeding order $n$ becomes colinear to the maximum eigenvector $ \\vec{v_{max}}$.\n",
    "In practice, this drawback has a high probability of occurrence especially in high dimensional settings, with for instance $k \\gg n$ dimensions, where the first $k$ vectors of the Krylov basis are linearly independants, though potentially carrying some informations about the solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arnoldi orthogonalization strategy for Krylov basis\n",
    "\n",
    "To overcome the numerical stability problem, Arnoldi designed a method able to exploit Krylov basis using Gram-Schmidt orthonormalization process.\n",
    "\n",
    "This simple method can be summarized as follows:\n",
    "\n",
    "\\begin{algorithm}\n",
    "  \\caption{Generate Arnoldi basis vectors $\\vec{v_j}$}\n",
    "    \\begin{algorithmic}\n",
    "      \\STATE \\textbf{Initialization}\n",
    "      \\STATE $\\vec{v_0} = \\frac{\\vec{y}}{\\| \\vec{y} \\|}$\n",
    "      \\STATE $j = 1$\n",
    "      \\STATE $\\vec{w} = M \\vec{v_{0}}$\n",
    "      \\STATE \\textbf{Iterations}\n",
    "      \\WHILE{$\\vec{w} \\neq \\vec{0}$ and $j < n$}\n",
    "      \\STATE $\\vec{w} = M \\vec{v_{j-1}}$\n",
    "      \\FORALL{$i \\in 0,\\dots j-1$}\n",
    "        \\STATE $\\alpha = \\langle \\vec{w},\\vec{v_i} \\rangle $\n",
    "        \\STATE $\\vec{w} = \\vec{w} - \\alpha \\vec{v_i}$\n",
    "      \\ENDFOR\n",
    "      \\IF{$\\vec{w} \\neq \\vec{0}$}\n",
    "        \\STATE $\\vec{v_j} = \\frac{\\vec{w}}{\\| \\vec{w} \\|}$\n",
    "      \\ENDIF\n",
    "        \\STATE $j=j+1$\n",
    "       \\ENDWHILE\n",
    "    \\end{algorithmic}\n",
    "\\end{algorithm}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define $h_{ij}$ such that:\n",
    "\t\n",
    "\\begin{align*}\n",
    "  h_{ij} = \n",
    "  \\begin{cases}\n",
    "    \\langle M\\vec{v_j},\\vec{v_i} \\rangle \\; \\text{ if } \\; j \\geq i\\\\\n",
    "    \\| M\\vec{v_j} - \\sum_{k=0}^{j} \\langle M\\vec{v_j},\\vec{v_k} \\rangle \\vec{v_k} \\| \\; \\text{ if } \\; i = j+1\\\\\n",
    "    0 \\; \\text{ otherwise }\n",
    "  \\end{cases}\n",
    "\\end{align*}\n",
    "\n",
    "Where the two first cases corresponds respectively to the orthogonalization coefficients, and the normalization coefficients of the Gram-Schmidt process described earlier.\n",
    "\n",
    "From there, we can define a $(p+1) \\times p$ matrix $H_{p+1 p}$ as follows:\n",
    "\n",
    "\\begin{equation}\n",
    "  H_{p+1 p} =\n",
    "  \\begin{pmatrix}\n",
    "   h_{0 0} & h_{0 1} & \\dots  & h_{0 p} \\\\\n",
    "   h_{1 0} & h_{1 1} & \\dots  & h_{1 p} \\\\\n",
    "      0\t   & h_{2 1} & \\ddots & \\vdots \\\\\n",
    "      0\t   & \\ddots  & \\ddots & h_{pp} \\\\\n",
    "      0    &    0    &   0    & h_{p+1 p} \\\\\n",
    "  \\end{pmatrix}\n",
    "\\end{equation}\n",
    "\n",
    "If we define $V_p$ as the matrix containing all first $p$ Arnoldi basis vectors $ \\vec{v_0},\\vec{v_1},\\dots,\\vec{v_{p-1}}$ as column vectors, given the orthonormal property obtain through Gram-Schmidt, we know that $V_p$ is a unitary matrix, so we have\n",
    "\t\n",
    "\\begin{equation}\n",
    "  V_p^\\intercal V_p = Id_p\n",
    "\\end{equation}\n",
    "\n",
    "Using the matrix $H_{p+1 p}$, we can write:\n",
    "\n",
    "\\begin{equation}\n",
    "  M V_p = V_{p+1} H_{p+1p}\n",
    "\\end{equation}\n",
    "\n",
    "The matrix product $V_{p+1} H_{p+1p}$ allows us to perform the column-wise inversion of the Gram-Schmidt process over $V_{p+1}$ to recover $M V_p$.\n",
    "\n",
    "From the recurrence relationship based on $H_{p+1 p}$, we can write:\n",
    "\n",
    "\\begin{align*}\n",
    "  M V_p = V_{p+1} H_{p+1p} \\\\\n",
    "  V_p ^\\intercal M V_p = V_p ^\\intercal V_{p+1} H_{p+1p}\n",
    "\\end{align*}\n",
    "\n",
    "Using the unitary property of $V_p$, we see that $V_p^\\intercal V_{p+1}$ is a $p \\times p+1$ matrix composed of a $p \\times p$ identity matrix and a last $p+1 ^{th} $ column containing, at each row $i$, the result of $ \\langle \\vec{v_{p+1}},\\vec{v_i} \\rangle $ which are all zeros.\n",
    "We can deduce that $V_p^\\intercal V_{p+1}$ will discard the last row of $H_{p+1p}$, and then give $H_{p p}$ :\n",
    "\n",
    "\\begin{equation}\n",
    "  H_{p p} = V_{p} ^\\intercal M V_{p}\n",
    "\\end{equation}\n",
    "\n",
    "This result is important because, it shows that $H_{p p}$, is the expression of the projection of the linear operator represented by $M$ over the order-p Krylov subspace in the orthonormal basis of Arnolid vectors $V_p$, which can be seen as a rotation matrix.\n",
    "A point that will be very important in the next section, is that, by construction $H_{p p}$ is a Hessenberg superior matrix,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x8545150>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD9NJREFUeJzt3VGM3NV1x/Hf8Tpsw9K4xo3XxU6WpCiGOiSuSdymDo4j\naEDtgylVKUkqQVshHkCJlBdIeHAiFTX0AYm2iloRB5kUREkaYqJIQAABgSqxG4dgwDYU4U1s8EA3\n1MGQuPb65GHG3fEyM/cwM/ufWc73I62Y/c/Z/707zM8zs3PmXnN3AchlwaAnAKB6BB9IiOADCRF8\nICGCDyRE8IGEegq+mV1oZrvN7Bkzu6ZfkwIwt6zb9/HNbIGkZySdJ+kFSdslXeruu2fV0SgADIi7\nW6vjC3s451pJz7r7pCSZ2R2SNkraPbtwU9PlhyRtmHX90mOXFQe76nduKc+o9lK55neXlmueu2fW\ngX+T9FdN3wfOobeVS8Y+UK55LTDUWIt/W//vi9JJX2w6z2OBE50WqJkul0ycUa6Z3DbrwM2Srjjx\n0JK15fNM3Vmu0R8Fan5YuL7V7/0NSX/R9H1gvlpRLlkSuO9E7he/apl5Sb091V8u6WdN3+9rHAMw\n5PjjHpBQL0/190t6d9P3KxrH3uChpsu/0cOAgxN4Sj5sRjYMegZv0ppBT6ALvzfoCZxo+iHp2EOh\n0l6Cv13SGWY2IelFSZdK+mSrwg09DDIcCP7cO2fQE+jCqkFP4EQjG078/z79pbalXQff3afN7GpJ\n96n+kmGzu+/q9nwAqtPLI77c/R5JK/s0FwAV4Y97QEJdN/CEBzDzp6YnOtasevr58onOPhIY7dVA\nzamBmmcL1z8cOMefBWqeDNRMBWrOCtQE3htW4P131QI1vwzUdL5P1LV/H3pGZD7jgZp9hetb/t16\nlsh9K3IffUegJvLO+VjbBh4e8YGECD6QEMEHEiL4QEIEH0iI4AMJEXwgoUrex5cd6lz0lyeXT3RH\nZJ7PBWoi7+m+Xrg+0PA4sqRcE/hoe0zktom8Jx6oGQ+MVevX/6tIY2mgZ2BRoM/h4I7AWCVvD9RE\nei4CRgM1hxfwPj6AGQQfSIjgAwkRfCAhgg8kRPCBhAg+kBDBBxKqpoEntPp/wd8GmiM2RxbriCwS\nUdiYY0lgwYqpyO36hr1H3mgk0PAxHVjqcDRwnsP/Xa7RzwM1gbFGf7Ncczjwe00ExpqMLKLxi9JA\ngXMEGtF0tFzyW4FFUyJ9aHuMBh4AMwg+kBDBBxIi+EBCBB9IiOADCRF8ICGCDyQ0fxp4Ij4daPK5\nLdKkUmrQiZwj0vARaOCJrEKzPNDEsn9vYKzITjARgSaVicAKRUcD983928o1oZVxlhauj+xEFLn9\nIrswRbYMj+y2QwMPgCYEH0iI4AMJEXwgIYIPJETwgYQIPpAQwQcS6qmBx8z2Sjoo6ZikI+6+tkVN\ndQ08EVcGmjn+tbRKT6QhpLQNlyQ9GaiJNAIFmmHGAo1ArwUak0YCqw9FtgZbGNiuK9LAo8DqOmPL\nA+cpeG0qUBRoXNK/BGo2BWp2Bmo+2LaBJ7I5WSfHJG1w91d6PA+ACvX6VN/6cA4AFes1tC7pe2a2\n3cyu6MeEAMy9Xp/qr3P3F83snar/A7DL3R/tx8QAzJ2egu/uLzb++7KZ3SVpraQWwb++6fK5ktb3\nMiyAlrZL+q9QZdfBN7OTJS1w90NmNibpE5K+1Lr6um6HARD24cbXce3fQejlEX9c0l31t+u0UNJt\n7n5fD+cDUJGug+/uz0ta3ce5AKjIW2sFnn4pbde1ubDFlqTyii6SRgOnORxpBAo0w0S2d1oSuC9M\nReYTaOsYCTTVTO8NjBVpcHo1UFNqBAqseLMo8DsdDExF95dLxs8v19QWsAIPgBkEH0iI4AMJEXwg\nIYIPJETwgYQIPpAQwQcSooGnG38eWIHnPyInCqweE6oJNAtpRaAmsE3UROD+Mhlp8oks0xPpdok0\nL5VWVIrUnFk+xaKTyjWRXynUSBX5vdlCC0ATgg8kRPCBhAg+kBDBBxIi+EBCBB9IiOADCdHAM1cu\nCDT53Pt84ES/CtScFah5LFATWGUm5P2BmsB2U+OBhqJaYNsvBbb9Km55Fmg4GgvcfpEoLAxk8mjk\n934fDTwAZhB8ICGCDyRE8IGECD6QEMEHEiL4QEIEH0iIBp5BOifQ5POjXYETBc4TWaVnJLDN1vR3\nAmOtC9REVsWJOBKoCTQCabxwfWSbyb3lkoXvKdeMBDJ5eEe5Rh+igQfADIIPJETwgYQIPpAQwQcS\nIvhAQgQfSKgYfDPbbGY1M3ui6dhiM7vPzPaY2b1mtmhupwmgn4oNPGb2UUmHJN3q7h9oHLtB0pS7\n/4OZXSNpsbtf2+bnaeDpRWQln/sD55mONMxEGoEiDV/bAzWRhpjAtlUKNB31pcnnW+VTjFxcrpmO\n3H4/D9Q8F6j5w+4beNz9UUmvzDq8UdKWxuUtki4KzALAkOj2Nf5Sd69JkrsfUGzXRgBDol9/3Jvb\nhn8AfRV5odVKzczG3b1mZsskvdS5/Pqmy+dKWt/lsADa+5GkyId34sE3nbgR+d2SLpd0g6TLJG3t\n/OPXBYcB0L1zGl/HbW5bGXk773ZJ/ynpfWb2UzP7a0lflvTHZrZH0nmN7wHME8VHfHf/VJurzu/z\nXABUhM49ICFW4Hkr+Fig8ebhwt9fJWk08K7s4cj9JTBWccWbWIlqLftTTjQaWdFmf6Eg8o717HaX\nVmqBmrMDNZGmpFFW4AEwg+ADCRF8ICGCDyRE8IGECD6QEMEHEiL4QELdfjoPw+ThwOo65wUaUB7Y\nFhhsX6DmwkBNoPFmKtIs9ItyyeFTAucprXpzWuAc/WrOiYg0SbXHIz6QEMEHEiL4QEIEH0iI4AMJ\nEXwgIYIPJETwgYRo4MnigUCTz5+uLdd89/2BwXYHataUS47uDJzn9EDN0UDNqZ2vXhJpOIpsU/Z6\noOZQoCbQuNQBj/hAQgQfSIjgAwkRfCAhgg8kRPCBhAg+kBDBBxKigQczvhtoQFl1crnmqYne5yIp\ntk1U5C4cqSmssDMVabyJ/N6BRqBFgdWSDva29R2P+EBCBB9IiOADCRF8ICGCDyRE8IGECD6QUDH4\nZrbZzGpm9kTTsU1mts/MdjS+IlunABgSkc6GWyT9k6RbZx2/0d1v7P+UMNSeijT5LCnX/E9grNqr\ngaK3BWoCTTNF04GaQOONtpdLDi4PnGdxoKa94iO+uz8q6ZUWV/Xj1gQwAL28xr/azB43s6+a2aK+\nzQjAnOs2+F+R9F53Xy3pgCSe8gPzSFcf0nH3l5u+vVnSdzr/xPVNl8+VtL6bYQF09Iik74cqo8E3\nNb2mN7Nl7n6g8e3Fkp7s/OPXBYcB0L31OvFB9e/bVhaDb2a3S9ogaYmZ/VTSJkkfN7PVko5J2ivp\nyu4nC6BqxeC7+6daHL5lDuYCoCLm3tsH+osDmLn02pyOgXloZWBBjz2R++adgZrADkEqzWcqcI6z\nAjWR3+m5csnIGeWa6QVy95Zvu9OyCyRE8IGECD6QEMEHEiL4QEIEH0iI4AMJEXwgIRp4MLwuCDT5\n3Bu5/+4sl4ye3fn6w7cFxok08EQWDgnsyDMa+CT8YaOBB8AMgg8kRPCBhAg+kBDBBxIi+EBCBB9I\niOADCdHAg/ntg28v1/zk9XLNxFjn6yf7lJPIJjn7IyeK7GdDAw+AJgQfSIjgAwkRfCAhgg8kRPCB\nhAg+kBDBBxKigQdvfecFmnwe2FYoKKzQIym2+fRkoObUQE3Eb9PAA2AGwQcSIvhAQgQfSIjgAwkR\nfCAhgg8kRPCBhIoNPGa2QtKtksYlHZN0s7v/o5ktlvTvqu/3s1fSJe5+sMXP08CD4fcHhSafHx4J\nnCSwdM6i08s1Bx8LjLU2UDPaUwPPUUmfc/dVkj4i6SozO1PStZLud/eVkh6U9PnAuQAMgWLw3f2A\nuz/euHxI0i5JKyRtlLSlUbZF0kVzNUkA/fWmXuOb2emSVkv6gaRxd69J9X8cJC3t9+QAzI1w8M3s\nFEnflPTZxiP/7D8OzO2nfQD0TeTjRDKzhaqH/uvuvrVxuGZm4+5eM7Nlkl5qf4brmy6fK2l9d7MF\n0MHDkh4JVYaCL+lrkp5295uajt0t6XJJN0i6TNLWFj/XcF1wGADd+1jj67i/a1tZDL6ZrZP0aUk7\nzezHqj+l/4Lqgb/TzP5G9Q8ZX9LDjAFUqBh8d39M0kibq8/v73QAVIEVeICI8cAqPrVI481ZgZrI\nCjwd/qT2/5axAg+AGQQfSIjgAwkRfCAhgg8kRPCBhAg+kBDBBxKigQfol5WBJp89pa26pFgDT2As\nvYsGHgAzCD6QEMEHEiL4QEIEH0iI4AMJEXwgIYIPJEQDD1CliUDjzWTkRJEtvXrbQgvAWwzBBxIi\n+EBCBB9IiOADCRF8ICGCDyRE8IGEorvlAuiHyV8GiiKr6zzQ0zR4xAcSIvhAQgQfSIjgAwkNIPiP\nVD9kz5jz3Jtv85Xm55zrBhD871c/ZM+Y89ybb/OV5uec63iqDyRUyfv4a9bMrAXwwgvSaae1XBtg\naDHnuTff5isNes7vKFbs2NH+uopW4AEwCO1W4Jnz4AMYPrzGBxIi+EBClQbfzC40s91m9oyZXVPl\n2N0ys71m9hMz+7GZRfY4rpSZbTazmpk90XRssZndZ2Z7zOxeM1s0yDnO1mbOm8xsn5ntaHxdOMg5\nNjOzFWb2oJk9ZWY7zewzjeNDfTt3UlnwzWyBpH+WdIGkVZI+aWZnVjV+D45J2uDuv+/uawc9mRZu\nUf02bXatpPvdfaWkByV9vvJZddZqzpJ0o7uvaXzdU/WkOjgq6XPuvkrSRyRd1bjvDvvt3FaVj/hr\nJT3r7pPufkTSHZI2Vjh+t0xD/JLI3R+V9MqswxslbWlc3iLpokonVdBmzlL9th467n7A3R9vXD4k\naZekFRry27mTKu/QyyX9rOn7fY1jw84lfc/MtpvZFYOeTNBSd69J9TutpKUDnk/U1Wb2uJl9dVif\nNpvZ6ZJWS/qBpPF5ejsP7yPZEFnn7msk/YnqT/E+OugJdWE+vGf7FUnvdffVkg5IunHA83kDMztF\n0jclfbbxyD/7dp0Pt7OkaoO/X9K7m75f0Tg21Nz9xcZ/X5Z0l+ovWYZdzczGJcnMlkl6acDzKXL3\nl32mqeRmSR8e5HxmM7OFqof+6+6+tXF43t3Ox1UZ/O2SzjCzCTM7SdKlku6ucPw3zcxObvwrLzMb\nk/QJSU8OdlYtmU58fXy3pMsbly+TtHX2DwyBE+bcCM5xF2v4buevSXra3W9qOjYfbueWKu3ca7xF\nc5Pq/+BsdvcvVzZ4F8zsPao/yrvqn2u4bdjmbGa3S9ogaYmkmqRNkr4t6RuS3qX6FoyXuPv/DmqO\ns7WZ88dVf+18TNJeSVcef/08aGa2TvXP4O5U/b7gkr4gaZukOzWkt3MntOwCCfHHPSAhgg8kRPCB\nhAg+kBDBBxIi+EBCBB9IiOADCf0aiYe9GCnSvuMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x4c2b6d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Arnoldi method for basis orthogonalization / equivalent\n",
    "to matrix factorization as a orthogonal/Hessenberg superior/orthogonal\n",
    "\"\"\"\n",
    "size = 25\n",
    "M = np.random.rand(size,size)\n",
    "y = np.random.rand(size)\n",
    "\n",
    "#Initialize matrix Vp and Hpp\n",
    "Vp = np.zeros((size,size))\n",
    "Hpp = np.zeros((size,size))\n",
    "Vp[:,0] = y/np.linalg.norm(y)\n",
    "\n",
    "for i in np.arange(1,size+1):\n",
    "    #Compute next Vp\n",
    "    v = np.dot(M,Vp[:,i-1])\n",
    "    #orthogonalize\n",
    "    orth = np.dot(Vp[:,:i].T,v)\n",
    "    Hpp[:i,i-1] = orth\n",
    "    if( i < size ):\n",
    "        v = v - np.dot(Vp[:,:i],orth)\n",
    "        #Update Hpp\n",
    "        Hpp[i,i-1] = np.linalg.norm(v)\n",
    "        #Normalize vector and update V\n",
    "        Vp[:,i] = v / Hpp[i,i-1]\n",
    "\n",
    "#check orthogonality of Vp\n",
    "assert( np.allclose(np.identity(size),np.dot(Vp,Vp.T)) )\n",
    "\n",
    "#Check the equivalence between M and VptHppVp\n",
    "assert( np.allclose(Hpp,np.dot(Vp.T,np.dot(M,Vp))) )\n",
    "\n",
    "#Show matrix Hpp\n",
    "plt.imshow(Hpp,interpolation='none')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Symmetric matrices and Lanczos method for Krylov basis\n",
    "\n",
    "If the matrix $M$ is symmetric, following the relation found earlier related to the unitary change of basis $V_p$, it can be easily shown that $H_{p p}$ is also a symmetric matrix:\n",
    "\t\n",
    "\\begin{align*}\n",
    "  H_{p p} &= V_{p} ^\\intercal M V_{p} \\\\\n",
    "  H_{p p}^\\intercal &= (V_{p} ^\\intercal M V_{p})^\\intercal \\\\\n",
    "  H_{p p}^\\intercal &= V_{p} M^\\intercal V_{p}^\\intercal \\\\\n",
    "  H_{p p}^\\intercal &= V_{p} M V_{p}^\\intercal \\\\\n",
    "  H_{p p}^\\intercal &= V_{p}^\\intercal M V_{p} \\\\\n",
    "  H_{p p}^\\intercal &= H_{p p} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "As, by construction, as seen earlier, $H_{p p}$ is a Hessenberg superior matrix, we can conclude that, in the case of $M$ being, symmetric, we have that $H_{p p}$ is both Hessenberg superior and inferior, so it is a tridiagonal symmetric matrix:\n",
    "\n",
    "\\begin{equation}\n",
    "  H_{p p} =\n",
    "  \\begin{pmatrix}\n",
    "    h_{0 0} & h_{0 1} &    0    &    0   &  \\dots &   \\dots    &   0      \\\\\n",
    "    h_{1 0} & h_{1 1} & h_{1 2} &    0   & \\ddots &   \\ddots   & \\vdots   \\\\\n",
    "      0\t    & h_{2 1} & h_{2 2} & \\ddots & \\ddots &   \\ddots   & \\vdots   \\\\\n",
    "    \\vdots  &    0    & h_{3 2} & \\ddots & \\ddots &    0       &    0     \\\\\n",
    "    \\vdots  & \\ddots  &     0   & \\ddots & \\ddots & h_{p-2p-1} &    0     \\\\\n",
    "    \\vdots  & \\ddots  & \\ddots  & \\ddots & \\ddots & h_{p-1p-1} & h_{p-1p} \\\\\n",
    "       0    & \\dots   &  \\dots  &  \\dots &    0   & h_{pp-1}   & h_{pp}   \\\\\n",
    "  \\end{pmatrix}\n",
    "\\end{equation}\n",
    "\n",
    "This property has a huge impact over the Arnoldi orthogonalisation algorithm seen previously: we can see that the Gram-Schmidt orthogonalization process for the vector $M \\vec{v_j}$ will only imply the computation of the correlation with the two previous vectors of the process: $h_{j-2j}$ and $h_{j-1j}$ instead of all previously constructed vectors. The Arnoldi algorithm in this case now reads:\n",
    "\n",
    "\\begin{algorithm}\n",
    "  \\caption{Generate Arnoldi basis vectors $\\vec{v_j}$}\n",
    "    \\begin{algorithmic}\n",
    "      \\REQUIRE $M \\in S^n_{++}$ the positive definite cone\n",
    "      \\STATE \\textbf{Initialization}\n",
    "      \\STATE $\\vec{v_0} = \\frac{\\vec{y}}{\\| \\vec{y} \\|}$\n",
    "      \\STATE $j = 1$\n",
    "      \\STATE $\\vec{w} = M \\vec{v_{0}}$\n",
    "      \\STATE \\textbf{Iterations}\n",
    "      \\WHILE{$\\vec{w} \\neq \\vec{0}$ and $j \\leq n$}\n",
    "      \\STATE $\\vec{w} = M \\vec{v_{j-1}}$\n",
    "      \\IF{$j > 1$}\n",
    "        \\STATE $h_{j-2j-1} = h_{j-1j-2}$\n",
    "        \\STATE $\\vec{w} = \\vec{w} - h_{j-2j} \\vec{v_{j-2}}$\n",
    "      \\ENDIF\n",
    "      \\STATE $h_{j-1j-1} = \\langle \\vec{w},\\vec{v_{j-1}} \\rangle $\n",
    "      \\STATE $\\vec{w} = \\vec{w} - h_{j-1j-1} \\vec{v_{j-1}}$\n",
    "      \\IF{$\\vec{w} \\neq \\vec{0}$}\n",
    "        \\STATE $h_{jj-1} = \\| \\vec{w} \\|$\n",
    "        \\STATE $\\vec{v_j} = \\frac{\\vec{w}}{h_{jj-1}}$\n",
    "      \\ENDIF\n",
    "        \\STATE $j=j+1$\n",
    "       \\ENDWHILE\n",
    "    \\end{algorithmic}\n",
    "\\end{algorithm}\n",
    "\n",
    "The Lanczos method for computing a numerically stable basis of the Krylov subspace then have a constant computation burden at every iteration, instead of a linearly growing number of correlation to compute at each iterations.\n",
    "\n",
    "We can also add an important comment about the fact that $M$, $V_{p} ^\\intercal M V_{p}$ and $H_{p p}$ have the same eigenvalues, which could be easily proven using the unitary property of $V_p$ : $V_p^\\intercal V_p = Id_p$.\n",
    "It appears that finding the eigenvalues of the tridiagonal $H_{p p}$ is often much simpler than performing a SVD directly over $M$, see for instance the Differential Quotient Difference with Shifts (DQDS) algorithm or more recent work by Coakley in \\cite{coakley2013fast} that present a fast $n log(n)$ divide and conquer algorithm to find eigenvalues of such matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAAD8CAYAAACbxyOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACVNJREFUeJzt2l2MXGUBxvHnabcLpS2tpWoKSpuqF1LiBRcoMbSmFxpU\nJF6YqERLMISkaEKMIWpLoBU/EuXDrwuTItEGBSWNGDURFQxC9UJLaJDE0tAihVJIKRRqBWRfL84Z\nPYw7s7Obcc8+u/9fMsnMOe+c857Z+e85M7supQhAjnltTwDA5BAtEIZogTBEC4QhWiAM0QJhiHaa\n2H7B9uoe6zba/sMgY2cC26tsj9meMe8f2w/ZXtf2PKbDjHnRh8X2ftsbupa9Jooh7+8e25dONK6U\nsqSUcqDfkEmMnQmm5Q/8tm+xvW3CyZRydinl3umYU9tmXbR9tPJfJLbnt7HfuWIuvr5zKdr/qM/G\nn7f9V9tHbN9se7Ret97247a/YPsZ24/a/niP7Vwn6XxJ37F9zPa36uVjtjfZ3itpb2PZmvr+cts/\nt/287T9JekvXdptj3297dz32MdvXdI39pO0D9Vy3NK80bI/avsn2E7YP2r7R9oKu4/ys7cP1mEsa\n2+273wFe38/Z3lO/Ltttv8H2r+rt3WV7aWP8T2wfsn3U9u9tv71efpmkiyVdVW/nzsb2r7L9oKQX\nbc/vOu5f2v5GY/u32d4+6PxnvFLKrLpJ2i9pQ9eySyTd2zVmj6TTJS2TdJ+kbfW69ZJekfR1SQsk\nrZP0oqS39djfPZIu7Vo2JunX9bZPqpe9KmlNff+2+naypLWSDnbNrzl2naS19f2zJR2S9KH68VmS\nXpB0nqSRes4vdY5f0jZJuySdVt/ul7S16zivkTRf0gWSjktaOsB+V9VznNfnZ7BL0gpJKyUdlvQX\nSe+QNCrpd5Ku7vr5nFK/3jdIeqCx7pbOz6Zr+7vrn99JjWWd436jpKckvUdV9PskndL2e3No7/G2\nJzD0A6p+eMckPdu4HR8n2ssajy+Q9EjjzfyypJMb62+XtLnH/npFu36cZWtUXd28rMYvAUlf7prf\nWCfacfZ3o6Tr6/tXS7q1sW5hV7T7JL2vsf69kh5tHOfxZnh1XOcOsN9Bov1Y4/Edkr7bePxpSTt7\nPHdZffxL6se9ot04zrINjccflvR3SU9LOq/t9+Uwb7P18viiUsryzk3SpnHGHGzcf0zVb+2Oo6WU\nf/ZZP4iDPZa/XtWZrXv/47L9Ttt3237a9nOSLld1BlM9p8c7Y0spJyQdaTz9dFVv3OZ+msdxpJQy\n1nj8D0mLB9jvIA437p8Y53FnP/Nsf832vno/+1V9/zDRvnq9vh2/UPU6/62U8sdJzHvGm63ReoAx\nb27cXyXpycbj19le2Hh8Ztf6pl5fcPVa/oykf3Xt/8w+87xV0s8knVFKWSbpe/rv8R2S9KbOwHrO\npzWe+6SqY+voPs5++u13mC6WdKGqs+QySavr/XT2NdnXt+Mrkh6WtNL2R4cwzxljtkY7iCtsn2F7\nuaQvqvqM2WFJW20vsH2+pA9I+mmP7RxWddk7kPrMtlPStbYX2j5L0sY+T1ms6sz/iu1zJTW/FLtD\n0oW231V/wXRt13N/LGmL7RW2V6i6nN4x4FT77VcaXsCLVV3SH7W9SNJX9dogJ/X6SlL999qNkj6h\n6vPyt22vHMpsZ4DZGO2gf9r5kaS7VH3ue0TV58qOQ5KOqjor7ZB0eSllb4/tfFPSR+pvoW/qM4fm\nss9IWlLv5/v1rdfYTZK+ZPt5SVtUfb6uBpXycL2t2+u5HlP1Ge6lesh1kv6s6ku3B+v7zePsN8ee\n++1zjL3W9Rv7Q1WX8E9IekjVF1hNN0taa/tZ2zv7bK9Iku0lkn4g6YpSylOllPskbVf12XhWcP2h\nfU6xvV/Sp0opd4+zbr2kHaWUfpesM1J9pnpO0ltLKT0/JyPbbDzTzim2P1hfZi+SdL2kPQQ7u83V\naGfT5cVFqi6ND6r6J41Z9aUL/tecvDwGks3VMy0Qa6TfStuchoGWlFLG/bNa32il6h9Tk4yOXdn2\nFCZl87ylEw/CHLS15xouj4EwRAuEIVogDNECYYgWCEO0QBiiBcIQLRCGaIEwRAuEIVogDNECYYgW\nCEO0QBiiBcIQLRCGaIEwRAuEIVogDNECYYgWCEO0QBiiBcIQLRCGaIEwRAuEIVogDNECYYgWCEO0\nQBiiBcIQLRCGaIEwRAuEIVogDNECYYgWCEO0QBiiBcIQLRCGaIEwRAuEGZlowOirV07HPIZm8503\ntD2FSfpN2xOYgl1tT2BO40wLhCFaIAzRAmGIFghDtEAYogXCEC0QhmiBMEQLhCFaIAzRAmGIFghD\ntEAYogXCEC0QhmiBMEQLhCFaIAzRAmGIFghDtEAYogXCEC0QhmiBMEQLhCFaIAzRAmGIFghDtEAY\nogXCEC0QhmiBMEQLhCFaIAzRAmGIFghDtEAYogXCEC0QhmiBMEQLhCFaIAzRAmGIFggzMtGAzfOX\nTsc8hui3bU9gkt7d9gSm4ETbE5iCB9qewNBwpgXCEC0QhmiBMEQLhCFaIAzRAmGIFghDtEAYogXC\nEC0QhmiBMEQLhCFaIAzRAmGIFghDtEAYogXCEC0QhmiBMEQLhCFaIAzRAmGIFghDtEAYogXCEC0Q\nhmiBMEQLhCFaIAzRAmGIFghDtEAYogXCEC0QhmiBMEQLhCFaIAzRAmGIFghDtEAYogXCEC0QhmiB\nMCNtT2D47m97ApN0ou0JTMHqticwBQfansDQcKYFwhAtEIZogTBEC4QhWiAM0QJhiBYIQ7RAGKIF\nwhAtEIZogTBEC4QhWiAM0QJhiBYIQ7RAGKIFwhAtEIZogTBEC4QhWiAM0QJhiBYIQ7RAGKIFwhAt\nEIZogTBEC4QhWiAM0QJhiBYIQ7RAGKIFwhAtEIZogTBEC4QhWiAM0QJhiBYIQ7RAGKIFwhAtEIZo\ngTBEC4QZaXsC2N32BKbgQNsTmIJT257A0HCmBcIQLRCGaIEwRAuEIVogDNECYYgWCEO0QBiiBcIQ\nLRCGaIEwRAuEIVogDNECYYgWCEO0QBiiBcIQLRCGaIEwRAuEIVogDNECYYgWCEO0QBiiBcIQLRCG\naIEwRAuEIVogDNECYYgWCEO0QBiiBcIQLRCGaIEwRAuEIVogDNECYYgWCEO0QBiiBcIQLRCGaIEw\nIxMNOOecldMxD0Q5te0JTMGiticwKbt3917nUkrvlXbvlQD+r0opHm9532gBzDx8pgXCEC0QhmiB\nMEQLhCFaIMy/Ab09PXtHY/YKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x6d76a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Lanczos method for symmetric matrices orthogonalization / equivalent\n",
    "to matrix factorization as a orthogonal/tridiagonal/orthogonal\n",
    "\n",
    "Although the method appeared numerically unstable in our settings,\n",
    "for dimension exceeding 6, it will be more useful, in the next steps\n",
    "\"\"\"\n",
    "size = 5\n",
    "M = np.random.rand(size,size)\n",
    "M = np.dot(M,M.T) #Symmetric\n",
    "y = np.random.rand(size)\n",
    "\n",
    "#Initialize matrix Vp and Hpp\n",
    "Vp = np.zeros((size,size))\n",
    "Hpp = np.zeros((size,size))\n",
    "Vp[:,0] = y/np.linalg.norm(y)\n",
    "\n",
    "for i in np.arange(1,size+1):\n",
    "    #Compute next Vp\n",
    "    v = np.dot(M,Vp[:,i-1])\n",
    "    #orthogonalize\n",
    "    minId = np.maximum(0,i-2)\n",
    "    orth = np.dot(Vp[:,minId:i].T,v)\n",
    "    Hpp[minId:i,i-1] = orth\n",
    "    if( i < size ):\n",
    "        v = v - np.dot(Vp[:,minId:i],orth)\n",
    "        #Update Hpp\n",
    "        Hpp[i,i-1] = np.linalg.norm(v)\n",
    "        #Normalize vector and update V\n",
    "        Vp[:,i] = v / Hpp[i,i-1]\n",
    "\n",
    "#print str(np.dot(Vp,Vp.T))\n",
    "#print str(Hpp)\n",
    "#print str(np.dot(np.dot(Vp.T,M),Vp))\n",
    "\n",
    "#check orthogonality of Vp\n",
    "assert( np.allclose(np.identity(size),np.dot(Vp,Vp.T)) )\n",
    "\n",
    "#Check the equivalence between M and VptHppVp\n",
    "assert( np.allclose(Hpp,np.dot(Vp.T,np.dot(M,Vp))) )\n",
    "\n",
    "#Show matrix Hpp\n",
    "fig,ax = plt.subplots(1)\n",
    "ax.set_title(\"Hpp tridiagonal matrix\")\n",
    "ax.imshow(Hpp,interpolation='none')\n",
    "ax.get_xaxis().set_visible(False)\n",
    "ax.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Krylov basis and practical resolution of linear equalities\n",
    "\n",
    "As seen earlier, the solution $M^{-1} \\vec{y}$ of $M \\vec{x} = \\vec{y}$ can be expressed in terms of $M^p \\vec{y}, p = 0,1,\\dots,r-1$, which led us to define the order-r Krylov basis.\n",
    "\t\n",
    "We have then seen a way to express the p-order Krylov basis in a more stable way using the Arnoldi basis, and its Lanczos version $V_p$. We can now define $\\vec{x_p}$, the approximation of the solution of $M \\vec{x} - \\vec{y} = 0$ in the order-p Arnoldi basis:\n",
    "\n",
    "\\begin{equation}\n",
    "  \\vec{x_p} = V_p \\vec{s_p}\n",
    "\\end{equation}\n",
    "\n",
    "From this can be defined common error metrics over this approximate solution, using the following vectors:\n",
    "\n",
    "- The error vector: $\\vec{e_p} = \\vec{x} - \\vec{x_p}$\n",
    "- The residual vector: $\\vec{r_p} = \\vec{y} - M \\vec{x_p} $\n",
    "\n",
    "Krylov subspace related methods aims at minimizing $\\vec{e_p}$ or $\\vec{r_p}$ under appropriate norms.\n",
    "One of the most reknown method, first derived by Lanczos aims at minimizing $Lcz(\\vec{x_p})$ the scalar product of those two error vector, that can be also interpreted of the squared norm of the error vector using the scalar product defined by the matrix $M$:\n",
    "\n",
    "\\begin{align*}\n",
    "  Lcz(\\vec{x_p}) &= \\| \\vec{e_p} \\|_M^2 \\\\\n",
    "  &= \\| \\vec{x} - \\vec{x_p} \\|_M^2 \\\\\n",
    "  &= ( \\vec{x} - \\vec{x_p})^\\intercal M (\\vec{x} - \\vec{x_p})\\\\\n",
    "  &= \\vec{e_p}^\\intercal . (M \\vec{x} - M \\vec{x_p})\\\\\n",
    "  &= \\vec{e_p}^\\intercal . ( \\vec{y} - M \\vec{x_p})\\\\\n",
    "  &= \\vec{e_p}^\\intercal . \\vec{r_p}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "The Lanczos method can be viewed as the task of finding the vector $\\vec{x_p} \\in \\mathcal{K}_p$ such that its distance to $\\vec{x}$ is minimum in the inner product space $\\mathcal{M}^n$ defined by $M$ from $\\mathbb{R}^n$, for this method, we need $M$ to be symmetric and positive definite.\n",
    "\n",
    "The vector $\\vec{x_p}$ that satisfies this definition is the orthogonal projection of $x$ over $\\mathcal{K}_p$ using the inner product $\\langle .,. \\rangle_M$.\n",
    "\n",
    "Following the definition of the inner product $\\langle .,. \\rangle_M$ and the definition of the orthogonal projection operator,\n",
    "\n",
    "\\begin{aligned}\n",
    "    P_{\\mathcal{K}_p} : \\mathcal{M}^n &\\rightarrow \\mathcal{K}_p \\\\\n",
    "    \\vec{x} &\\rightarrow \\vec{x_p}\n",
    "\\end{aligned}\n",
    "\n",
    "in the space $\\mathcal{M}^n$ we have that the range $\\mathcal{K}_p$ and the nullspace $\\mathcal{M}^n \\setminus \\mathcal{K}_p$ are orthogonal subspaces in direct sum, which means for us that $\\mathcal{M}^n = \\mathcal{K}_p \\bigoplus_{M} \\mathcal{M}^n \\setminus \\mathcal{K}_p$ and :\n",
    "\n",
    "\\begin{equation}\n",
    "  \\forall (a,b) \\in \\mathcal{M}^n \\times \\mathcal{M}^n \\setminus \\mathcal{K}_p, \\langle a,b \\rangle_M = 0\n",
    "\\end{equation}\n",
    "  \n",
    "By construction, it is obvious that $ \\vec{e_p} = \\vec{x} - \\vec{x_p} \\in \\mathcal{M}^n \\setminus \\mathcal{K}_p$ so we have\n",
    "\t\n",
    "\\begin{align*}\n",
    "  \\langle \\vec{x_p}, \\vec{x} - \\vec{x_p} \\rangle_M &= 0 \\quad \\forall \\vec{x_p} \\in \\mathcal{K}_p, \\text{ and } x \\in \\mathbb{R}^n \\\\\n",
    "  \\vec{x_p} M ( \\vec{x} - \\vec{x_p} ) &= 0 \\\\\n",
    "  \\vec{x_p} . (\\vec{y} - M \\vec{x_p}) &= 0 \\\\\n",
    "  \\vec{x_p} . \\vec{r_p} &= 0\n",
    "\\end{align*}\n",
    "\n",
    "So we have that, in $\\mathbb{R}^n$, the residual $\\vec{r_p}$ is always orthogonal to all vectors in $\\mathcal{K}_p$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lanczos iterations : tridiagonal matrix factorization\n",
    "\n",
    "Let's recall that $Lcz(\\vec{x_p})$ reads:\n",
    "\n",
    "\\begin{align*}\n",
    "  Lcz(\\vec{x_p}) &= ( \\vec{x} - \\vec{x_p})^\\intercal M (\\vec{x} - \\vec{x_p}) \\\\\n",
    "  &= (\\vec{x_p}^\\intercal M \\vec{x_p} + \\vec{x}^\\intercal M \\vec{x} - 2 \\vec{x_p}^\\intercal M \\vec{x}) \\\\\n",
    "\\end{align*}\n",
    "\n",
    "We can further develop this expression using $\\vec{x_p} = V_p \\vec{s_p}$, and delete the constant terms to get:\n",
    "\n",
    "\\begin{align*}\n",
    "  Lcz(\\vec{x_p}) &= \\vec{x_p}^\\intercal M \\vec{x_p} + \\vec{x}^\\intercal M \\vec{x} - 2 \\vec{x_p}^\\intercal M \\vec{x} \\\\\n",
    "  Lcz(\\vec{s_p}) &= \\vec{s_p}^\\intercal V_p^\\intercal M V_p \\vec{s_p} - 2 \\vec{s_p}^\\intercal V_p^\\intercal \\vec{y} \\\\\n",
    "  &= 2( \\frac{1}{2} \\langle V_p^\\intercal M V_p \\vec{s_p}, \\vec{s_p} \\rangle - \\langle V_p^\\intercal \\vec{y}, \\vec{s_p} \\rangle ) \\\\\n",
    "  &= 2( \\frac{1}{2} \\langle H_{pp} \\vec{s_p}, \\vec{s_p} \\rangle - \\langle V_p^\\intercal \\vec{y}, \\vec{s_p} \\rangle ) \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Whose minimization can be reduced to\n",
    "\n",
    "\\begin{equation}\n",
    " Lcz(\\vec{s_p})= \\frac{1}{2} \\langle H_{pp} \\vec{s_p}, \\vec{s_p} \\rangle - \\langle V_p^\\intercal \\vec{y}, \\vec{s_p} \\rangle \\\\\n",
    "\\end{equation}\n",
    "\n",
    "which is a famous case of convex, and differentiable objective whose jacobian along $\\vec{s_p}$ is:\n",
    "\n",
    "\\begin{equation}\n",
    "  \\frac{d Lcz(\\vec{s_p})}{d \\vec{s_p}} = H_{pp} \\vec{s_p} - V_p^\\intercal \\vec{y}\n",
    "\\end{equation}\n",
    "\n",
    "That vanish when $\\vec{s_p}$ is the solution of $H_{pp} \\vec{s_p} - V_p^\\intercal \\vec{y} = 0$\n",
    "\n",
    "The positive definiteness of the tridiagonal $H_{pp}$ can be exploited by factorizing it using an alternative Choleski decomposition following the pattern $L_pD_pL_p^\\intercal$, where $D_p$ is a diagonal matrix, and $L_p$ is an inferior bidiagonal matrix, with all diagonal entries equal to zeros.\n",
    "\n",
    "We can perform the following iterative factorization with, at first step:\n",
    "\n",
    "\\begin{equation}\n",
    "  \\begin{pmatrix}\n",
    "    h_{00}\n",
    "  \\end{pmatrix} =\n",
    "  \\begin{pmatrix}\n",
    "    1\n",
    "  \\end{pmatrix}\n",
    "  \\begin{pmatrix}\n",
    "    h_{00}\n",
    "  \\end{pmatrix}\n",
    "  \\begin{pmatrix}\n",
    "    1\n",
    "  \\end{pmatrix}\n",
    "\\end{equation}\n",
    "\n",
    "And at following steps, we can incrementally build the Choleski decomposition $L_pD_pL_p^\\intercal$ of the tridiagonal $H_{pp}$:\n",
    "\n",
    "\\begin{align*}\n",
    "  H_{pp} &=\n",
    "  \\begin{pmatrix}\n",
    "    &   &            &   & \t    & 0       \\\\\n",
    "    &   & H_{p-1p-1} &   & \t    & \\vdots  \\\\\n",
    "    &   &            &   & \t    & 0       \\\\\n",
    "    &   &            &   & \t    & h_{p-1p}\\\\\n",
    "    & 0 & \\dots      & 0 & h_{pp-1} & h_{pp}\n",
    "  \\end{pmatrix} \\\\ &=\n",
    "  \\begin{pmatrix}\n",
    "    &   &        &    & \t &    0    \\\\\n",
    "    &   & L_{p-1}&    & \t &  \\vdots \\\\\n",
    "    &   &        &    & \t &    0    \\\\\n",
    "    &   &        &    & \t &    0    \\\\\n",
    "    & 0 & \\dots  & 0  & l_{pp-1} &    1\n",
    "  \\end{pmatrix}\n",
    "  \\begin{pmatrix}\n",
    "    &   &        & \t&   &    0    \\\\\n",
    "    &   & D_{p-1}& \t&   &  \\vdots \\\\\n",
    "    &   &        & \t&   &    0    \\\\\n",
    "    &   &        & \t&   &    0    \\\\\n",
    "    & 0 & \\dots  & \\dots & 0 &  d_{pp}\n",
    "  \\end{pmatrix}\n",
    "  \\begin{pmatrix}\n",
    "    &   &        \t  & \t  &   &    0      \\\\\n",
    "    &   & L_{p}^\\intercal & \t  &   &  \\vdots   \\\\\n",
    "    &   &        \t  & \t  &   &    0      \\\\\n",
    "    &   &       \t  & \t  &   &  l_{p-1p} \\\\\n",
    "    & 0 & \\dots  \t  & \\dots & 0 &     1 \n",
    "  \\end{pmatrix}\n",
    "\\end{align*}\n",
    "\t\n",
    "Where $l_{pp-1} = \\frac{h_{pp-1}}{d_{p-1p-1}}$ and $d_{pp}=h_{pp}-l_{pp-1}h_{pp-1}$\n",
    "\n",
    "In the meantime, the vector $\\vec{y_p} = V_p^\\intercal \\vec{y}$ has also to be updated as follows:\n",
    "\n",
    "\\begin{equation}\n",
    "  \\vec{y_p} =\n",
    "  \\begin{pmatrix}\n",
    "    \\begin{pmatrix}\n",
    "      \\\\\n",
    "      \\vec{y_{p-1}}\n",
    "      \\\\\n",
    "      \\\\\n",
    "    \\end{pmatrix} \\\\\n",
    "    \\langle \\vec{v_p},\\vec{y} \\rangle  \\\\\n",
    "  \\end{pmatrix}\n",
    "\\end{equation}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None, None]"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAACxCAYAAAC7mqV5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEflJREFUeJzt3XmwZGV9xvHvMzPs+y4QHBxiXDDGkBI1FSBFjIpLiHGJ\n0ZARXFBcUkkpiYoso6KJiGuwNCIVBsNmEEURtQAXBDQRS6JWBDLswrDIACKgYd788b4NZ5rbffvO\n3Nt9Yb6fqlvT3eft83v7nPecfvqc0z0ppSBJktZvCybdAUmSNHkGAkmSZCCQJEkGAkmShIFAkiRh\nIJAkSTwKA0GSu5PsPmDa0iTfGaXtfJBkcZLVSebNekry4yT7Trofa6N//U8x/dwkBw2Ytsa6GNZ2\nvkhydZL9B0z7ZJJ3jTifjZOck2RVktPXsi/zelubTpJ3JPn0pPsxnyQ5Ksnydnu3JHclyaT7NUiS\n/ZJcP+l+dM237WLR2j4xydXAa0opF3QeWwq8tpSyz2x0rq/ehcDyUspnh7UrpWwxzawe/OGFEdrO\nB2P5oYgkJwHXl1KOHNqZUp4yjv7MxFRjcYiBy7OU8vxRnztC23mtlPLGGTR/KbADsE1Zyx8uma/b\nWpL9gFNKKbsNa1dKef+YujQWSa4BdgR+AzwA/BRYDnx6huu4AJRSrge2nOVuzoVx7U9n6/1qrObi\nk+dEfukoycJJ1F1fuHzXa4uBK9YmDKzruBnDuAvT7LMepWO/AC8opWxFXb8fAP4BOHGivVpPzNcx\nNaeHotshy39M8pMktyc5McmGbdp+Sa5vh+JuTbIiySsHzOe9wD7AJ9phqY+1x1cnOSzJFcAVnceW\ntNvbJvlSkjuTXArs0TffbtvnJ7mstb02yVF9bf8myTWtr0d0D8cm2TDJR5LcmOSGJB9OskHf6/z7\nJCtbm1d35ju07gjL921JLm/L5TNJdmyHs+9M8vUkW3Xan5HkpiR3JPlmkie1x18HvAo4vM3ni535\nH57kR8Avkyzse91fSXJcZ/6nJfnMqP2fkAVJPt4Of/+0e0g9yYVJDmm3FyQ5rq3vq4AXdGfS13ZJ\nkvOT3JbkliSnJNmy03avzjo+oy2nZZ3pr0tyZXv+2Ul27kxbneTQJFck+UWST3SmDa07TJKTen0Y\nMEaXtmlHA0cCr2hj4+D2+CFt+d2e5KtJHtvX52Hb5ZZJTm59vjqdUxepp3UuSnJ8ktuAh20PqYeq\nz0iyvPXpR0ken7qvWZm6nT670/7Vra93Jbkqyevb45sC5wK7pB66vSvJY9r8z2zzXwUszZqHx1+e\nur/avN0/oG1X242y7OeRAJRS7i6lfBn4S+prffKUjZPd237jziRfA7bvTOs/pTblMu+0PzzJz1P3\nl6+Z4fj4TpIPtu3hf5M8rzN9aN2hC6P24Y1tW7wzybK2jV2cus88Ncmi1nbr1NNot7Rt4Jwku7Rp\nM36/SrJBkh8meXN7fEHbDo4Ytf+zopSyVn/A1cD+fY+9Gvh2X5vLgV2ArYGLgGVt2n7Uw1UfBDYA\n9gV+CTx+QL0LgUP6HlsNfK3Ne6P22APAknb7tPa3MbAncENf/7pt9wX2bLefAtwE/Fm7/2TgbuBZ\n1NMsHwTu771+YBlwMbBd+/sucEzf6zwKWAgcANwDbDVC3cWtjwuGrIOLqRvmzsBK4AfAU4ENgfOB\nd/etn03b8j4e+GFn2km9ddM3/8va+tuo81jvde8E3Az8MTVQXAVsurZjajbH4oB2S9u6eGtbFy8H\nVgFb948x4A3Uw6i9sXtBd130td0D+JM2NrYDvgkc36ZtAFwDvLnVfHEbO73tYH/gVuD3WtuPAd/q\nG+NfArYAdgNuAZ4zXd3plkt3fY8wRo8CTu4890DqDu13qB8q3gl8dwbb5cnAF9pYXAz8DDi4bx0d\n1ua90RR9Pwr4FfDs1ubf2jJ+R+v/a4EVnfYHALu32/u01/a0zmu/bor53w+8qN3feIplsBz4LLAt\ncCNwwLjH/VxsM8C1wKEDnnMxD+2v9wHu6i0T+vZV0yzz5wE/B57Ylu3yGY6P+4FDqIHmDcCNa7uu\n+17f6lZ3M+BJwH3Ufehi6vb3E+Cg1nZb6ra8UWt/OvCFzrzW5v1qT+D2tlze1ZZ3xjou1nFA3QX8\novN3Dw8PBK/rW1lXdlbOr4GNO9NPB941oN6gBbzfFI8toe4ofk0nYADv6+vf6t7KmKLeh4EPtdvv\nBj7XmbYJawaCq4DndqY/h7ZDaq/zHjpv6tQ37r1HqDtKIPirzv3PA//Suf9m4KwBz926vf4t2v1B\ngWDpFI/t37n/YuA66hvVs8Y5eAf1aUi7pcANfY99D3hV/xij7ghe32n3pwwIBFPUORD4Qbu9L/Xa\njO707/DQm/FngA90pm3Wxu1jO2P0WZ3ppwOHT1d3uuXCwwPBwDHKw98Mz6XtoNv9Be35u3X6vF9f\nve52eT/whM601wMXdNbRNdOsx6OAr3Xuv5C6L0q7v3lbV1sOeP4XgLd0XvtUgeCbUzzWXQZbUd88\nLwdOmMS4n4ttBrgEeMcUj+/WxuUmncc+x4BAMM0yPxF4X2faHjMcH1d0pm3S6u64Nut6ijH6zM79\n/wLe3rl/HJ3A3ffcpwG3d+7P6P2qc//vgP+hBoMp35vm8m9dTxkcWErZtvdHTfX9bujcvpb6iavn\njlLKfUOmj+KGAY/vQP200F9/SkmekeSCdghoFXAoDx0S2wV48OrUUsq91BVGZ/p1fXW6r+P2Usrq\nzv1fUXda09UdxcrO7XunuN+rsyDJB9phtFXUHUIZodag5dvzZepy/lkp5ZIZ9HtSbuy7P2jMrbHO\nGT52dmyHE29oy/YUHlquO09RszvfXbrzLqXcQx1bu3badNdpd+wMqztTA8foFBYDH22HbH/R+lv6\n+jxo3GxPPaLRv710nzvKleD94/y20vao7X54aDkdkOSSdmj3DuoHk+mW09A+lFLuBM6kfqo7foT+\nPlLsSv1w128X6v763s5jw7aJYcu8f9vq3h5lfNzcu9H6s67ruuuWzu1h+9NNknyqnZ5aBXwL2DqZ\n9lsW0+1PT6ZuX+eWUlbMoN+zYl0DwShfMelevbuYeqioZ5skm3TuP7ZveleZ4eO3Av/XV/+xA9pC\nTbtnA7uWUrYGPsVDr+8m4Ld6DVufu+cLf059bT39r3OYYXVn06uAF1E/FWwN7N7q9GrNdPn2HEs9\ntL5zklfMQj/n2q599weNuZt4+Ngd5Fhq0t+zLdu/Zs2x01+zO981xk6Szahja7odx3R159J11MPK\nvQ8D25RSNi+lXNppM2jc3EY9JdC/vXRD03RjbmSp1yx9HvhnYIdSyjbAV1nHcZ/kadTD1qcCH5+d\n3k5WkqdT36wvmmLyTUy9v55qPtMt8zX2p33zGWV8DOr/dHVn09uAxwNPb9te76vY67o/PQE4B3hu\nkj9c517O0Di+3/6mJLsm2ZZ6rvG0zrQAx7QLKvahXrh15oD5rKQeUhpJ+7RzFnB0S3NPph5uGmRz\nagL+TZK9ge4Fjp8HXpTkmakXCx7d99xTgSOSbJ9ke+ophuUjdnVYXZi9wbw59VDcHe1N5/2sOThn\ntHwBUn+PYClwEPX6hI+nc0HcmG2YZKPO36CreHdK8pYki5K8jHq+7itTtDsDeGsbu9tQr8AeZAvq\n9S93J9kVeHtn2iXAA0nelHpR5oHA3p3ppwIHJ3lqko2ob/KXlvo1rukMqzuXPgW8s21TJNkqyUtH\neWLbLs8A3pdk8ySLqYdJR91eZmrD9ndbKWV1kgOop/R6VgLbZcSLMaH+LgO1v/9IDQW7JJnJ1zjn\nlSRbJHkhdSwuL6X8pL9NKeU66iH03v76j6gfMNaYVft3umV+BnXMPzH1ws4jeOjri+syPqarO5s2\npx4xuKu9tx3dN31t9qcHAXtR96V/C5zcls/YrEsgGDXF/zvwdep59iup5/F7bgLuoH5KWk791HHF\ngPl8FHhZOxT0kSF96D72FupO8ybqBUD93wnttj0MeE+SO6kD9MEfYCml/LTN6/TW17uoh5bub03e\nS91YLgd+1G53X+ewPg6sO+Q1Dpo2rO3J1E92NwI/pl6w0nUisGc7DHzWkPkVqDsR6sVcbyql3FxK\nuYh6PvykIX2YS1+hHua+t/171IB2l1KT/W3Ae4CXlFJWtWnd1/uv1AuAeuvzP/rm0217DPAH1AsU\nz+m2LaX8BvgL6oVud1AD3zm0sVNKOZ8aIM+irpvHAd0jLcPW6cC6Izx3OgOfW0o5m/o1tdPa4dLL\nqReKDXtu97G3UtfRCuDb1N8BmO1x03uD+WWrd2Y7vfEK4IsPNirlZ9Q3whVt7D9mhHkfC1xbSvl0\nKeXX1ED8niR7TPO8+eactt+5jnpB5nHUgDPIK4FnUk8RvZu6/XeNuszPo148eyH14tTeqcbe/nSm\n42OkuiOYyf70I9SLHm+j7kvP7Zs+o/erJLtRTz0dVEr5VSnlVOA/qdeUjU3vIpy5mfmQH4xJ/UGQ\n5aWUYYfx56X2CXsV8NullIHn0aSppH4F9pOllP4dqrTeSfJE4L+pV96vnq695s68+Unc+S7JC9up\nh82ADwGXGwY0iiT7JtmpnTJYCvwucN6k+yVNSpI/T/39lm2AfwK+ZBiYvLkOBHN3+GH8DqSeLriB\n+jWZR8IFdJofnkA99XAH9XzoS0opK4c/RXpUO5R62vVKHvrdCU3YnJ4ykCRJjwyeMpAkScP/t8Mk\nHj7QrCqljPW/R3UMa7aNewyD41iza9AYnva/Px76f+HOkWU/nMC1Jb9/zPhrrkd22mkzVq48fCK1\nr169w9hrPm6Bp0QfnSa3nxj3vnjZwG/u6pFt8Bj2lIEkSTIQSJIkA4EkScJAIEmSMBBIkiQMBJIk\nCQOBJEnCQCBJkjAQSJIkDASSJAkDgSRJwkAgSZIwEEiSJAwEkiQJA4EkScJAIEmSMBBIkiQMBJIk\nCQOBJEnCQCBJkjAQSJIkDASSJAkDgSRJwkAgSZIwEEiSJAwEkiQJA4EkScJAIEmSgEXTNVh22epx\n9GNNB4+/JOw9iaLA9ydUd/2x5CU3j73migceM/aaSxYeNvaaGp9lHDnmis8bc72e8yZUVx4hkCRJ\nBgJJkmQgkCRJGAgkSRIGAkmShIFAkiRhIJAkSRgIJEkSBgJJkoSBQJIkYSCQJEkYCCRJEgYCSZKE\ngUCSJGEgkCRJGAgkSRIGAkmShIFAkiRhIJAkSRgIJEkSBgJJkoSBQJIkYSCQJEkYCCRJEgYCSZKE\ngUCSJGEgkCRJGAgkSRKwaNoWex0zhm70e8YEam4ygZoAO06g5i0TqDk55exlY695xIJjx15zxQPv\nHHtNgCULD5tI3fVPxlzvvDHX63GfOCkeIZAkSQYCSZJkIJAkSRgIJEkSBgJJkoSBQJIkYSCQJEkY\nCCRJEgYCSZKEgUCSJGEgkCRJGAgkSRIGAkmShIFAkiRhIJAkSRgIJEkSBgJJkoSBQJIkYSCQJEkY\nCCRJEgYCSZKEgUCSJGEgkCRJGAgkSRIGAkmShIFAkiRhIJAkSRgIJEkSBgJJkgQsmnQHpva9CdTc\naQI1AbaYQM17J1Bz0wnUbMr4S5664Pqx1zzl+7eOvSbAigdOmEjdJQsPm0hdzbVbJlBz4wnUBLhv\nQnWn5hECSZJkIJAkSQYCSZKEgUCSJGEgkCRJGAgkSRIGAkmShIFAkiRhIJAkSRgIJEkSBgJJkoSB\nQJIkYSCQJEkYCCRJEgYCSZKEgUCSJGEgkCRJGAgkSRIGAkmShIFAkiRhIJAkSRgIJEkSBgJJkoSB\nQJIkYSCQJEkYCCRJEgYCSZKEgUCSJAGLJt2B+WPlhOreO4Ga206g5iYTqDk5pWTsNRc+/cix1wQ4\nMssmUvfq1SeMvebj/Aj1KHXfhOrOr7dgh7ckSTIQSJIkA4EkScJAIEmSMBBIkiQMBJIkCQOBJEnC\nQCBJkjAQSJIkDASSJAkDgSRJwkAgSZIwEEiSJAwEkiQJA4EkScJAIEmSMBBIkiQMBJIkCQOBJEnC\nQCBJkjAQSJIkDASSJAkDgSRJwkAgSZIwEEiSJAwEkiQJA4EkScJAIEmSgEXTNdhrr53H0Y/12GYT\nqLnl2Ctut91GfOMbYy8LrD9jOJSJ1N05e02k7oZsM4Gq50+gZrW+jOP1y8KxV7zsssHTUsrgnUiS\nyexh9KhVSsk46zmGNdvGPYbBcazZNWgMDw0EkiRp/eA1BJIkyUAgSZIMBJIkCQOBJEnCQCBJkoD/\nB2tKLctJabkBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x79bd150>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Here, given a symmetrix matrix M, we will simultaneously build\n",
    "its tridiagonal Lanczos factorization, Hpp as seen previously,\n",
    "And the Choleski factorization of Hpp as LDLt\n",
    "\"\"\"\n",
    "\n",
    "#Defines how to solve L D Lt x = y\n",
    "def solveCholeskiForm(L,D,y):\n",
    "    #Solve LDz = y by forward substitution + trivial inversion of D\n",
    "    def forwardSubst( L, x ):\n",
    "        out = np.zeros_like(x)\n",
    "        out[0] = x[0]/L[0,0]\n",
    "        for i in range(1,L.shape[0]):\n",
    "            out[i] = x[i]/L[i,i]-np.dot(out[:i],L[i,:i])\n",
    "        return out\n",
    "    z = forwardSubst(L,y)/np.diagonal(D)\n",
    "    #Check that LDz = y\n",
    "    assert( np.allclose(np.dot(L,np.dot(D,z)),y) )\n",
    "    #Solve Ltx = z by backward substitution\n",
    "    def backwardSubst( L, x ):\n",
    "        out = np.zeros_like(x)\n",
    "        out[-1] = x[-1]/L[-1,-1]\n",
    "        for i in range(L.shape[0]-1)[::-1]:\n",
    "            out[i] = x[i]/L[i,i]-np.dot(out[i:],L[i,i:])\n",
    "        return out\n",
    "    x = backwardSubst(L.T,z)\n",
    "    #Check that Lt Sp = z\n",
    "    assert( np.allclose(np.dot(L.T,x),z) )\n",
    "    return x\n",
    "\n",
    "size = 5\n",
    "M = np.random.rand(size,size)\n",
    "M = np.dot(M,M.T) #Symmetric\n",
    "y = np.random.rand(size)\n",
    "error = []\n",
    "\n",
    "#Allocate matrices and vectors\n",
    "Hpp = np.zeros((size,size))\n",
    "Vp = np.zeros((size,size))\n",
    "yp = np.zeros(size)\n",
    "L = np.identity(size)\n",
    "D = np.zeros((size,size))\n",
    "\n",
    "#Initialize\n",
    "Vp[:,0] = y/np.linalg.norm(y)\n",
    "yp[0] = np.dot(Vp[:,0],y)\n",
    "\n",
    "for i in np.arange(1,size+1):\n",
    "\n",
    "    #Compute next Vp\n",
    "    v = np.dot(M,Vp[:,i-1])\n",
    "    #orthogonalize\n",
    "    minId = np.maximum(0,i-2)\n",
    "    orth = np.dot(Vp[:,minId:i].T,v)\n",
    "    Hpp[minId:i,i-1] = orth\n",
    "\n",
    "    if( i < size ):\n",
    "        v = v - np.dot(Vp[:,minId:i],orth)\n",
    "        #Update Hpp\n",
    "        Hpp[i,i-1] = np.linalg.norm(v)\n",
    "        #Normalize vector and update V\n",
    "        Vp[:,i] = v / Hpp[i,i-1]\n",
    "        #Update Yp\n",
    "        yp[i] = np.dot(Vp[:,i],y)\n",
    "\n",
    "    #Update the LDL^t factorization\n",
    "    if( i > 1 ):\n",
    "        L[i-1,i-2] = Hpp[i-1,i-2]/D[i-2,i-2]\n",
    "        D[i-1,i-1] = Hpp[i-1,i-1]-L[i-1,i-2]*Hpp[i-1,i-2]\n",
    "    else :\n",
    "        D[i-1,i-1] = Hpp[i-1,i-1]\n",
    "\n",
    "#check orthogonality of Vp\n",
    "assert( np.allclose(np.identity(size),np.dot(Vp.T,Vp)) )\n",
    "#Check the LDLt Choleski factorization of Hpp\n",
    "assert( np.allclose(Hpp,np.dot(L,np.dot(D,L.T))) )\n",
    "#Check the equivalence between M and VptHppVp\n",
    "assert( np.allclose(Hpp,np.dot(Vp.T,np.dot(M,Vp))) )\n",
    "\n",
    "#Solve the problem in the Krylov space\n",
    "Sp=solveCholeskiForm(L,D,yp)\n",
    "#Now in \"normal space\"\n",
    "x = np.dot(Vp,Sp)\n",
    "#current estimate\n",
    "assert(np.allclose(np.dot(M,x),y))\n",
    "\n",
    "#Show matrix Hpp, L and D\n",
    "fig,ax = plt.subplots(1,3,figsize=(9,3))\n",
    "ax[0].set_title(\"Hpp tridiagonal matrix\")\n",
    "ax[0].imshow(Hpp,interpolation='none')\n",
    "ax[1].set_title(\"L bidiagonal inferior matrix\")\n",
    "ax[1].imshow(L,interpolation='none')\n",
    "ax[2].set_title(\"D diagonal matrix\")\n",
    "ax[2].imshow(D,interpolation='none')\n",
    "map(lambda a : a.get_xaxis().set_visible(False),ax)\n",
    "map(lambda a : a.get_yaxis().set_visible(False),ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lanczos iterations in practice\n",
    "  \n",
    "We now have everything we need to design an algorithm that iteratively build:\n",
    "\n",
    "\n",
    "- The Arnoldi orthogonalization of the Krylob basis $V_p$\n",
    "- The tridiagonal $H_{pp}$ matrix\n",
    "- The triangular matrix $L_p$, part of the $L_pD_pL_p^\\intercal$ decomposition of $H_{pp}$\n",
    "- The triangular matrix $D_p$, part of the $L_pD_pL_p^\\intercal$ decomposition of $H_{pp}$\n",
    "- The projection of $\\vec{y}$ over the basis $V_p$ : $\\vec{y_p}$\n",
    "\n",
    "\n",
    "Which can be implemented as follows:\n",
    "\n",
    "\\begin{algorithm}\n",
    "  \\caption{Solve $M \\vec{x} - \\vec{y} = 0$}\n",
    "    \\begin{algorithmic}\n",
    "      \\REQUIRE $M \\in S^n_{++}$ the positive definite cone\n",
    "      \\STATE \\textbf{Initialization}\n",
    "      \\STATE $\\vec{v_0} = \\frac{\\vec{y}}{\\| \\vec{y} \\|}$\n",
    "      \\STATE $vec{y_p} = \\vec{0}$\n",
    "      \\STATE $vec{L_p} = O_{n\\times n}$\n",
    "      \\STATE $vec{D_p} = O_{n\\times n}$\n",
    "      \\STATE $j = 1$\n",
    "      \\STATE $\\vec{w} = M \\vec{v_{0}}$\n",
    "      \\STATE \\textbf{Iterations}\n",
    "      \\WHILE{$\\vec{w} \\neq \\vec{0}$ and $j \\leq n$}\n",
    "      \\STATE $\\vec{w} = M \\vec{v_{j-1}}$\n",
    "      \\IF{$j > 1$}\n",
    "        \\STATE $H_{j-2j-1} = H_{j-1j-2}$\n",
    "        \\STATE $\\vec{w} = \\vec{w} - H_{j-2j} \\vec{v_{j-2}}$\n",
    "      \\ENDIF\n",
    "      \\STATE $H_{j-1j-1} = \\langle \\vec{w},\\vec{v_{j-1}} \\rangle $\n",
    "      \\STATE $\\vec{w} = \\vec{w} - H_{j-1j-1} \\vec{v_{j-1}}$\n",
    "      \\IF{$\\vec{w} \\neq \\vec{0}$}\n",
    "        \\STATE $H_{jj-1} = \\| \\vec{w} \\|$\n",
    "        \\STATE $\\vec{v_j} = \\frac{\\vec{w}}{H_{jj-1}}$\n",
    "        \\STATE $\\vec{y_p}[j] = \\langle \\vec{v_j},\\vec{y} \\rangle $\n",
    "        \\IF{$j > 1$}\n",
    "          \\STATE $L_{j-1j-2} = \\frac{H_{j-1j-2}}{D_{j-2j-2}}$\n",
    "          \\STATE $D_{j-1j-1} = H_{j-1j-1}-L_{j-1j-2}*H_{j-1,j-2}$\n",
    "        \\ELSE\n",
    "          \\STATE $D_{j-1j-1} = H_{j-1j-1}$\n",
    "        \\ENDIF\n",
    "      \\ENDIF\n",
    "        \\STATE $j=j+1$\n",
    "       \\ENDWHILE\n",
    "      \\STATE \\textbf{Last Part}\n",
    "      \\STATE Solve diagonalized system: $L_pD_pL_p^\\intercal \\vec{s_p} = \\vec{y_p}$\n",
    "      \\STATE Solution to initial problem is $\\vec{x_p} = V_p^\\intercal \\vec{s_p}$\n",
    "    \\end{algorithmic}\n",
    "\\end{algorithm}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x8e33e50>]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAEhCAYAAADS7c8nAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FNX6wPHvm9CrYkEFQUEFRWlSf6gEAQEL9oIdr4oU\nEZUrimVZrw29CnhBEXu59msBlS6xKyBNKQqCSFGw0Xvy/v44k7BsdpNNspvZTd7P8+zDzuyZmTeT\nZd6cM2fOEVXFGGOM8VOa3wEYY4wxloyMMcb4zpKRMcYY31kyMsYY4ztLRsYYY3xnycgYY4zvLBmF\nEJHNfsdQGCLypIi0D1sXEJFbvPeVRGSKiNxTyP0mxXkQkXEi0rgI2zUTkR4hy2eJyG1xiKe+iHxX\n3P0U4ngzRKRlnPdZU0T6hix3FJEJxdhfsbY3Joclo32l2kNXbYGvI30gIuWBt4FZqnpvhM8ln/2W\n2HkQkajfQVW9XlWXFGG3zYHTQ/YzQVUfLkp8kcKK0378sj/QL2xdcX+mpDwnIpLudwwmdpaMCiAi\nZ4rI1yLyrVfLOMhbHxCRZ72/XpeJyI0h21wpIvNFZK6IvOitmysic7x/t4nIySKyv4i865X9UkRO\n8Mp2DCn/rYhUjRBXY+BHjfzUcnngDe/zO73y9UVkiYi86P11f5eIjAjZ37Ui8mjOYsj6R0TkOy/G\nC711r4XVPJ4XkfPC4usoIp+IyAfecZ8I+WyziPxbROYC7USks/ezzheRZ7xEuk/NQES6eudotoi8\nISJVvPWtReQLEZnn/Z5qAPcCF3n7vFBErhKR/4Sch+le+akiUjfkZxjl7WtZ+M8Tem5F5BURWSQi\nb4pIJW/7u0XkGxFZICJjQ37WgSKy0Dveq966Kt53J+d71dNbX8k7twtF5B2gUqQARGSFiDzgfUdm\niUhLEZksIktFpE9IucEiMtM7dsBb/SDQwDs3w7111UXkLRFZLCIvh2wf7ffS3Ss7G4h4nkQkLeS7\nM09E+hewzxUiMsw7H/NF5BhxVni/05z9/igiB4nIgSLytnfOvxGvhUDc/8uXRORz4CURqez9nr4X\nkXe8c17QdypPLN76qiLynPc7nici5+a3H1NIqmov7wVsirCuZsj7fwCPeO8DwOdAOeAA4A8gHWgC\nLAH298rtF7a/M4FPvLKPA3d76zsBc73344H23vsqQFqEuG4Gro6wPgD8CbwWtr4+sAdo7S1XBZYC\n6d7yF8BxoecBOB+Y7L0/GFgJ1AbOAV7w1pf31lcMO15HYJt3XAGmAOd5n2UD53vvKwK/AA295ReB\ngd77GUBL7/x+AlT21t8G3OUd+yegpbe+mnderwIeD4kld9k7t5d773sD73rvnwfe8N4fCyyNcG7r\ne7G385afBW4J/z0DLwFneO/XAOW99zW8f+8HLs35fgE/AJW93+kz3voTgN05P1tYHCuA6733jwHz\nve/JgcBv3vquwFPeewEmACd5P8OCsN/T38ChXrkvgf+L9nsJWd/AW/8GMD5CjDcAbwKSc34K+F2v\nAPp57/sC47z3I4CrvPdtgCne+/8C/+e9PxxYFPL9nwVU8JZvBZ703jcBdpHPd6qAWB4CHgu9NkTZ\nz91+X8tS8WU1o4Id7v3VuQAYjPtC5/hQVfeo6p/AOtyFuhPwlqr+DaCqG3IKi8jRwCPAhaqahbs4\nvOyVmwHUEpFquMQwQlxta39VzY4QVzdgUpSYPwP+zzteqJWqOss73lbgY+BMEWkElFPVRWHlOwCv\neeXXA5lAa2AikOH9VdsD+FRVd0aIY6aqrlT3v/Q17+cFyALe8d43Apar6k/e8ovAKWH7aQccB3wh\nrjZ1Je6i2ghYq6pzvBi3eOc1P+1zfibcue8Q8tl73n4W45JvJL+oak7T6CshP1Nn76/uBbjvQM73\nZD7wqohc5v3cAKcBt3s/SyZQAajn/dyveDF8520bTc59mu+Ar1V1m6r+AezwahKnAV1FZA4wB3eu\nwr8POWaq6q/e72kecATRfy+NvfXLQ85BJF1wyVC9n2dDPvvM8a7377deDOAS2iXe+0twyS9n/6O9\nczgeqBZSIxmvqru89ycBr3sxLAQWeOsjfafqFRBLF2BMTgFV3RjDfkyMyvkdQAr4D/BvVf1QRDri\n/vLKEXoBzmLv+cxzP0ZcU9sbwD+8CzvkbWsXAFUdLiIfAGfgvuSnqeqPIfuqjKux/RYl5k9x/9En\nikgHVV3nrd8aVu5ZYCiuJvd8lH1Fim+niGQC3YGL2XtxDxf+8+Usb8+5SIXut4DjTlHVy/ZZKXJ8\nDNsWFFOo0N9ntP3m+ZlEpCLuItVSVdd6TWI5TWxn4C64PYE7xTXFCq5muDR0R5L3Nl5+P1tOrNlh\ncWfjvocCPKiqT4cdo34++4ICvscxxFWQWH6m3BhU9SsRaSgiB+Jq5Dn3PwVoq6q799m5O4fh3/NI\nx4/4ncovlnz2l99+TIysZrSvSP9RagBrvfdXxbDtx8AFIlILQET299Y/Dzynql+GbPMZcLlXLgP4\nXVW3iEgDVV2o7qb7LNxfo6E64ZqwolLVd4F/A5ND2twlrMxMXBNHL/ZNKDnlPgMu9tr/DwJOBmZ6\nn72Ja+Y6ieg1tDbi7tGk4ZLWZxHi+AGoLyINvOUrcLWFUF8DHUSkIeTeczna2/YQETnRW19N3E3r\nzbjfWyRfej8vuHP/WZRy0S6a9UWkrff+UlxTbSVckvrTq9leEFK+nqp+AtzuxVQVmIxr8sKLu7n3\n9lPgMm/d8UDTKDHkJyfuycA13h9BiMhh3gV9M1A9hv1E+70s8dYf6a3vFWFbgKlAH+/3kfP/IJbf\ndSTv4pojF4W0NEwBbsopICLNomz7Be67h4gcBxzvrY/2ncrPVKB/yDH3K+J+TASWjPZVWUR+EZFV\n3r+DgGHA2yIyC/g9n21zmiMW4e4JfOJV2x8VkXq4G73XyN6OCS2BIHCiiMwHHsBV8QEGiXfjF9fG\nPTHsWD2IngD2BqQ6FtccNp69F8xwbwJfeE0O4T/Lu7hmjfnANOCfIbW6Kbi/+Keq6p4oIcwGRgML\ngZ9U9b3Q/XvH2IlLam975yELeGrfH0P/AK4GXvPKfAk08v4qvhjXXDPPi6kiLlEf553nC8NiGgj0\n9spfxt4LWrRaXLglQH8RWYS7D/Kkd+6e9n7OiXgJW0TKAa94MX8LjFLVTcC/cB0hFojrTJLz1/6T\nuOamhbjv3ewoMeRXu8v53U0FXgW+8poO3wKqq+pfwJfesYfns33E34u3/nrgI3EdGNZF2AfAM8Aq\nYIH3/6BXAb/r/H6mN3G/q9dD1t0EtPI6GHwP9Im4JTwBHOiVuRf3O9oY7TtVQCz34ZrSv/N+powC\n9mMKIefmokkh3kWgbQz3R2LZ1wTcTdl8a1pF2G9H4FZV7VmMfSwAzlLVlfGLzJQlXq28vNe03ABX\nu2mUzx9Qxid2zygFqWqr4u5DRGri/oKfG+9EFA8iMgWYb4nIFFMVYIbX2QagryWi5GQ1I2OMMb6z\ne0bGGGN8Z8nIGGOM7ywZGWOM8Z0lI2OMMb6zZGSMMcZ3loyMMcb4zpKRMcYY3yX0oVcRKuLG26rg\nvd5XZWiEco/jhrjZClytyrxExmWMMWWdBKU7MBJXKXlWAzo87PP9gOeAhsB24BoN5BnZP24SWjNS\nZSfQSZUWuEEfTxXZZ8h+ROgBNFTlaNz4UmPz7skYY0y8SFDScONGdsNNd9JLghI+IPNQYK4GtBne\nnGCJjCnhzXSqbPPeVvSO93dYkbNxk5GhyjdATRFqJzouY4wpw9oASzWgKzWgu3GD0J4dVuY43CwE\naEB/AI6QoJvpOhESnoxESBNhLvAbkKlKeDWvDm503xxrvHXGGGMSI/y6u5q81935eNPKS1Da4CYN\nrJuogEqiZpTtNdPVBU4RoWOij2mMMabYHgL2l6DMwc3jNJe9sxXHXYmN2q3KJhE+BFrh5ozPsQY3\nwVuOut66fYiIjehqjDFFoKrhk0WuYd/p0fNcdzWgm4FrcpYlKCuA5SRIQmtGIhwoQk3vfWWgK+Tp\nKTceb1I5EdoBG1QjT9ilqkn/CgQCvsdgcVqcqRqjxRn/VxSzgKMkKPUlKBWAS7xr8d7rd1BqStBN\nvSFBuQ74RAO6pUjJIAaJrhkdCrwoguAS38uqTBehDy63jFPlIxFOF2EZrmt37wTHZIwxZZoGNEuC\nMgA3O3JO1+7FEhR3bQ7oOOBY4EUJSjZuhtx/JDKmhCYjVb4DWkZY/1TY8oBExmGMMWZfGtBJhE2R\nrgF9KuT91+GfJ5KNwBBnGRkZfocQE4szvlIhzlSIESzOsiplZnoVEU2VWI0xJlmICJq3A0PSsZqR\nMcYY31kyMsYY4ztLRsYYY3xnycgYY4zvLBkZY4zxnSUjY4wxvrNkZIwxxneWjIwxxvjOkpExxhjf\nWTIyxhjju5RKRjNWzPA7BGOMMQmQUslo0ORBZGUnbKJBY4wxPkmpZLRfpf14Zs4zfodhjDEmzlJq\n1O45a+fQ4789WDJgCftV2s/vkIwxJumlyqjdKZWMVJXrxl9HjYo1eLTbo36HZIwxSS9aMpKgdAdG\nsnem1+Fhnx8AvIKbsTsdeFQD+kKi4kypZjqA+069jxfnv8iPf/7odyjGGJOSJChpwGigG9AE6CVB\naRxWbAAwTwPaHOgEPCpBSdjs4CmXjGpXq82QDkO4dcqtfodijDGpqg2wVAO6UgO6G3gdODuszG9A\nde99deBPDeieRAWUcskIYGDbgSz5YwlTfpridyjGGJOK6gCrQpZXe+tCPQ00kaCsBeYDNyUyoIRV\nuRKpYrmK/Lvrv7l58s3Mv2E+5dJS8scwxpi4y8zMJDMzMx67ugOYrwHtJEFpCEyVoDTVgG6Jx87D\npWTNCKBno54cWu1Qxs4e63coxhiTNDIyMhg2bFjuK4o1QL2Q5breulAdgLcANKA/ASuA8PtKcZOy\nyUhEGNFtBPd+ci9/bf/L73CMMSaVzAKOkqDUl6BUAC4BxoeVWQx0AZCg1AaOAZYnKqCU69odrt+H\n/SiXVo7HezzuQ1TGGJPcCujaPYq9XbsfkqD0AVQDOk6CciDwPK4GJcCDGtDXEhZnqiejP7b9wbFj\njuWTqz/huIOO8yEyY4xJXqny0GvKNtPlOLDKgdx58p3cMvkWUiWxGmOM2VfKJyOA/q378/OGn/lo\n6Ud+h2KMMaYISkUyKp9ense6PcYtU25hV9Yuv8MxxhhTSAlNRiLUFeFjERaK8J0IAyOU6SjCBhHm\neK+7inKs048+nQb7N2DMzDHFD9wYY0yJSmgHBhEOAQ5RZZ4I1YBvgbNVWRJSpiNwqyo9899X5A4M\noRb/vphTXjiFRf0WcVDVg+LwExhjTGqzDgyAKr+pMs97vwXXbz18yAlw3QaL7diDjuXS4y/lnhn3\nxGN3xhhjSkiJ3TMS4QigOfBNhI/bizBPhA9FKFb/7EBGgHeWvMOCdQuKsxtjjDElqESSkddE9zZw\nk1dDCvUtUE+V5rghzd+Ltp+HHy74WLUq1+KeU+7h5sk3W1dvY4xJEQl/6FWEcsAHwERVRsVQfgVw\noip/7btedP/9A7RuDe3bu/GXMjIyIu5jT/Yemo9tzn2n3sc5jc+Jw09hjDGpye4Z7fUcsChaIhKh\ndsj7NoCEJ6Ic8+cPY9myYey//7CoiQigXFo5RnQbweApg9m5Z2fxojfGGJNwie5N1wH4FPgOUO81\nFKgPqCrjROgP9AV2A9uBm1Xz3lfK6U23ciVkZMCtt8KAAfkfv+drPTmp3knc1uG2eP5YxhiTMlKl\nZpSSY9P9/LNLSEOGQN++0bdZ+udS2j/bnoX9FlK7Wu3oBY0xppSyZBRn4c8ZLV/uEtLdd8N110Xf\nbvCUwWzYsYFnej6T+CCNMSbJWDKKs0gPvS5bBp06QTAI11wTebuNOzbSaHQjJl42kRaHtiiBSI0x\nJnmkSjJK6bHpjjoKpk+He+6BF1+MXKZmpZrc2+lebpp0k3X1NsaYJJXSyQjgmGNg2jQYOhReeSVy\nmX+0+Aebdm7i7UVvl2xwxhhjYpLSzXShFi2CLl3g0UehV6+8n2f+nMnV713N4v6LqVy+cgIjNcaY\n5FHATK8j2TvT6/CwzwcDl+F6QZcHjgUO1IBuSEicpSUZAXz/PXTtCqNGwUUX5f38/DfPp+UhLbnz\nlDsTFKUxxiSXSMlIgpIG/Ah0BtYCs4BLNKBLIuwCCcqZwCANaJdExZnyzXShjj8eJk+GgQPhf//L\n+/kjXR9hxNcjWLt5bckHZ4wxyaMNsFQDulIDuht4HTg7n/K9gNcSGVCpSkYATZvCpEnQvz+8++6+\nnzXYvwHXtbyOO6bf4U9wxhiTHOoAq0KWVxN5RgUkKJWB7kCEP/Hjp1wid+6X5s3ho4+gRw9IT4ee\nITMlDT15KI1GN2Lmmpm0qdPGvyCNMSYBMjMzyczMjOcuzwI+T9S9ohylMhkBtGwJH3wAZ5wBzz/v\n/gWoXrE69596P4MmDeKLa75AJOm73xtjTMzCB5EOBoORiq0B6oUs1/XWRXIJCW6ig1LYTBeqdWuY\nMAF693ZNdzmuan4Vu7J28dr3CT+/xhiTjGYBR0lQ6ktQKuASzvjwQhKUmkBH4P1EB1SqkxFA27bw\n3ntw5ZUwdapblyZpjOw+ktun3c7WXVv9DdAYY0qYBjQLGABMARYCr2tAF0tQ+khQrg8peg4wWQO6\nPdExlaqu3fn5/HM491x4/XXo3Nmtu+TtS2h8YGOGZQyLT5DGGJNkUmU4oDKTjAA+/RTOPx/eessN\nsvrLxl9o8VQL5vWZx+E1D49PoMYYk0RSJRmV+ma6UKecAm++CRde6BJTvZr16N+6P0OmDfE7NGOM\nKdPKVDICN8r366+7GtIXX8CQDkP47JfP+HLVl36HZowxZVaZaqYLNWUKXH45vP8+/FT1FUZ9M4pv\nrv2GNClz+dkYU4pZM12SO+00N+3E2WfDUdsvJV3SeXn+y36HZYwxZVKZTUbgRmh47jk4u2caN9Qf\nxdCPh7Jl1xa/wzLGmDKnTCcjgDPPhKefhiGXt6V5jVN58LMH/Q7JGGPKnDJ7zyjcu+/CdbeuYc+1\nTZnbdzZH7n9kwo5ljDElxe4ZpZhzz4WxD9dhzxeDuP7tf/odjjHGlCmWjEJccAE8eflgPv5hFs/P\n+MTvcIwxpsywZBTmil6V6Xf0w1z/zk18tzDL73CMMaZMsGQUwePXX0SDutU5eeBzLIk4Ca8xxph4\nsmQUgYjw6hUj0Yy7ObXHRpYu9TsiY4wp3SwZRXHiYSdyftPTaTrgPjp3hmXL/I7IGGNKL+vanY/f\ntvzG8U8cz03VvuKZ4UczYwY0aFCiIRhjTLFY1+5S4JBqhzD4/wYzu9ZghgyBU0+Fn3/2OypjjCl9\nElozEqEu8BJQG8gGnlbl8QjlHgd6AFuBq1WZl7dMydeMAHbs2cFxY45j3FnjWPxhF0aMgMxMqFev\nwE2NMcZ30WpGEpTuwEhcpeRZDejwCGUygBFAeeB3DWinRMWZ6JrRHuAWVZoA7YH+IjQOLSBCD6Ch\nKkcDfYCxCY6pUCqVq8S/T/s3gyYNom//PQwc6KahWLXK78iMMaZoJChpwGigG9AE6CVBaRxWpiYw\nBjhTA3o8cGEiY0poMlLlt5xajipbgMVAnbBiZ+NqT6jyDVBThNqJjKuwzm18LgdVPYhx345j0CDo\n18812a1Z43dkxhhTJG2ApRrQlRrQ3cDruGtxqEuB/2lA1wBoQP9IZEAlds9IhCOA5sA3YR/VAULr\nGWvIm7B8JSKM7DaS4CdB/t7+N7feCtde6xLSr7/6HZ0xxhRa+HV3NXmvu8cAtSQoMyQosyQoVyQy\noHKJ3HkOEaoBbwM3eTWkIhk2bFju+4yMDDIyMoodW6yaHdKMcxqdQ/CTICO7j2TIEMjKcglpxgw4\n5JASC8UYY6LKzMwkMzMzHrsqB7QETgWqAl9JUL7SgCbkQZeEJyMRyuES0cuqvB+hyBrg8JDlut66\nPEKTkR/+deq/OG7McdzQ6gYaH9iYoUNhzx7o3NklpIMP9jU8Y4zJ84d6MBiMVGwNENoNK9J1dzXw\nhwZ0B7BDgvIp0AxISDIqiWa654BFqoyK8vl44EoAEdoBG1RZVwJxFdrBVQ/mjpPu4JbJt+Suu+ce\nN8Bq587w++8+BmeMMbGbBRwlQakvQakAXIK7Fod6HzhJgpIuQakCtMXd90+IRHft7gB8CnwHqPca\nCtQHVJVxXrnRQHdc1+7eqszJuy9/unaH25W1i+OfOJ5R3UfR4+geAKjCXXfBBx/Axx/DAQf4HKQx\nxngK6No9ir1dux+SoPQBVAM6ziszGOgNZAFPa0D/k++xgtIUOIKQVjcN6DsxxZkMF/hYJEsyApjw\nwwRum3YbC25YQPn08oBLSLffDlOmwPTpUKuWz0EaYwwlNwKDBOU5oCmwEPdcKbjEdk1M2yfLBb4g\nyZSMVJVur3TjjKPP4KZ2N4Wsh3/+090/mjYN9t/fxyCNMYYSTUaLNKDHFXV7Gw6oCESEEd1GcN9n\n9/HHtj9C1sMjj8DJJ0O3brBhg49BGmNMyZopQSlyMrKaUTEM+GgAqsqYM8bss14VBg6EWbNcs12N\nGj4FaIwp80qwZpSB6wTxK7ATEFwzXdOYtk+2C3w0yZiM/tz2J8eOOZaPr/qY4w8+fp/PVKF/f5g/\nHyZNgurVfQrSGFOmlWAyWgbcguuwlnPPCA3oypi2T7YLfDTJmIwAHv/mcSb8OIEpl09BZN/fd3Y2\n9O0LixbBxIlQrZpPQRpjyqwSTEZfaUDbF3n7ZLzAR5KsyWh31m6ajW3GQ10eomejnnk+z86G665z\nk/N99BFUrepDkMaYMqsEk9ETwH7ABFwzHRB7127rwFBM5dPL81i3x7h1yq3s3LMzz+dpafD003Dk\nkXDWWbBtmw9BGmNM4lXGJaHTgLO815mxbhxzzUiEGoQ+yKT8VagwiylZa0Y5znj1DDod0YnB/zc4\n4udZWXD11fDbbzB+PFSuXLLxGWPKppKoGUlQ0oGBGtARRd1HgTUjEfqI8BuwAPjWe80u6gFLq8dO\ne4yHPn+I9VvXR/w8PR1eeAEOOgjOOQd27CjZ+IwxJlE0oFlAr+Lso8CakQhLgfaqJHQui4Ike80I\n4OZJN7N191bGnTUuapk9e+Cyy2DzZnj3XahYsQQDNMaUOSV4zyhnRtg3cEO7AaABzTO8W8TtY0hG\nk4FzVfH1bkcqJKO/t/9N4zGNmXz5ZJof0jxqud27oVcv2LkT3n7bEpIxJnFKMBnNiLBaNaCnxrR9\nDMmoJfA88DWhPSSUgYWIs9hSIRkBPDnrSd5Y+AYzrpqRp6t3qN274eKLXW+7N9+EChVKMEhjTJlR\nUsmouGJJRrOAzwh/kEl5MbGhhceRGsloT/YeWjzVgmEdh3H+cefnW3bXLjf9RLly8MYbUL58CQVp\njCkzSrBmVBMIAKd4qz4B7tWAboxp+xiS0VxVWhQryjhIlWQEMH35dK6bcB2L+i+iUrlK+ZbduRPO\nP9/1rnvtNZeYjDEmXkowGf0P+B5yKypXAM00oOfFtH0MyegB4GfCH2Syrt35Ouf1c2hbpy13nHxH\ngWV37IBzz4X99oOXX7aEZIyJnxJMRvM0oM0LWhd1+xiS0YoIq1WVBrGHWXyployW/bWMts+05fu+\n33No9UMLLL9jB/Ts6bp+v/SS6wpujDHFVcDkeiPZO7ne8LDPO+Jme13urXpHA3pf1OME5SvgnxrQ\nz73lDsC/Yx0iKN9kJEIarlv3F7HsLJFSLRkB3Db1Nn7f9jvPn/18TOW3b4czz4Q6deD55y0hGWOK\nL1IykqCkAT8CnYG1uGnIL9GALgkp0xG4VQOad5yzSMcJSjPgJaAmbsTuv4CrNaDzY9k+34deVckG\nRseyI5PXXafcxaRlk5i9NrZnhCtXhgkTYNUquPZa19POGGMSoA2wVAO6UgO6G3gdODtCuZib9zSg\n8zWgzXCzvZ6gAW0RayKC2Mammy7C+SKxB2WcGhVr8K9O/2LQpEHEWqurUgU++AB++gn69LGEZIxJ\niDrAqpDl1d66cO0lKPMkKB8WNHGeBKWiBOVSYAAwSIJyjwTlnlgDiiUZ9QHeAnaJsEmEzSJsivUA\nZV3v5r3Zunsrbyx8I+ZtqlZ1I3wvXgz9+llCMsb44lugntcBYTTwXgHl38fVrvbgRmDIecWkwH5b\nqti0cMWQnpbOqO6juOLdK+jZqCdVyleJabtq1dwcSN26wY03wujRblpzY4zJT2ZmJpmZmQUVWwPU\nC1mu663LpQHdEvJ+ogTlCQlKLQ1otJ7UdTWg3YsQMhDjqN0i9GTvg0yZqnxQ1AMWVSp2YAh14VsX\ncsLBJ3BPx5hrrQBs3AinnQZt28KoUZaQjDGFE6UDQzrwA64Dw6/ATKCXBnRxSJnaGtB13vs2wJsa\n0COiHico44D/aEC/K1KcMXTtfghoDfzXW9ULmK1KwQ/QxFGqJ6OfN/zMieNOZP4N86lbo26htt2w\nAbp2hZNOgsces4RkjIldAV27R7G3a/dDEpQ+uPHkxklQ+gN9gd3AduBmDeg3UY8TlEXAUcAK3DOp\n4u2raUxxxpCMFgDNvZ51iJAOzFUlpgPES6onI4A7p9/Jyo0reeW8Vwq97d9/Q+fO7vXww5aQjDGx\nKcGHXutHWq8BXRnT9jEmo4ycERdEqIVrqrNkVEhbdm2h0ehG/O+i/9GubrtCb//nny4Z9egBDzxg\nCckYU7BUGSg1lt50DwJzRXhBhBdxPSzuT2xYpVO1CtV44NQHuGnSTWRr4bvIHXAATJsGH34Id98N\nKZ6bjTEmV6wdGA7F3TcCmKnKbwmNKmIMqV8zAsjWbNo9044b29zIFc2uKNI+fv8dOnVyI34PGxbf\n+IwxpUuq1IxiTUZ1gPqEdAVX5dMExhUhhtKRjAC+XPUlF711EUsGLKFahWpF2se6dS4h9erlaknG\nGBNJqUlGIgwHLgYWsnc+I1WlwPGKRHgWOBNYF+kekwh5B+JTIg7EV5qSEcCl/7uUhvs35F+n/qvI\n+/jtN8gZvyq6AAAgAElEQVTIgKuugjtKtG+jMSZVJDoZSVA2A5Euzjm96WrEtJ8YktEPQFPVvdNH\nxBykcBKwBXgpn2R0a2yJrXQlo1UbV9H8qebMuX4O9feL2AklJmvXuoR07bVw223xi88YUzqkSs0o\nlplzlgPlofDJSJXPRSjoSpv0JykRDq95ODe2uZHbpt3GGxfEPlRQuMMOgxkzoGNHNw/SLbfEMUhj\njCkkCcrBQO6sohrQX2LZLpZktA2YJ8J09p1cb2Bhg4yivQjzcENR/FOVRXHab9K7rcNtNB7dmM9W\nfsbJ9U8u8n7q1HEJKSPDTTtx003xi9EYY2IhQekJPAocBqzH9TNYDDSJZftYktF475UIbiA+ZZsI\nPXAD8R0TrfCwkK5jGRkZZGRkJCisklGlfBUe6vIQgyYPYtZ1s0iTWHraR3b44fDxxy4hlSsH/fvH\nL05jjInBv4B2wDQNaAsJSifg8lg3jqk3XXF4zXQTYnlI1ptV9sRIU5qXtntGOVSVDs914NqW13JN\ni2uKvb8VK1wvu9tvhxtuiEOAxpiUVoIjMMzWgLaSoMwHWmhAsyUoOXMcFSiWmlFxCVHuC4lQW5V1\n3vs2gERKRKWZiDCq+yjOfv1sLjjuAmpUjKnjSVRHHgnTp7uElJ4O110Xp0CNMSZ/GyQo1YBPgf9K\nUNZTiCkkit4uFAMRXgW+BI4R4RcReovQR4TrvSIXiPC9CHNxc7FfnMh4klXrOq3p2rArD3z2QFz2\n17ChS0j33gvPPReXXRpjTEHOxhtQFZgE/AScFevG+TbTeYOiDldlcDGDLLbS2kyXY+3mtZzw5AnM\nvHYmDWs1jMs+f/jBjWV3//3uWSRjTNmTKl27Y3nO6GtVCj+qZ5yV9mQEcP+n9/Ptr9/yzsXvxG2f\nS5bAqae6kb4vj/lWojGmtCjBe0ahD79WwD0StDXWh15juWc0V4TxuKnHc9v/VInfFdMAcEv7Wzh2\nzLHMWDGDTkd2iss+Gzd2g6t26eLuIfXqFZfdGmPMPjSgubOCS1AE12wXc0UmlprR85GOq0rxu34V\nQlmoGQG8tfAt7vvsPuZcP4f0tPS47ff7790EfaNGwUUXxW23xpgk52cznQRlrga0RUxlU+UCX1aS\nkarS8YWOXHbCZfRp1Seu+54/H7p1gzFj4Pzz47prY0ySKmCm15Hsnel1eMTtg9Ia1xHtYg1o1BYx\nCcp5IYtpQCugowa0fSxxFtibToS6Irwrwnrv9T8RCjdvtolZTlfvQGaADTs2xHXfzZrBxInQr5+b\nvnxnoQd4MsaUBhKUNGA00A03QkIvCUrjKOUeAibHsNuzQl7dgM24prrYYoqhmW4q8CrwsrfqcuAy\nVbrGepB4KCs1oxzXjr+WmhVr8mi3R+O+7yVLYPBg13R3771w2WXufpIxpvSJVDOSoLQDAhrQHt7y\n7bgRtoeHlbsJ2IWbz+6D/GpGxRVLB4aDVPe5b/SCCIMSFZBx7j/1fpo80YQ+rfpwzAFRR0gqksaN\n4YMP4LPP3EgNjzzipjE/80ybytyYMqIOsCpkeTXQJrSABOUw4BwNaCcJyj6fhZX7D5GnkABAAxrT\nOKaxPPT6pwiXi5DuvS4H/oxl56boalerzZAOQ7h1yq0JO8bJJ8Pnn7vnkO64Y++yMcbg7icNCVmO\n9qfqbNw4o5WAlsBS79Uc18U7JrHUjK4B/gOMwGW/L4HesR7AFN3AtgMZN2ccU36awmkNT0vIMUSg\nZ0844wx45RXXZNe0qaspnXBCQg5pjEmgzMxMMjMzCyq2BqgXslzXWxeqFfC61037QKCHBGW3BnSf\ngbM1oC8CSFD6AidpQPd4y2OBz2KNO5YRGAaqMiLWHSZKWbtnlOP9Je8z9OOhzOszj/Lp5RN+vJ07\n4ckn4cEHXc+7YNCNd2eMSU1R7hmlAz8AnYFfgZlALw3o4oj7CMrzwIQCetP9ALTXgP7lLe8PfK0B\nbRRLnPk206mSBdhjkj7q2agnh1Y7lLGzx5bI8SpWhEGDYOlSaNAAWrVy8yOtX18ihzfGlAANaBYw\nAJgCLARe14AulqD0kaBcH2mTGHb7EDBXgvKCBOVFYA4Q84CbsfSmG4Eb1uEN9h2BYU6sB4mHsloz\nAvhu3Xd0fqkzi/sv5oAqB5Tosdevd/eUXnkFBgyAW2+FGsUbWNwYU4JK8qFXCcohQFtv8RsN6G8x\nbxtDMpoRYbWqcmrsIRZfWU5GAP0+7Ee6pPOf0//jy/FXrIBAACZPdp0d+vZ1tShjTHJLdDKSoDTW\ngC6RoLSM9LkGNKaKS0H3jNKAC1R5s2hhxk9ZT0Z/bPuDY8ccS+ZVmTQ5OKZZfBNiwQIYOtQ9oxQM\nusFX7RklY5JXCSSjcRrQ6yUokSsuAY2p4hJLzWi2Kq2KEmQ8lfVkBDDy65FMXDaRSZdNQnx+ICjn\nGaWNG13Pu7POsmeUjElGqTKFRCzPGU0TYbAIh4tQK+eV8MhMHv1b92flhpV8uPRDv0PJfSbpwQdd\nTcmeUTKmbJOgXChBqe69v0uC8o4EJaZBUiG2ZHQx0B83ley33mt2UYI1xVM+vTyPdXuMWybfwq6s\nXX6Hg4irEc2fD9df75rszjzTNeUZY8qcuzWgmyUoJwFdgGeBmLsBF5iMVDkywqtBMQI2xXD60afT\nsFZDRs8c7XcoudLT4cor3cyyXbvCaafBFVe4Tg/GmDIjy/v3DGCcBvRDCjECQ9RkJMJtIe8vDPss\n5r7jJv4eO+0xHvz8QX7f+rvfoeyjYkX3TNLSpdCwoXtGaeBAWLfO78iMMSVgjQTlKVxr2kcSlIrE\n1voGBRS8JOT9HWGfdY89PhNvxx50LJcefyl3z7jb71Aiql4dhg2DxYtdU95xx7lu4Zs2+R2ZMSaB\nLsJNNdFNA7oBqAX8M9aN80tGEuV9pGVTwgIZAd5Z/A4L1iXvDZqDD3Yzy377rWuyO/poGDnS5lEy\npjTSgG4D1gMneav24AZMjUl+yUijvI+0bEpYrcq1CHQMMGjSIJK9y/sRR8BLL8G0aTB9OjRqBC++\nCFlZBW5qjEkREpQAbpTvnJa08sArsW6fXzJqJsImETYDTb33Ocs2nnMS6NOqD+u3rue9Je/5HUpM\nTjgBJkxwQws9/bSbeXb8eEjyXGqMic25QE+8YeM0oGuB6rFuHDUZqZKuSg1VqqtSznufs5z44aNN\ngcqllWNEtxEMnjqYnXtSp+3rpJPcQ7MPPQR33rl32RiT0nZpQBWv5UyCUrUwG8fc08Ekp64Nu9Lk\noCaM/Hqk36EUioh7JmnePLjhBtc1/Iwz7BklY1LYm15vuv0kKNcB04BnYt24wOGAkoUNBxTd0j+X\n0v7Z9nzf73sOqXaI3+EUyc6d8NRTbmihLl3g3nvdFBbGmOIp4VG7uwKn4Tq5TdaATo1521S5wFsy\nyt/gKYP5e/vfPHv2s36HUiybN8Njj8Hjj8Oll8Jdd0Ht2n5HZUzq8nNsOgnKxRrQN2IqmyoXeEtG\n+du4YyONRjfio8s+ouWhEUdyTynr17ta0ssvQ//+MHiwzaNkTFGUwKjdVYE+QEPcRH1jcR0ZHgCW\nakDPjmk/ibzAi/AscCawTpWmUco8DvTA9cC4WpV5kctZMirIuG/H8fKCl/n06k99H9U7Xn7+2T0w\nO2mSGyW8b1+oVMnvqIxJHdGSkQSlOzAS13fgWQ3o8LDPewL/ArJxQ/3cpgH9OMJ+/gdsBr4CugKH\nAzuAmzSgEa/nkSS6A8PzQLdoH4rQA2ioytG4zFoyc2uXUv9o8Q8279zMW4ve8juUuDniCPdM0rRp\nMGOGe0bphRfsGSVjikOCkgaMxl2fmwC9JCiNw4pN04A204C2AHoD46Ls7mgN6NUa0JyhgI7EjcIQ\ncyKCBCcjVT4H/s6nyNnAS17Zb4CaItgdgiJKT0tnZPeR3Db1Nrbv3u53OHF1wgnumaRXX4Vnn4Wm\nTeH99+0ZJWOKqA2uCW2lBnQ38DruepzLG1EhRzXgjyj72hOyTRawWgO6o7AB+d21uw6wKmR5jbfO\nFFHGERmceNiJPPrVo36HkhAdOsCnn8Lw4a5zQ86yMaZQwq+9q4lw7ZWgnCNBWQx8BAyMsq9mEpRN\n3ssNkuC9l6DEPCJluUIE77thw4blvs/IyCAjI8O3WJLZI10fofXTrendvDd1apS+3J7zjFKPHq6m\ndNVVbjDWBx5wozoYU5ZlZmaSmZkZl31pQN8D3vPmKHoZaBShTHo8jpXw3nQi1AcmROrAIMJYYIYq\nb3jLS4COquSZdMA6MBTOHdPuYM3mNbx07kt+h5Jw9oySMdFF6sAgQWkHDNOAdveWbwc0vBND2DY/\nAW00oH8mIs6SaKYToo/yPR64EkCEdsCGSInIFN7Qk4cybfk0vln9jd+hJFzFim7epKVL3cjgrVvD\njTfaPErG5GMWcJQEpb4EpQJuyqDxoQUkKA1D3rcESFQiggQnIxFeBb4EjhHhFxF6i9BHhOsBVPkI\nWCHCMuApoF8i4ylLqleszv2n3s+gyck/qne8VK/uuoEvWQLlyrmmu3vusXmUjAnndTQYAEzBPRv0\nugZ0sQSljwTleq/Y+RKU7yUoc4BRuJ5yCWMPvZZi2ZpNm6fbcHO7m7ms6WV+h1Pifv7ZTfI3caI9\no2TKLj9HYCgMv3vTmQRKkzRGdh/J7dNvZ+uurX6HU+KOOMI9kzR9OmRm2jNKxiQzS0al3En1TqLD\n4R14+IuH/Q7FN8cf755JsmeUjEle1kxXBvyy8RdaPNWCuX3mUq9mPb/D8ZUqfPQR3HEHVKvm5lQ6\n5RS/ozImcVKlmc6SURlxz4x7WPrXUl47/zW/Q0kKWVnw2mtw991w7LGuW3jz5n5HZUz8pUoysma6\nMmJIhyF8/svnfPHLF36HkhTS0+Hyy13Pux49oHt3uOwyWL7c78iMKZssGZURVStU5cHOD3LTpJvI\n1my/w0kaFSu6Z5KWLnUdHFq3hgED7BklY0qaJaMy5NITLqVcWjleml/6R2UorOrV3TNJS5ZA+fLu\nGaW774aNG/2OzJiywZJRGZImaYzqPoo7P76TzTs3+x1OUjroIBgxAubMgVWr4Jhj3MyzOwo9BrEx\npjAsGZUxbeu25cyjz6TZ2GY88sUj/LktYaN7pLT69fc+o/TJJ64J7/nn7RklYxLFetOVQarKzDUz\neWL2E7y/5H3OaXwO/Vv3p3Wd1n6HlrS+/NKN4vDnn3D//XD22W70cGOSXar0prNkVMb9se0Pnp3z\nLE/OfpKDqx5Mv9b9uLjJxVQuX9nv0JKOqhta6I47oEoV94xSx45+R2VM/iwZxZklo8TKys5i4rKJ\njJk1htlrZ9O7eW9uaHUDDfa3uRjCZWfvfUapUSNXU2rWzHUXNybZWDKKM0tGJWfZX8sYO3ssL8x7\ngbZ129K/dX+6H9WdNLFbjKF27YJx49yss2vXutpSzZrutd9+e99HWxe+XL06pNkpNnFmySjOLBmV\nvG27t/HG928wZtYY/tr+F31b9eWaFtdwQJUD/A4t6WRnw5Ytriv4hg3u35xX+HK0dVu3uoRU2CQW\nuq5qVbuXZfZlySjOLBn5J6fDw5hZYxj/w3jOPfZc+rXqZx0e4iwry829VJRElrNu506oUaNoNbOc\ndZUqWUIrTSwZxZklo+Tw+9bfeW7uc7kdHvq37s9FTS6yDg9JYvdul9AKm8RCX1lZRa+Z5bwqVvT7\nTJgclozizJJRconU4aFvq74cuf+RfodmimnnzuLVzjZudPe+ipLE9tsPatd2zY0mPqIlIwlKd2Ak\n7nnTZzWgw8M+vxQY4i1uBvpqQL9LWJypcoG3ZJS8rMODCaXqRqwoaiLbuNHN0Nuvn/VQjIdIyUiC\nkgb8CHQG1gKzgEs0oEtCyrQDFmtAN3qJa5gGtF3C4kyVC7wlo+S3bfc2Xv/+dcbMGsOGHRvo26ov\nvZv3tg4PplCWLIE+fWD7dtdb0ab2KJ4oyagdENCA9vCWbwc0vHYUUn4/4DsN6OGJitP+dDVxU6V8\nFa5pcQ2zr5vNq+e9yoJ1C2j4eEN6v9+b2Wtn+x2eSRGNG7tp4vv2hW7dYPBg11PRxFUdYFXI8mpv\nXTTXAhMTGVC5RO7clE0iQtu6bWlbt21uh4cL3ryA2tVq069VPy4+/mIqlavkd5gmiYlA795w5plw\n661u6vjRo92yyV9mZiaZmZlx258EpRPQGzgpbjuNdJxUafqyZrrUlpWdxUdLP+KJ2U9YhwdTaNOn\nww03uCa7UaPgsMP8jih15NNMN0wD2t1bjthMJ0FpCvwP6K4B/SmRcVoznSkR6WnpnNXoLCZeNpGv\n/vEVWdlZtH66NWe+eiYTl060Cf9Mvjp3hu++c1PEN2sGY8bYCOrFNAs4SoJSX4JSAbgEGB9aQIJS\nD5eIrkh0IgKrGRkfRerwcE2La6hVuZbfoZkktmiR6+CQMxxTs2Z+R5TcCujaPYq9XbsfkqD0wdWQ\nxklQngbOA1YCAuzWgLZJWJypcoG3ZFR6qSrfrPmGJ2Y9wYQfJ3Bu43Pp17ofrQ5r5XdoJkllZ7v5\npe64A66+GgIBezYpGnvoNc4sGZUNv2/9nWfnPsvY2WOpXa127ggP1uHBRLJ+PdxyC3zxhWu6O/10\nvyNKPpaM4sySUdmS0+FhzKwxzPl1Tu6UFtbhwUQydarrCt6ypevgcOihfkeUPFIlGVkHBpOUcjo8\nTLp8El/+40v2ZO+xDg8mqq5dXQeHY46Bpk3hySddU55JHQmvGYmw7/hHyvCwzzsC7wPLvVXvqHJf\n3v1Yzaissw4PJhYLF7oODllZ8NRTLjmVZalSM0poMhIh8vhHypKQMh2BW1Xpmf++LBkZJ1KHh/6t\n+3PiYSf6HZpJEtnZ8OyzcOedcM01cM89bvLDsihVklGim+naAEtVWanKbuB14OwI5ZL+RJnkISK0\nq9uOl859iR8H/MgxBxzD+W+eT9tn2vLS/JfYsWeH3yEan6WlwXXXuaa7VavcCA6TJvkdlclPomtG\n5wPdVLneW74caKPKwJAyHXEPVq0G1gD/VGVR3n1ZzchEF97h4ZoW19DnxD7W4cEAMHmyGwW8dWsY\nORIOOcTviEqO1Yxi9y1QT5XmwGjgPZ/jMSkotMPDF9d8we6s3bR+ujVnvXaWdXgwdOvmakkNGrh7\nSE89ZR0ckk2ia0Zu/COlu7fsxj8K68QQts0K4ERV/tp3vWggEMhdzsjIICMjIyFxm9Jh2+5tvPbd\na4yZNYZNOze5KS1a9LYOD2Xc99+7Dg6qbgSH44/3O6LESpWaUaKTUTrwA64Dw6/ATKCXKotDytRW\nZZ33vg3wpipH5N2XNdOZosnp8DBm1hg++PED6/BgyM6Gp5+Gu+6Ca6+Fu+8uvR0cLBnlHMB17d47\n/pHykAhu/CNlnAj9gb7AbmA7cLMq3+TdjyUjU3zrt67nubnP8eTsJzm02qH0b92fC5tcaCM8lFG/\n/QY33wwzZ8ITT7jmvNLGklGcWTIy8ZSVncWHSz/kiVlP5HZ4uKHVDRyx3xF+h2Z8MHEi9O8P7drB\niBFQu7bfEcVPqiSjZOjAYEyJS09Lp2ejnrkdHnZl7aLVuFac9dpZTFo2yTo8lDE9erh7SfXqwQkn\nuCY86+BQsqxmZIzHOjwYgAULXAeH9HTX665JE78jKp5UqRlZMjImjKry9eqveWL2E3zw4wec1/g8\n+rfpT8tDW/odmikh2dmup93dd8P117uODpUr+x1V0VgyijNLRsYP1uGhbPv1Vxg0CL791g2+2rWr\n3xEVniWjOLNkZPyU0+FhzKwxzP11rnV4KGM++sh1cOjQAR57DA4+2O+IYlfATK97B7EO6PCwzxsB\nzwMtgaEa0McSGad1YDAmBjkdHiZfPnmfDg89X+vJxKUT2ZW1y+8QTQKdfrrr4FCnjntI9plnUruD\ngwQlDTfiTTegCdBLgtI4rNifwI3AIyUSU6rUNqxmZJJNToeHp+c8zeI/FnNyvZPp2qArXRt25dgD\nj0Uk6VtGTBHMn+86OFSoAGPHwnHH+R1R/iLVjCQo7YCABrSHt+xGxwmrHXmfBYDNVjMyJklVKV+F\nf7T8B19f+zU/DfyJK5tdyffrv+f0/57O4SMO5+r3rua/C/7Lui3r/A7VxFGzZm6a80sugY4dXSeH\n7dv9jqrQ6gCrQpZXe+t8U87PgxtTWhxY5UAuanIRFzW5CFVl2V/LmPLTFN5a9Bb9P+rPEfsdQZcG\nXejaoCsn1z+ZKuVL6dgzZUR6uhsF/JxzXAeHnNllu3TxOzLIzMwkMzPT7zAKzZrpjEmwPdl7mLlm\nJlN/msrU5VOZ99s82tVtl9uk1/yQ5qSJNVKksg8+gAED4OSTXQeHgw7yO6K98mmmG6YB7e4t+95M\nZ8nImBK2aecmMn/OzE1Of27/k85Hds5NTvVq1vM7RFMEW7fCsGHw0kvw4IPQuzckw23DKMko8iDW\nAV2cZ3uXjLZoQB9NaJypcoG3ZGRKq182/sK05dOYunwq05ZPo1blWi4xNehKpyM7UaNiDb9DNIUw\nb557ULZyZdfB4dhj/Y2ngK7dewexDuhDEhQ3iHVAx0lQagOzgepANrAFOE4DuiUhcabKBd6SkSkL\nsjWb+b/NZ+pyV2v6evXXNK3dNDc5tanThvLp5f0O0xQgK8vdQwoGoW9fGDoUKvn0nLQ99BpnloxM\nWbR993Y+/+Xz3OS04u8VdDyiY25yOuaAY6wLeRJbswZuusmNdzd2LJx6asnHYMkoziwZGeOGJ8pp\n0pv601TSJC33XlOXBl04sMqBfodoIpgwwXVwyMiAf/+7ZDs4WDKKM0tGxuxLVVnyx5LcWtOnKz/l\nqFpH5daaOtTrYGPoJZEtWyAQgFdegeHD4aqrSqaDgyWjOLNkZEz+dmft5uvVX+cmp+/Xf8//Hf5/\nucnphNonWBfyJDBnjhvBoWpVN0VFo0aJPZ4lozizZGRM4WzYsYEZK2bkJqdNOzflPnjbtUFX6tTw\n9YH7Mi0rC8aMgXvvdc13d9wBFSsm5liWjOLMkpExxfPzhp9zn22avmI6tavWzr3f1LF+R6pXrO53\niGXO6tUwcCAsXOhqSRkZ8T+GJaM4s2RkTPxkZWcx97e5uclp5pqZtDy0ZW5yanVYK8ql2WhhJeX9\n9+HGG6FzZ3jkETgwjv1QLBnFmSUjYxJn666tfLry09wmvdWbVtPpiE50bdCV0xqeRsNaDf0OsdTb\nvBnuuQdee811cLjyyvh0cLBkFGeWjIwpOb9u/nVvF/LlU6lcrnLu/abODTpTq3Itv0Mstb791o3g\nULOmezbpmGOKtz9LRnFmycgYf6gqC39fmNuk9/kvn9P4wMa5TXrt67anYrkE3X0vo/bsgdGj4b77\n3D2lIUOK3sHBklGcWTIyJjns3LOTr1Z/lZuclvyxhJPqnZSbnJoc1MRGhYiTVatcb7sff3QdHE45\npfD7sGQUZ5aMjElOf23/i49XfJybnHbs2ZHbpNelQRcOrX6o3yGmvPfecx0cTjsNHn4YDjgg9m0t\nGcWZJSNjUsNPf/2Ue6/p4xUfU7dG3dxnm06pfwpVK1T1O8SUtGmTm1X2jTdcj7vLL4+tg4Mloziz\nZGRM6tmTvYfZa2fn1prm/DqH1nVa5yanloe2JD0t3e8wU8qsWW4Eh1q13MjgRx+df3lLRnFmyciY\n1Ld552Y+WflJbnJat3Udpx55am5yOnL/I/0OMSXs2QOPPw4PPOCmPb/tNqhQIXJZS0Y5BxC6AyPJ\nmcBJyTutrfA40APYClytyry8ZSwZGVParN60ep+JBatXqE6Heh3Yr+J+VKtQLeZX1QpVy+S4eytX\nug4OP/3kOjicfHLeMgVMrrf32hx5yvF9r80BzXNtjpeEJiMR0oAfcVPbrgVmAZeosiSkTA9ggCpn\niNAWGKVKu7z7So1klJmZSUYixvSIM4szvlIhzmSPMVuz+W7dd7wy4xXqHFmHLbu2xPzatnsblctX\nzj9plY89ueW88pvIMFnOpyq8+67rAt6jh3tgtlbIY2BRph2PfG0O6JKQMu7aHNAzJCju2hzQPNfm\neEn0eB9tgKWqrAQQ4XXgbNibjLzllwBU+UaEmiLUVmVdgmNLiGT5ghbE4oyvVIgz2WNMkzSaHdKM\nd394l0G9BhVq22zNZtvubTEnrz83/hlTufS09KiJasWSFbTf3L7QCa5SuUpx7fouAuedB126wJ13\nQpMmbs6kSy/Nt4ODuzYH1F2bg5L/tTmg30hQakpQamtAE3JtTnQyqgOsCllejTsJ+ZVZ461LyWRk\njCl5aZKWe7GPF1VlV9auqInqxSUv0vLQlrnL67euZ/nfy9myO/8EtytrV9xrcFXKV6FGjXT+8x+4\n4go3gsOLL7oODlEk3bXZRkI0xpgIRISK5SpSsVxFDqiS98GeucylT6s+hd7vnuw9bN21Naaa2aad\nm1i7eW2BCW7b7m1UKldp7z20PtVY+kc1jr0/fsk54VQ1YS/QdqCTQpZvBx0SVmYs6MUhy0tAa+fd\nF2ove9nLXvYq/CvP9XQY7RjGpJDl2xnGkLAyYxnGxSHLSxhGnmtzvF6JrhnNAo4SoT7wK3AJ0Cus\nzHigP/CGCO2ADZHuF6VC10RjjEkR7toclNiuzUFx1+YE3S8C16UvYVTJAgYAU4CFwOuqLBahjwjX\ne2U+AlaIsAx4CuiXyJiMMaas04DmvTYHdLEEpY8E5XqvjLs2B6VErs0p89CrMcaY0iupnhITkWdF\nZJ2ILMinzOMislRE5olI85KMLySGfOMUkY4iskFE5nivu0o6Ri+OuiLysYgsFJHvRGRglHK+ntNY\n4vT7nIpIRRH5RkTmenE+EKWc3+eywDj9PpdhsaR5MYyP8rnv/9+9OKLGmSznU0R+FpH53u9+ZpQy\nSXE+I0pkB4bCd3jgJKA5sCDK5z2AD733bYGvkzTOjsD4JDifhwDNvffVgB+Axsl2TmOM0/dzClTx\n/i2dqwIAAAZnSURBVE0HvgY6JNu5jDFO389lSCw3A69EiidZzmcMcSbF+QSWA/vn83nSnM9Ir6Sq\nGanq58Df+RQJeUBWvwFqikjtkogtVAxxAvje4UJVf1N1w3eo6hZgMe45gVC+n9MY4wSfz6mqbvPe\nVsS1KoR/B3w/l96xC4oTkuD7KSJ1gdOBZ6IUSYrzGUOckATnExdDftf0pDif0SRVMopBtIewklF7\nryr8oYgc53cwInIErjb3TdhHSXVO84kTfD6nXlPNXOA3IFNVF4UVSYpzGUOckBzfzxHAP3HdjyNJ\nivNJwXFCcpxPBaaKyCwRuS7C58lyPiNKtWSUKr4F6qlqc2A08J6fwYhINeBt4Cav5pGUCojT93Oq\nqtmq2gKoC5wiIh1LOoZYxBCn7+dSRM4A1nk1YiE5ahZ5xBin7+fT00FVW+Jqcf1F5CSf4iiSVEtG\na4DDQ5breuuSiqpuyWkqUdWJQHkRqVXAZgkhIuVwF/iXVfX9CEWS4pwWFGcynVNV3QR8CLQK+ygp\nzmWOaHEmybnsAPQUkeXAa0AnEXkprEwynM8C40yS84mq/ur9+zvwLnmH90mG8xlVMiaj/P5KGg9c\nCSDiPYSliXsIqwBR4wxthxWRNrgu9H+VVGBhngMWqeqoKJ8nyznNN06/z6mIHCgiNb33lYGukGeq\nE9/PZSxx+n0uAVR1qKrWU9UGuAcuP1bVK8OK+X4+Y4kzGc6niFTxWhYQkarAacD3YcV8P5/5Saqx\n6UTkVSADOEBEfgECQAXccBbjVPUjETldRJbh5tfonYxxAheISF9gN7AduNinODsAlwHfefcQFBgK\n1CeJzmksceL/OT0UeFFEcm4Sv6yq00WkD0l0LmOJE//PZVRJeD4jSsLzWRt4V0QUd13/r6pOSZXz\nCfbQqzHGmCSQjM10xhhjyhhLRsYYY3xnycgYY4zvLBkZY4zxnSUjY4wxvrNkZIwxxneWjExKEZFs\nEXkkZPlWEbnHe/+8iJxXjH0/Im4Ki+Fh688Skdu892eLSOOiHiPCMZuJSI9IxzKmLLFkZFLNTuC8\nBA23ch3QVFWHhK5U1Qmq+rC3eA7QpDA7FZH0fD5ujhtLLNKxjCkzLBmZVLMHGAfcEuXzrt6oxUu8\nQS7zCKkBzReRC7117+PmUvo2Z11I+atE5D8i0h7oCTwsbhK1I0WkgYhM9I75iYgc423zvIg8KSJf\nA8NFpLWIfCki34rI5yJytMj/t3c/r3GUcRzH3x8hICpBkEpBaAmpoNIGVAr+aEXIuRShiSiBEnLI\nrT31H9BDFfFQRHoq/upBDKQVeyi0UjHYgsFgSqEUEgoeeylYPejBj4fn2e647MruabLwecHCzDwz\nzzyzsPPlmR2+X00A7wHztb+5zrlqH3slfa+SDfqKSimDTt9nJP0kaaszG5S0u45hQ9LNmtkiYizs\nqHRAEUMw8CklddCHfdr32j4oaR9wTdK07b87jfXGPWP7gKSngXVJP9o+Kun3mvW473lt31Cp9Pmd\n7dXa31Vg2fZ2zUt2Fpitxzxj+5W63xPAIdv/SJoFTts+Vh8xvmz7RN3vON1SBZ8An9k+L2mxrr9V\n23bbfl3S85ScY6vAu8Bl26drOqDHRvliI9qUYBRjx/Yfkr4ATlJygTV9U/fZkrQNPAc0y8MfomRf\nxvY9ST8AB4FLjFjGoCakfA1YqTd/gInGLiuN5SeBLyU9Swk2w/z2XqUbfL4CmsH3Yr2G2zWoAqwD\n5+qM61vbm6NcT0Sb8pguxtUZYAl4vGd7M9mietb7aQagURM1PgLct/2S7RfrZ3+j/c/G8vuUjM8H\ngCPAo0P0/3/j+auxLADba8AblLIAn0taGOYiInaCBKMYN50b733KLGipp31OxTQwBdzpaV8D3lap\nhroLOEy3quwwM6MHwGQdwwPgrqRjDwcnzQw4bpJu7ZhmtuSH/fVxHXinLi/Usfejeu49wD3b5ygl\nsgc9cozYcRKMYtw0ZwsfA081thn4DfiZUlRuufl/EYDtC5THdpvAVeBULUbW2/cgXwOn6osIU5TS\nF0v1JYNblBcc+vX1EfCBpF/47+/uGvBC5wWGnmNOAIuSfq3nOTmg7876m8CmpA1gnjJ7jBgLKSER\nERGty8woIiJal2AUERGtSzCKiIjWJRhFRETrEowiIqJ1CUYREdG6BKOIiGhdglFERLTuX1qqBR+z\n8bLFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x8c6fbd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Here, given a symmetrix matrix M, we will simultaneously build\n",
    "its tridiagonal Lanczos factorization, Hpp as seen previously,\n",
    "And the Choleski factorization of Hpp as LDLt\n",
    "\"\"\"\n",
    "\n",
    "size = 5\n",
    "M = np.random.rand(size,size)\n",
    "M = np.dot(M,M.T) #Symmetric\n",
    "y = np.random.rand(size)\n",
    "xref = np.dot(np.linalg.inv(M),y)\n",
    "error = []\n",
    "residual = []\n",
    "\n",
    "#Allocate matrices and vectors\n",
    "Hpp = np.zeros((1,1))\n",
    "Vp = np.zeros((size,1))\n",
    "yp = np.zeros((1,1))\n",
    "L = np.ones((1,1))\n",
    "D = np.zeros((1,1))\n",
    "\n",
    "#Initialize\n",
    "Vp[:,0] = y/np.linalg.norm(y)\n",
    "yp[0] = np.dot(Vp[:,0],y)\n",
    "\n",
    "\"\"\"\n",
    "Here, we iteratively build a new problem in K\n",
    "and solve it, before adding another vector to the\n",
    "Krylov basis \"\"\"\n",
    "for i in np.arange(1,size+1):\n",
    "\n",
    "    #Compute next Vp\n",
    "    v = np.dot(M,Vp[:,i-1])\n",
    "    #orthogonalize\n",
    "    minId = np.maximum(0,i-2)\n",
    "    orth = np.dot(Vp[:,minId:i].T,v)\n",
    "    Hpp[minId:i,i-1] = orth\n",
    "\n",
    "    if( i < size ):\n",
    "        #Resize matrices and vectors for the new problem\n",
    "        Hpp=np.pad(Hpp,((0,1),(0,1)), mode='constant')\n",
    "        Vp=np.pad(Vp,((0,0),(0,1)), mode='constant')\n",
    "        yp=np.pad(yp,((0,1),(0,0)), mode='constant')\n",
    "        L=np.pad(L,((0,1),(0,1)), mode='constant')\n",
    "        L[i,i]=1\n",
    "        D=np.pad(D,((0,1),(0,1)), mode='constant')\n",
    "\n",
    "        v = v - np.dot(Vp[:,minId:i],orth)\n",
    "        #Update Hpp\n",
    "        Hpp[i,i-1] = np.linalg.norm(v)\n",
    "        #Normalize vector and update V\n",
    "        Vp[:,i] = v / Hpp[i,i-1]\n",
    "        #Update Yp\n",
    "        yp[i] = np.dot(Vp[:,i],y)\n",
    "\n",
    "        #check orthogonality of Vp\n",
    "        assert( np.allclose(np.identity(i+1),np.dot(Vp.T,Vp)) )\n",
    "\n",
    "    #Update the LDL^t factorization\n",
    "    if( i > 1 ):\n",
    "        L[i-1,i-2] = Hpp[i-1,i-2]/D[i-2,i-2]\n",
    "        D[i-1,i-1] = Hpp[i-1,i-1]-L[i-1,i-2]*Hpp[i-1,i-2]\n",
    "    else :\n",
    "        D[i-1,i-1] = Hpp[i-1,i-1]\n",
    "\n",
    "    #Check the LDLt Choleski factorization of Hpp\n",
    "    assert( np.allclose(Hpp[:i,:i],np.dot(L[:i,:i],np.dot(D[:i,:i],L.T[:i,:i]))) )\n",
    "\n",
    "    #Solve the problem in the current K space\n",
    "    Sp=solveCholeskiForm(L[:i,:i],D[:i,:i],np.reshape(yp[:i],i))\n",
    "    #current estimate\n",
    "    xk = np.dot(Vp[:,:i],Sp)\n",
    "    error.append(np.linalg.norm(xk-xref))\n",
    "    residual.append(np.linalg.norm(np.dot(M,xk)-y))\n",
    "\n",
    "#Check the equivalence between M and VptHppVp\n",
    "assert( np.allclose(Hpp,np.dot(Vp.T,np.dot(M,Vp))) )\n",
    "\n",
    "\n",
    "fig, ax1 = plt.subplots()\n",
    "fig.suptitle(\"Lanczos / Krylov projection based method convergence\")\n",
    "ax1.set_xlabel(\"Nb of iterations\")\n",
    "#axe 1 : Error\n",
    "ax1.set_ylabel('Error norm', color='b')\n",
    "for tl in ax1.get_yticklabels():\n",
    "    tl.set_color('b')\n",
    "ax1.plot(np.arange(1,size+1),error,color='b',label='Error')\n",
    "#axe 2 : Residual\n",
    "ax2 = ax1.twinx()\n",
    "ax2.set_ylabel('Residual norm', color='g')\n",
    "for tl in ax2.get_yticklabels():\n",
    "    tl.set_color('g')\n",
    "ax2.plot(np.arange(1,size+1),residual,color='g',label='Residual')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conjugate Gradient approach\n",
    "Although the previous algorithm features some interesting properties, we can notice that it is not exactly a method that iteratively builds a solution vector. Instead, it iteratively builds a new problem, where $\\vec{s_p}$, the $M$-projection of the solution $\\hat{x}$ in the Krylov sub-space $\\mathcal{K}_p$ expressed in the basis $V_p$ can be solved whithout too much effort, thanks to the triangular/diagonal structure of the $LDL^\\intercal$ Choleski factorization.\n",
    "\n",
    "An alternative formulation where the current solution $\\vec{x_p}$ would be iteratively constructed in the Krylov subspace $\\mathcal{K}_p$ using sequential vectors $\\vec{v_k}, k = 0,1,\\dots,p-1$ from the basis $V_p$ would allow us to save the computation needed to solve the full projection problem at each iteration, and the main update would look like:\n",
    "  \n",
    "\\begin{equation}\n",
    "  \\vec{x_{p+1}} = \\vec{x_{p}} + \\alpha_p \\vec{v_{p}}\n",
    "\\end{equation}\n",
    "\n",
    "This is the basic idea of the conjugate gradient algorithm, we will explain here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General Idea of Conjugate Gradient in the framework of Krylov methods\n",
    "\n",
    "We must recall that the solution vector $\\vec{s_p}$ at each step is the orthogonal projection of the general solution $\\hat{x}$ over the Arnoldi basis $V_p$.\n",
    "But it is orthogonal with respect to a specific inner product definition that uses $M \\in S^n_{++}$, although the Arnoldi basis $V_p$ column vectors are orthogonal with respect to the canonical inner product based on the identity matrix.\n",
    "So there is no guarantee that the $p$ first coordinates of $\\vec{s_p}$ in $V_p$ will remain the same in $\\vec{s_{p+1}}$ expressed in the new basis $V_{p+1}$.\n",
    "This is why, at each iteration of the previous algorithm, the linear combination of vectors from $V_p$ had to be fully recomputed by solving a structured linear set of equations.\n",
    "\n",
    "To overcome this problem, let's first recall that the problem to solve at each iteration as seen previously, reads:\n",
    "      \n",
    "\\begin{align*}\n",
    "  H_{pp} \\vec{s_p} - V_p^\\intercal \\vec{y} = 0 \\\\\n",
    "  H_{pp} \\vec{s_p} = V_p^\\intercal \\vec{y} \\\\\n",
    "  H_{pp} \\vec{s_p} = \\vec{y_p}\n",
    "\\end{align*}\n",
    "\n",
    "As $\\vec{y_p}$ and $\\vec{y_{p+1}}$ only differs in the last coordinate that have been added to $\\vec{y_{p+1}}$, as seen earlier, if we want to have a coordinate-wise resolution of the problem in the basis $V_p$, the same property should apply to $H_{pp} \\vec{s_p}$, which could be translated in:\n",
    "\n",
    "\\begin{align*}\n",
    "  \\begin{pmatrix}\n",
    "    1 \t   & 0 \t    & \\dots  & 0 \\\\\n",
    "    0 \t   & \\ddots & \\ddots & \\vdots \\\\\n",
    "    \\vdots & \\ddots & 1      & 0 \\\\\n",
    "    0 \t   & \\dots  & 0      & 0 \\\\\n",
    "  \\end{pmatrix}\n",
    "  H_{p+1p+1} \\vec{s_{p+1}} = H_{pp} \\vec{s_p} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Which imply that $H_{pp}$ should be constructed iteratively so that $H_{pp}$ and $H_{p+1p+1}$ only differs from the last line and the last column, but also that $H_{pp}$ should be diagonal for every $p$.\n",
    "\n",
    "To achieve this simple construction scheme, it is obvious that $H_{pp}$ should be diagonal. To do so, we have to find a new Krylov basis $W_p$ such that our new $H_{pp}$ is diagonal:\n",
    "\n",
    "\\begin{equation}\n",
    "  H_{pp} = W_{p} ^\\intercal M W_{p}\n",
    "\\end{equation}\n",
    "\n",
    "Fortunately, we don't need to take the Krylov basis design problem from the beginning to handle this constraint. We have already seen that the residue $\\vec{r_p}$ is orthogonal to $\\mathcal{K}_p$, and by construction, $\\vec{r_p} \\in \\mathcal{K}_{p+1}$ :\n",
    "\n",
    "\\begin{equation}\n",
    "  \\vec{r_p} = M \\vec{x_p} - \\vec{y}\n",
    "\\end{equation}\n",
    "\n",
    "with $\\vec{y} \\in \\mathcal{K}_1$, $\\vec{x_p} \\in \\mathcal{K}_p$ so $M \\vec{x_p} \\in \\mathcal{K}_{p+1}$. We conclude that $\\vec{r_p}$ lies in the orthogonal complement of $\\mathcal{K}_p$ in $\\mathcal{K}_{p+1}$ that we can write $\\mathcal{K}_{p+1} \\setminus \\mathcal{K}_p$ which is of dimension $1$ and has an orthonormal basis $\\vec{v_{p}}$. So every $\\vec{r_p}$ can be written $\\alpha \\vec{v_{p}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Construction of a new Krylov space basis\n",
    "\n",
    "The last remark in the previous part can be directly translated into a matrix equality, let $R_p$ be the matrix of all ordered $p$ first residual vectors $\\vec{r_k}, k \\in 0,1,\\dots, p-1$, we can write:\n",
    "    \n",
    "\\begin{equation}\n",
    "  R_p = V_p \\Delta_p\n",
    "\\end{equation}\n",
    "\n",
    "Where $\\Delta_p$ is a diagonal matrix, and we can now extend properties valid for the decomposition of $H_{pp}$ :\n",
    "\n",
    "\\begin{align*}\n",
    "  &R_p^\\intercal M R_p \\\\\n",
    "  = &\\Delta_p^\\intercal V_p^\\intercal M V_p \\Delta_p \\\\\n",
    "  = &\\Delta_p^\\intercal H_{pp} \\Delta_p \\\\\n",
    "  = &\\tilde{H}_{pp} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Where the matrix $\\tilde{H}_{pp}$ that is equivalent to the projection of the linear operator $M$ over $\\mathcal{K}_p$ expressed in the basis $R_p$ is still tridiagonal and positive definite, and admits an alternative Choleski decomposition of the form $ \\tilde{H}_{pp} = \\tilde{L}_p \\tilde{D}_p \\tilde{L}_p^\\intercal $ such that we can write\n",
    "\n",
    "\\begin{align*}\n",
    "  &R_p^\\intercal M R_p &= \\tilde{L}_p \\tilde{D}_p \\tilde{L}_p^\\intercal \\\\\n",
    "  \\Leftrightarrow & \\Delta_p^\\intercal V_p^\\intercal M V_p \\Delta_p &= \\tilde{L}_p \\tilde{D}_p \\tilde{L}_p^\\intercal \\\\\n",
    "  \\Leftrightarrow & \\tilde{L}_p^{-1} \\Delta_p^\\intercal V_p^\\intercal M V_p \\Delta_p \\tilde{L}_p^{-\\intercal} &= \\tilde{D}_p \\\\\n",
    "  \\Leftrightarrow & W_{p} &= R_p \\tilde{L}_p^{-\\intercal} \n",
    "\\end{align*}\n",
    "\n",
    "Column vectors from $W_{p}$ are derived from residual vectors, they form a basis of $\\mathcal{K}_p$ and they features the desired property stated in the previous section: they are all orthogonal to each other with respect to the inner product $\\langle .,. \\rangle_M$ in $\\mathcal{M}^n$.\n",
    "\t\n",
    "Knowing that $\\tilde{L}_p$ is bidiagonal inferior by construction, as seen in the Choleski factorization, we can also derive a short recurrence pattern over $\\vec{w_0}, \\vec{w_1}, \\dots, \\vec{w_{p-1}}$ the column vector of $W_p$, assuming that $\\gamma_0, \\gamma_1, \\dots, \\gamma_{p-1}$ are the subdiagonal elements of $\\tilde{L}_p$:\n",
    "\n",
    "\\begin{align*}\n",
    "  W_p \\tilde{L}_p^t &= R_p \\\\\n",
    "  \\begin{pmatrix}\n",
    "    \\begin{pmatrix}&\\\\&\\vec{w_0}\\\\&\\end{pmatrix}\n",
    "    \\begin{pmatrix}&\\\\&\\vec{w_1}\\\\&\\end{pmatrix}\n",
    "    \\dots\n",
    "    \\begin{pmatrix}&\\\\&\\vec{w_{p-1}}\\\\&\\end{pmatrix}\n",
    "    \\end{pmatrix}\n",
    "  \\begin{pmatrix}\n",
    "    1 & \\gamma_0 & 0 & \\dots & 0 \\\\\n",
    "    0 & 1 & \\gamma_1 & \\ddots & \\vdots \\\\\n",
    "    \\vdots & \\ddots & \\ddots & \\ddots & 0 \\\\\n",
    "    \\vdots & \\ddots & \\ddots & 1 & \\gamma_{p-2} \\\\\n",
    "    0 & \\dots & \\dots & 0 & 1 \\\\\n",
    "  \\end{pmatrix} &=\n",
    "  \\begin{pmatrix}\n",
    "    \\begin{pmatrix}&\\\\&\\vec{r_0}\\\\&\\end{pmatrix}\n",
    "    \\begin{pmatrix}&\\\\&\\vec{r_1}\\\\&\\end{pmatrix}\n",
    "    \\dots\n",
    "    \\begin{pmatrix}&\\\\&\\vec{r_{p-1}}\\\\&\\end{pmatrix}\n",
    "  \\end{pmatrix}\n",
    "\\end{align*}\n",
    "\n",
    "This formulation may give us a way to construct $w_{j+1}$ with a simple iterative scheme, assuming that $\\vec{x_0} = \\vec{0}$:\n",
    "\n",
    "\\begin{align*}\n",
    "  &W_p \\tilde{L}_p^{\\intercal} &= R_p \\\\\n",
    "  \\Rightarrow & \\vec{r_0} &= \\vec{w_0} = \\vec{y} - M \\vec{x_0} \\; \\text{ and } \\\\\n",
    "  & \\vec{r_{j+1}} &= \\vec{w_{j+1}} + \\gamma_{j} \\vec{w_j} \\\\\n",
    "  \\Leftrightarrow & \\vec{w_{j+1}} &= \\vec{r_{j+1}} - \\gamma_{j} \\vec{w_j} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "This relationship between $\\vec{w_{j+1}}$ and $\\vec{w_j}$ can be further exploited, if we use the fact that all vectors from $W_p$ are pairwise orthogonal with respect to the inner product defined by $M$:\n",
    "\n",
    "\\begin{align*}\n",
    "  & \\langle \\vec{w_{j+1}},\\vec{w_j} \\rangle_M &= 0 \\\\\n",
    "  \\Leftrightarrow & \\langle \\vec{r_{j+1}} - \\gamma_{j} \\vec{w_j},\\vec{w_j} \\rangle_M &= 0 \\\\\n",
    "  \\Leftrightarrow & \\vec{r_{j+1}}^\\intercal M \\vec{w_j} - \\gamma_{j} \\vec{w_j}^\\intercal M \\vec{w_j} &= 0 \\\\\n",
    "  \\Leftrightarrow & \\gamma_{j} &= \\frac{\\vec{r_{j+1}}^\\intercal M \\vec{w_j}}{\\vec{w_j}^\\intercal M \\vec{w_j}}\n",
    "\\end{align*}\n",
    "\n",
    "We now need to see if $\\vec{r_{j+1}}$ can be found using a simple recursion. To do so, we have to use the coordinate-wise construction of $w_{j+1}$ exposed previously  :\n",
    "\n",
    "\\begin{align*}\n",
    "  \\vec{x_{j+1}} &= \\vec{x_{j}} + \\alpha_j \\vec{w_{j}} \\\\\n",
    "  M \\vec{x_{j+1}} - \\vec{y} &= M \\vec{x_{j}} + \\alpha_j M \\vec{w_{j}} - \\vec{y} \\\\\n",
    "  \\vec{r_{j+1}} &= \\vec{r_{j}} + \\alpha_j M \\vec{w_{j}} \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Here, we can use the fact that $\\vec{r_{p}}$ is orthogonal to $\\mathcal{K}_p$, and especially $\\vec{r_{j}}$ is orthogonal to $\\vec{w_{j-1}}$ so that we can write:\n",
    "\n",
    "\\begin{align*}\n",
    "  & \\langle \\vec{r_{j+1}},\\vec{w_{j}} \\rangle &= 0 \\\\\n",
    "  \\Leftrightarrow & \\langle \\vec{r_{j}} + \\alpha_{j} M \\vec{w_{j}},\\vec{w_{j}} \\rangle &= 0 \\\\\n",
    "  \\Leftrightarrow & \\langle \\vec{r_{j}},\\vec{w_{j}} \\rangle + \\alpha_{j} \\langle M \\vec{w_{j}},\\vec{w_{j}} \\rangle &= 0 \\\\\n",
    "  \\Leftrightarrow & \\vec{r_{j}}^\\intercal \\vec{w_{j}} + \\alpha_{j} \\vec{w_{j}}^\\intercal M \\vec{w_{j}} &= 0 \\\\\n",
    "  \\Leftrightarrow & \\alpha_{j} &= \\frac{ -\\vec{r_{j}}^\\intercal \\vec{w_{j}} }{ \\vec{w_{j}}^\\intercal M \\vec{w_{j}} }\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Conjugate Gradient algorithm in practice\n",
    "\n",
    "The force of the Conjugate gradient algorithm lies in the fact that it iteratively builds a diagonal problem which can be solved one coordinate per iteration, without \"forgetting\" about the previous iterations. More interestingly, it is not solved in the canonical coordinate system of $\\mathbb{R}^n$ but instead in a basis of pairwise orthogonal vectors with respect to the matrix $M$, called conjugate vectors, which are iteratively built using the gradient of the function $f(\\vec{x})$ we want to minimize at the current solution estimate : $f(\\vec{x}) = \\frac{1}{2} (\\vec{x}-\\vec{\\hat{x}} ) (M \\vec{x} - \\vec{y})  $.\n",
    "\n",
    "By design, these vectors are able to catch a large part of the error at every iteration, and especially, if the matrix $M$ defines an ellipsoid with an important anisotropy, the conjugate strategy helps to avoid the drawback of descent methods that are often ``stuck'' in valleys.\n",
    "\n",
    "The algorithm reads:\n",
    "\n",
    "\\begin{algorithm}\n",
    "  \\caption{Solve $M \\vec{x} - \\vec{y} = 0$}\n",
    "    \\begin{algorithmic}\n",
    "      \\REQUIRE $M \\in S^n_{++}$ the positive definite cone\n",
    "      \\STATE \\textbf{Initialization}\n",
    "      \\STATE $\\vec{r_0} = \\vec{y}$\n",
    "      \\STATE $\\vec{w_0} = \\vec{r_0}$\n",
    "      \\STATE $j = 1$\n",
    "      \\STATE \\textbf{Iterations}\n",
    "      \\WHILE{$j \\leq n$}\n",
    "      \\STATE $\\alpha_{i-1} = \\frac{ \\vec{r_{i-1}}^\\intercal \\vec{w_{i-1}} }{ \\vec{w_{i-1}}^\\intercal M \\vec{w_{i-1}} }$\n",
    "      \\STATE $\\vec{x_i} = \\vec{x_{i-1}} + \\alpha_{i-1} \\vec{w_{j-1}}$\n",
    "      \\STATE $\\vec{r_i} = \\vec{r_{i-1}} - \\alpha_{i-1} M\\vec{w_{j-1}}$\n",
    "      \\IF{$\\vec{r_i} \\neq \\vec{0}$}\n",
    "        \\STATE $\\gamma_{i-1} = \\frac{ \\vec{r_{i}}^\\intercal M \\vec{w_{i-1}} }{ \\vec{w_{i-1}}^\\intercal M \\vec{w_{i-1}} }$\n",
    "        \\STATE $\\vec{w_i} = \\vec{r_{i}} - \\gamma_{i-1} \\vec{w_{j-1}}$\n",
    "      \\ELSE\n",
    "          \\STATE End iterations\n",
    "      \\ENDIF\n",
    "       \\ENDWHILE\n",
    "    \\end{algorithmic}\n",
    "\\end{algorithm}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x122d4450>]"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALUAAADECAYAAADZLWDNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACTZJREFUeJzt3WusXFUdhvHnTy+AvVAKSm2hBQkq2nghFkMFTPvBqoVG\nYyKlXoiNFY0mfAA1rdVC1QaiBmIajAElUiNVEyRBKhql4VYqQQzGUAVCb9xKb+FU22qLyw9rnbI7\nneuZac85L+8vaZiZPXvttWee2bPndE6JlBJmSo4b7AmY9ZqjNjmO2uQ4apPjqE2OozY5PY86In4U\nEd/o9bi9FBFrI2Jhk+W3RcSuiFh/LOdVZx4XRsSGozDugoi4t9fjtrHdmRHxVET0RcS8o7ahlFLb\nf4BNwF7gFWAX8BBwJRCdjDPYf4C1wMIGyy4EtgAndLmNDwJbh8C+TgP+Bxw3BObyR+ArTZZvAvYD\nE2tu/2vZh6ntbKfTI3UC5qaUTioP1vXA14GfdDjOUHYmsCmltL/LcYL8eA1s5YgRXW6/dh7Ro/G6\nMQ14ssnyBGwELu+/ISKmAyfSyWPZ4SttIzC75rYZwKvAO8r124Dl5fIE4G7gZWBnuTy5su6ZwP3k\nI/8fgJXAqsryecDfye8K9wFvr5nL1cATwG7gDmB0k+1Oqaxb90gNLAT2AQeAPmBZG/twMvBT4Pmy\n/E7gDeR3tIPAnjLWJGA0cFO573PAjcCo6pEd+BrwIvAzKkd74JOVsfrIR7T7yrKPAo+Xx3EzsKwy\nv83l+elf9/3AFcCDlfvMBB4tj+OfgQtqHqvl5HflPuBeao6kNY/hIuBpYAdwFzCp3P5MeTz2lnFG\nNehrCfBo5bbvAYvLPrR1pO466soDd2WdqCcCHweOB8YAvwR+U1lvHXADMBL4QHlSbi/L3gr8C5gN\njAC+Wh6skZW5rAdOK+E9CXyhze02O/24Anigcr3VWPeQX1DjyzwvqkS6pWbs5WWfTyl/Hgauq9z/\nALACGFW2d8QY5b7jyv5+vly/GHhnuTyd/KKYV65PK0FEvX0kvyh3AQvIn7Hml+snVx6rp4Gzy5zW\nAisaPHazge3Au8s+/BC4v6afWa36AjYAbyvz2QKcQQenH72K+hFgcW3Ude73HmBnuTwV+C+Vc1dg\nFa9FvRRYXVkW5KPbxZW5XF5ZfgNwc6vtdhp1i314M/noM77O/epF/Qwwp3L9Q8Czlfvvp3IEazBG\nAL8FVjaZ443AD2qiPq7ePgKfBtbXrL8O+GzlsVpSWfYlYE2D7d4KXF+5PqY8x1Ob9VMn6iXkF/cc\n4Pfkg0XbUY+kN6aQX92HiYgTyW+3c8hH0wDGRkSQg9iVDj933QqcXi5PJr8DAJBSShGxtWyr37bK\n5b1lzKbbTeXRa1eLfTi97ENfm8NNJh95+m0ut/XbnlI60GKMFeRYrqrM8Xzy55vp5FOc0cCvO5jT\n5prbNnP44/xS5fJeYGyTsf7SfyWl9O+I2FnG2tJgnXp+DjwAnAXc3sF6QA9+pBcRM8g782CdxdcA\n5wAzUkoTyG+TkMN4EZgYESdU7n9G5fIL5KMMNcufa2NazbbbqaubjLWVvA/j66xX78XzPIfv0zTy\nfjZb55CImA9cBnwipfRqZdEvyOevU8ocf8xr+9rqRfwC+bNN1dQy104d9pxFxBjyaVY7z9khKaUt\n5KP2R8ifUToy4KgjYlxEXEI+n1yVUqr3qXYs+YNXX0RMBK7tX1Am/hhwbUSMiogLgEsr6/4KmBsR\nsyJiZERcQ357fqSN6TXc7gCMa7IPLwG/A26OiAllnheVxduAU2qCXw0sjYhTI+JU4JvkU66WIuK9\n5HPUj6WUat8VxwK7U0oHylF7QWXZdvJb99kNhl4DnBMR8yNiRERcBpxL/kDcqTuAz0XEuyLiePK7\nyvqU0tYBjLWQfKqyr9MVBxL13RHxCvntZDHw/TKBem4i/yRgB/k8bU3N8k+RP3nvIH+IWg38ByCl\n9BT5fG8l+YmZC1yaUjpY1m12BGq13U5OQVqN9RnyefU/yCFfVeb/T/KT/Gz5i5xJwHfIL+S/kX9q\n8xjw3TbnMY98+vNQROwpf4FxT1n2ZeDb5XlZSv4wS5nHvrKNh8s8zq8OWl4gl5Df3XaU/85NKe3u\nv0ub8yOl9CfyC/VO8pH+LPIHz0N3aTVEZayNKaXHO1j3kOjwFPOoiojVwIaU0nWDPRcbvgb1ux8R\n8b6IeEtkHyYfje4azDnZ8Nern34M1CTyW9VE8oeJL6aUnhjcKdlwN6ROP8x6oemROiJcvA1pKaUj\nfkzb8vTjW11udDnLuhzBNJ3W1dqLFp3LLbfMqrvMvyRgchy1yXHUJsdRmxxHbXIctclx1CbHUZsc\nR21yHLXJcdQmx1GbHEdtchy1yXHUJqfl96mXd/2N6l78xtjB1nexYWZb67s0NaXhEh+pTY6jNjmO\n2uQ4apPjqE2OozY5jtrkOGqT46hNjqM2OY7a5Dhqk+OoTY6jNjmO2uQ4apPTxjf4B/L/06zqxRf8\nu/1FA/+SweuJj9Qmx1GbHEdtchy1yXHUJsdRmxxHbXIctclx1CbHUZscR21yHLXJcdQmx1GbHEdt\nchy1yenFP/N/DHT7Jf9xPZjDnh6MYceCj9Qmx1GbHEdtchy1yXHUJsdRmxxHbXIctclx1CbHUZsc\nR21yHLXJcdQmx1GbHEdtcobJ96m71YvvQs/swRjrejCGteIjtclx1CbHUZscR21yHLXJcdQmx1Gb\nHEdtchy1yXHUJsdRmxxHbXIctclx1CbHUZscR21yXie/JNAL/oL/cOEjtclx1CbHUZscR21yHLXJ\ncdQmx1GbHEdtchy1yXHUJsdRmxxHbXIctclx1CbHUZscR21yHLXJcdQmx1GbHEdtchy1yXHUJsdR\nmxxHbXIctclx1CbHUZscR21yHLXJcdQmx1GbHEdtchy1yXHUJsdRmxxHbXIctclx1CbHUZscR21y\nHLXJcdQmx1GbHEdtchy1yXHUJsdRmxxHbXIctclx1CbHUZscR21yHLXJcdQmx1GbHEdtchy1yXHU\nJsdRmxxHbXIctclx1CbHUZscR21yHLXJcdQmZ2Tru7ypy0283OX6Zp3xkdrkOGqT46hNjqM2OY7a\n5Dhqk+OoTY6jNjmO2uQ4apPjqE2OozY5jtrkOGqT46hNTsvvU5933hu73MSILtc3O9LUqSc1XBYp\npcYLIxovNBsCUkpRe1vTqM2GI59TmxxHbXIctclx1CbHUZuc/wOmj30ay/PDXQAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12061850>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEhCAYAAACgIq2RAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXeYVEXWh9/DkLMEQUVAFEREdEEFRGHIEsQACCIqmNDF\nNWD4dFe3aV11zSAGlFUUEFFQCUpUyYoiSBBQMaIgCCIZCcP5/qg70DPMTN+Z6Thz3ue5z9xbXbfq\n12H6dFWdOkdUFcMwDMOIJUXiLcAwDMMofJjxMQzDMGKOGR/DMAwj5pjxMQzDMGKOGR/DMAwj5pjx\nMQzDMGJOoTc+IrIz3hpyg4i8KCLNM5UFRGSQd15SRGaKyL9z2W5CvA4i8rKI1M/DfWeKSKeQ64tE\n5J4I6KklIivz204u+pstIo0j3GYFEbk55LqViEzJR3v5ut8wwIwPQLJtdGoKLMrqAREpBkwAFqvq\ng1k8Ljm0G7PXQUSy/dyp6o2q+nUemj0L6BzSzhRVfTwv+rKSFaF24sUxwN8zleX3OSXkayIiKfHW\nYPjDjE8WiEhXEVkkIku8UURVrzwgIq94v06/E5F/hNxztYgsF5EvReR1r+xLEVnq/d0jIheIyDEi\n8p5X9xMROcOr2yqk/hIRKZOFrvrAt5r1zuBiwFve4//y6tcSka9F5HXv1/v9IvJMSHvXi8hT6Zch\n5U+IyEpPY0+v7M1MI4uRInJZJn2tRGSuiLzv9ftCyGM7ReRJEfkSaCYibb3nulxE/ucZzgy//EWk\nvfcafSEib4lIaa/8HBFZKCLLvPepPPAgcLnXZk8RuUZEhoW8Dh959WeJSI2Q5zDUa+u7zM8n9LUV\nkTEislpE3haRkt79D4jIZyKyQkSGhzzXW0VkldffWK+stPfZSf9cdfPKS3qv7SoReRcomZUAEflR\nRB7xPiOLRaSxiMwQkbUiMiCk3l0i8rnXd8ArfhSo4702j3ll5URkvIisEZHRIfdn975c6NX9Asjy\ndRKRIiGfnWUiMjBMmz+KyGDv9VguIvXE8aP3nqa3+62IVBWRKiIywXvNPxNvBkDc/+UoEVkAjBKR\nUt779JWIvOu95uE+U0dp8crLiMir3nu8TEQuzakdIxeoaqE+gB1ZlFUIOb8OeMI7DwALgKJAZWAL\nkAKcDnwNHOPVq5ipva7AXK/us8ADXnlr4EvvfDLQ3DsvDRTJQtcdQL8sygPAH8CbmcprAQeBc7zr\nMsBaIMW7Xgg0CH0dgO7ADO/8WOBnoBpwCfCaV17MKy+Rqb9WwB6vXwFmApd5jx0CunvnJYB1wMne\n9evArd75bKCx9/rOBUp55fcA93t9fw809srLeq/rNcCzIVoOX3uvbV/vvD/wnnc+EnjLOz8NWJvF\na1vL097Mu34FGJT5fQZGAV288/VAMe+8vPf3YaBP+ucL+AYo5b2n//PKzwAOpD+3TDp+BG70zp8G\nlnufkyrARq+8PfCSdy7AFOB87zmsyPQ+/Qkc59X7BDgvu/clpLyOV/4WMDkLjTcBbwOS/vqEea9/\nBP7und8MvOydPwNc452fC8z0zt8AzvPOTwRWh3z+FwPFves7gRe989OB/eTwmQqj5b/A06HfDdm0\n80C8v8uS7bCRT9ac6P2qXAHchfsAp/OBqh5U1T+ATbgv5tbAeFX9E0BVt6VXFpG6wBNAT1VNw30Z\njPbqzQYqiUhZnCF4Rtxo6hhVPZSFro7A9Gw0zwfO8/oL5WdVXez1txv4GOgqIqcCRVV1dab6LYA3\nvfq/A3OAc4BpQKr3q7UTME9V92Wh43NV/Vndf+Wb3vMFSAPe9c5PBX5Q1e+969eBlpnaaQY0ABaK\nGy1djfsSPRXYoKpLPY27vNc1J5qnPyfca98i5LGJXjtrcMY2K9apavpU55iQ59TW+1W9AvcZSP+c\nLAfGisiV3vMG6ADc6z2XOUBxoKb3vMd4GlZ692ZH+jrLSmCRqu5R1S3AX95IoQPQXkSWAktxr1Xm\nz0M6n6vqb977tAyoTfbvS32v/IeQ1yAr2uGMn3rPZ1sObabznvd3iacBnAHr7Z33xhm79Paf817D\nyUDZkBHHZFXd752fD4zzNKwCVnjlWX2maobR0g54Pr2Cqm730Y7hg6LxFpCgDAOeVNUPRKQV7pdV\nOqFfuGkceQ2PWk8RN3X2FnCd90UOR8+VC4CqPiYi7wNdcB/qDqr6bUhbpXAjso3ZaJ6H+8eeJiIt\nVHWTV747U71XgH/iRmojs2krK337RGQOcCHQiyNf5pnJ/PzSr/emfymFthum35mqemWGQpGGPu4N\npymU0Pczu3aPek4iUgL3pdRYVTd4U1zpU2ZdcF+w3YB/iZtaFdzIb21oQ3L0MlxOzy1d66FMug/h\nPocCPKqqIzL1USuHtiDM59iHrnD4eU6HNajqpyJysohUwY2409cvBWiqqgcyNO5ew8yf86z6z/Iz\nlZOWHNrLqR3DBzbyyfofozywwTu/xse9HwM9RKQSgIgc45WPBF5V1U9C7pkP9PXqpQKbVXWXiNRR\n1VXqFskX435thtIaNyWVLar6HvAkMCNkzlwy1fkcN2VxBRkNSHq9+UAvb/6+KnAB8Ln32Nu4aavz\nyX4Edq64NZYiOCM1Pwsd3wC1RKSOd30VbjQQyiKghYicDIfXTOp691YXkSZeeVlxi8w7ce9bVnzi\nPV9wr/38bOpl9yVZS0Saeud9cFOvJXFG6Q9v5NojpH5NVZ0L3OtpKgPMwE1h4ek+yzudB1zplTUE\nGmWjISfSdc8ArvV+9CAix3tf4DuBcj7aye59+dorP8krvyKLewFmAQO89yP9/8DPe50V7+GmF1eH\nzCTMBG5LryAiZ2Zz70LcZw8RaQA09Mqz+0zlxCxgYEifFfPYjpEJMz5QSkTWicgv3t/bgcHABBFZ\nDGzO4d706YXVuDn9ud4w/CkRqYlbmL1WjjgSNAaCQBMRWQ48ghuyA9wu3kItbo56Wqa+OpH9F/4R\nQarDcdNbkznyBZmZt4GF3hRC5ufyHm6aYjnwIXB3yKhtJu4X/SxVPZiNhC+A54BVwPeqOjG0fa+P\nfTgjNsF7HdKAlzI+Dd0C9APe9Op8Apzq/erthZt+WeZpKoEzzA2817lnJk23Av29+ldy5Assu1Fa\nZr4GBorIatw6xoveazfCe57T8Ay0iBQFxnialwBDVXUH8BDOcWGFOOeP9F/zL+Kmj1bhPndfZKMh\np9Fb+ns3CxgLfOpNBY4HyqnqVuATr+/Hcrg/y/fFK78RmCrO4WBTFm0A/A/4BVjh/R9cEea9zuk5\nvY17r8aFlN0GnO05BHwFDMjyTngBqOLVeRD3Hm3P7jMVRst/cFPjK73nlBqmHcMn6QuDRoLj/dM3\n9bG+4aetKbhF1BxHUnlotxVwp6p2y0cbK4CLVPXnyCkzChPeqLuYN1VcBzd6OTWHH0xGHLA1nyRB\nVc/ObxsiUgH3C/3LSBueSCAiM4HlZniMfFIamO05xwDcbIYn8bCRj2EYhhFzbM3HMAzDiDlmfAzD\nMIyYY8bHMAzDiDlmfAzDMIyYY8bHMAzDiDlmfAzDMIyYY8bHMAzDiDlR3WQqwiu4dAKbVI/ErBLh\nH7jkVgeBD1S5N5o6DMMwCjsSlBq41B/VcMFoR2hAn81UpxUwCUiPYP6uBvQ/0dAT7QgHI3ERokel\nF4iQClwEnKHKQRGqRFmDYRiG4X7sD9KALpOglAWWSFBmauCozMHzNJD3EFl+ieq0myoLcEmrQrkZ\n+K8qB706W6KpwTAMwwAN6EYN6DLvfBewBjghi6r5SZ/hm3is+dQDWoqwSITZIuQ7ZplhGIbhHwlK\nbeAs4LMsHm4uQVkmQflAgtIgWhriYXyKAseo0gyXfvbtOGgwDMMolHhTbhOA27wRUChLgJoa0LNw\nqVEmZr4/YjqiHVhUhFrAlHSHAxGmAo+pMte7/g5oqsofR98rFvXUMAwjD6jq0Wlyg1IUeB+YpgEd\nGq4NCcqPQBMN6NZI64vFyEfIOIc4EWgDIEI9oFhWhicdVU2oIxAIxF1DMmhKVF2myTQVBl058Cqw\nOjvDI0GpFnJ+LiDRMDwQfVfrsUAqUFmEdUAA9+RHirASlzP96uxbMAzDMCKBBKUFLjvsSgnKl7js\nrf8EagGqAX0Z6CFBuRk4AOzFS0ceDaJqfFTpk81DV0WzX8MwDCMjGtCFQEqYOs8Dz8dCj0U4yCWp\nqanxlnAUiagJElOXafKHafJPoupKdBI6k6mIaCLrMwzDSEREBM3C4SCRsJGPYRiGEXPM+BiGYRgx\nx4yPYRiGEXPCGh8RRIS+Ivzbu64pwrnRl2YYhmEUVPyMfF4AmgNXeNc7iZErnmEYhlEw8bPPp6kq\njUX4EkCVP0UoHmVdhmEYRgHGz8jngAgpuN2wiFAVl4jIMAzDMPKEH+PzLPAecKwIDwMLgEeiqsow\nDMMo0PjaZCpCfaAtLkDoR6qsibYw169tMjUMw8gtybDJNKzxEaFmVuWqrIuKogx9m/ExDMPILQXF\n+KzErfcIUBI4CfhGldOjLs6Mj2EYRq5JBuMT1ttNlTNCr0VoDPw9aooMwzCMAk+uIxyoshRoGgUt\nhmEYRiEh7MhHhEEhl0WAxsCGqCkyDMMwCjx+NpmWCzk/CHwAvBMdOYZhGEZhIOHz+bz6qnLMMRw+\nKlZ0f8uWBUno5TTDMIz4kNQOByJMwYtqkBWqdIuKokzMnQt//umObduOnO/bd8QQpf/NbKCyK6tQ\nAYpYPG/DMIy4ke3IR4RWOd2oytywjQuvAF2BTao0yvTYncATQBVVtmZ9f/au1vv3O2MUapAyG6js\nynbtciOnnAxUdmWVKkFKjlnQDcMw4ksyjHyiOu0mwvnALmBUqPERoQbwP+BUoElejE9+SEuDHTty\nNlBZGbE//4Rjj4XJk+GUUyIuyzAMIyIkg/Hx4+1WF3gUaIDbZAqAKnXC3avKAhFqZfHQM8DdwGT/\nUiNHSsqRkUxuGT4czj8f3nwTWreOvDbDMIzCgJ+Vj5HAizhPt9bAKGBMXjsUoRvwiyor89pGPLnp\nJhg7Fnr3hpdfjrcawzCM5MSP8SmlykeAqPKzKoOBLnnpTIRSwD+BQGhxXtqKJ23awIIF8PTTcNtt\ncPBgvBUZhmEkF372+ewToQiwVoRbgPVA2Tz2dzJQG1guggA1gCUinKvK71ndMHjw4MPnqamppKam\n5rHryFK3LixaBL16QdeuMG6cc04wDMMwwuMnsOg5wBqgIvAQUB54QpVFvjoQagNTMseI8x77EWis\nyp9Z35v4gUUPHoRBg2DWLJgyxRwRDMOIP8ngcOBn2i1NlV2q/KpKf1W658LwjAU+AeqJsE6E/pmq\npEfLTlqKFoVnn4Xbb4cWLeDjj+OtyDAMI/HxM/KZDVQHJgBvqfJVLIS5vhN/5BPK7NnOEeHBB2HA\ngHirMQyjsJIMIx+/mUyrA5cDvXDTbm+p8p8oa0s64wPw3Xdw0UXQvr1zSCjqZ1XNMAwjghQY43O4\nsnAGcA/QS5XiUVN1uL/kMz7gNqj27g2q8NZb5ohgGEZsSQbjE3bNR4TTRBjsZTQdhlvDqRF1ZUlM\nxYrw/vtQvz40awZr18ZbkWEYRmLhZ83nU2AcMF41tnl8knXkE8rLL8MDD7iICG3axFuNYRiFgWQY\n+SR8SoVE1ueX2bPhiisgEICbb463GsMwCjpmfPJJQTE+cMQRoW1bGDLEHBEMw4geZnzySUEyPgDb\nt7uICGlp8PbbeQtsahiGEY5kMD6+U6qJUDqaQgoDFSo4R4TTT3eOCN9+G29FhmEY8cFPSoXzcLl3\nygI1RTgTGKDK36MtriBStKibdjv9dLjgAnjjDWjXLt6qDMMo6EhQauCyElQDDgEjNKDPZlHvWaAT\nsBvopwFdFg09fkY+zwAdgT8AVFkOtIyGmMLEDTe4PUB9+8ILL8RbjWEYhYCDwCAN6OlAc2CgBKV+\naAUJSifgZA1oXWAAMDxaYnxNu6nyS6aitChoKXSkpsLChfDcczBwIBw4EG9FhmEUVDSgG9NHMRrQ\nXbiA0SdkqnYxbnSEBvQzoIIEpVo09PgxPr94U28qQjER7sKJjgl7D+yNVVdx4eST4dNP4ccfoVMn\nl6rbMAwjmkhQagNnAZ9leugEyDDYWM/RBioi+HH4vQkY6glYD8yE2K33jF4xmhub3Bir7uJChQou\nHcPdd0PTpu781FPjrcowjGRhzpw5zJkzx1ddCUpZXKDo27wRUFzwE+GghSoLw5VFAxHR+s/VZ9Xf\nV1FEfDvmJTX/+x/885/OEaF9+3irMQwjGcnO1VqCUhR4H5imAR2axePDgdka0Le866+BVhrQTZHW\n6OcbfZjPsqhQqmgppn83PVbdxZ3rr4fx4+Gqq+D55+OtxjCMAsarwOqsDI/HZOBqAAlKM2BbNAwP\n5DDyEaE5cB5wO87jLZ3ywKWqnBkNQRk1iI5aNorXl7/Oh1d/GO3uEooffnAREVq1gqFDoVixeCsy\nDCNZyGrkI0FpAcwDVuISeSrwT6AWoBrQl716zwEX4lyt+2tAl0ZFYw7GpxWQilvzCXW324lLix31\nWM0iovsO7uOkoScx7cppNKrWKNpdJhTbt7uYcPv3u4gIlSrFW5FhGMlAMkQ48LPmU0uVn2OkJ1Pf\nLrzOo/Mf5dut3zLy4pHxkBFX0tKcI8L775sjgmEY/igoxmc2bniWAVWiniAg3fhs3buVk589mdV/\nX81x5Y6LdrcJySuvwH33wZgx0KFDvNUkJxt2bqB62eqFxnnFKLwkg/Hx8194F3C3dzwALAO+8NO4\nCK+IsEmEFSFlj4uwRoRlIrwjQvlw7VQqVYkrGl7BC4sLbyiA666DCRPg6qvdptQCFG81JixYt4B6\nw+ox7qtx8ZZiGAZ5jGotwueqnOuj3vnALmCUKo28snbAx6ocEuG/gKpyX9b3H4lq/e0f33L+q+fz\n8+0/U6pYqVxrLiikOyK0bAnPPmuOCH749JdPuXjcxVxU7yK2/rWV93q9F29JhhFVCsTIR4RKIUcV\nEToCFfw0rsoC4M9MZR+qcsi7XITPlNz1KtejWY1mjF4x2k/1AkudOi4iwrp1cOGFsHVrvBUlNp+v\n/5yLx13MqEtH8WSHJ/noh4/YuW9nvGUZRqHHz7TbEtw02xLgU+BO4LoI9X8tMM1v5UHNB/HMomc4\npIfCVy7AlC8PkyfDWWe5iAhffx1vRYnJkg1LuOjNi3j14le58JQLOabUMbSo2YIP1n4Qb2mGUegJ\nG15HlZOi0bEI/wIOqDI2p3qDBw8+fN6qVStKFi3J9O+m07lu52jIShpSUuCpp6BBAzcF99Zb0Lp1\nvFUlDss2LqPL2C683PVlutbreri8Z4OeTFg9gd4Ne8dRnWEYOe3zuSynG1V511cHQi3cvqBGIWX9\ngBuANqrsy/7eozOZjl4+mlErRjHrqll+ui8UfPSR2w80ZAj06RNvNfFn5aaVdBjTgec6PUf3Bt0z\nPPbHnj+o82wdNgzaQJniZeKk0DCiS7Kv+VyUw9E1h/syI97hLoQLcZ5z3XIyPNnRq2EvVm9ezYpN\nK8JXLiS0besM0L33wqOPFm5PuNWbV9NxTEeGdBxylOEBqFy6Ms1qNGPq2qlxUGcYRjp58nbz3bgw\nFhcloTKwCQjgwjkUx0tOByzKLitqViMfoFBvOs2J9euhSxeXovu551zW1MLE11u+pu2otjze7nGu\nbHRltvVGLBnBrB9m8XbPt2OozjBiRzKMfPxsMq2AMxrp2UvnAg+qsj3K2rI1PumbTtcMXEP1stWj\nLSOp2LEDevZ0huett6Bs2Xgrig1r/1hL69db83Cbh7nmrGtyrLt592ZOGXYKv935G6WLlY6RQsOI\nHclgfPx4u72Ki+d2uXfsAOI65LBNp9lTvrwLxVOtmsuUunFjvBVFn++3fk/bUW0JpgbDGh6AqmWq\ncs7x5xSqaOmGkWj4MT4nqxJQ5QfvCAJ1oi0sHLc3u53hXwwv8JlO80KxYi4cT7du0Lw5rIlZ3tnY\n89O2n2g7qi3/uuBfXNfY/w6AHg16MH71+CgqMwwjJ/wYn71epALAJZID4v6Nb5tOc0YE/v1vGDzY\njYDmzYu3osizbvs62rzehrvPu5sBZw/I1b2X1r+UaWun2Y8Xw4gTfozPzcDzIvwkws/Ac7g0C3Hn\njmZ32KbTMFxzjQtG2qMHjCtAYc1+3fErbV5vw61Nb2XguQNzfX+1stX423F/Y+b3M6OgzjCMcIQ1\nPqos8xLHNQLOUOVvqiyPvrTwpNZOPbzp1Mie9u3hww/hnnvg8ceT3xV7w84NtHm9DTedfRO3N7s9\nz+30OM2m3gwjXviJ7XabF3l6J/C0CEtFSIig/iLCoGYu5I6RM40awSefuFHQwIFw8GC8FeWNjbs2\n0nZUW/qf1Z+7zrsrX21ddtplfLD2A/YdzPV2M8Mw8omfabdrVdkBdMDt17kK+G9UVeUC23Tqnxo1\nYP58WLsWLrsMdu+Ot6Lc8fvu32k7qi19GvbhvguyDISeK44rdxxnHHuGTb0ZRhzwY3zSfcU741Ij\nrAopizvFU4oz8JyBNvrxSYUK8MEHLiV369awaVO8Ffljy54ttBvVju6ndeeBVg9ErN2eDXoyYc2E\niLVnGIY/fEW1FmEmzvjMEKEckFAr/AOaDGDi1xPZuKsQbGqJAMWLw8iR0KmTc8X+5pt4K8qZrXu3\n0n50e7rW60owNRjRti877TKmfDOF/Wn7I9quYRg548f4XAfcC5yjyh5caJz+UVWVSyqXrmybTnOJ\nCASDcP/9Lir2ggXxVpQ12/7aRofRHWh3UjsebvMwIpEddJ9Q/gROq3oaH/7wYUTbNQwjZ/x4ux0C\nagP/FuEpoKUqCbfAYptO88a118KoUW4NaHyCOX5t/2s7Hcd05Pya5/N4+8cjbnjSSU+zYBhG7PDj\n7fYCbl/PSuArYIAIz0dbWG6pV7keTWs0tU2neaBjR5g5EwYNcjmCEsEVe+e+nXR6oxPnHH8Oz3R8\nJmqGB9zU26RvJnEg7UDU+jAMIyN+pt3aAB1VGanKSNzaT9voysob6W7Xtuk095x1lnPFfu01uO02\nSEuLn5Zd+3fReWxnGlVrxLBOw6JqeABqVqhJ3Up1+fjHj6Paj2EYR/BjfL4DaoZcnwisjY6c/JG+\n6XTGdzPiLSUpOfFE54q9apWLiLBnT+w17N6/m65ju3Jq5VN5ocsLUTc86djUm2HElmyNjwhTRJgM\nlAPWiDBHhNnAGq8s4UjfdPr0oqfjLSVpqVgRpk2DcuWgTRvYvDl2fe89sJdu47pRu2JtXr7oZYqI\nn99GkaF7g+5M/GaiTb0ZRozIKY12q5xuVGVuVBRl0JB1Pp+c2J+2n9pDajO973QaVWsU/gYjS1Rd\nYNI334SpU6Fevej299fBv7h43MVULV2V1y95nZQiKdHtMAvOHXEuj7R9hHZ12sW8b8OIJMmQzyeq\nmUzzS16MD8Aj8x9h7da1luk0AowYAQ88AO++C+edF50+9h3cx6VvXUr5EuUZc9kYihaJTwrWxxc+\nzg9//sDwrsPj0r9hRIoCYXxEaAYMA07D7fFJAXarUj7q4vJofP7Y8wenDDvFMp1GiGnTXHTsF1+E\n7t0j2/b+tP30eLsHxVOK82b3NymWUiyyHeSCH/78geavNGf9oPVxM4CGEQliaXwkKI1w23EO/9No\nQN8Nd5+fSfXngCtwTgalgOsh8VytQ7FNp5GlUyeYMcN5wQ0ZErl2D6QdoPeE3hSRInE3PAB1jqlD\njfI1mP/z/LjqMIxkQYLyKi7bdXfgIu/o6uteHyOfL1Q5W4QVqjTyyr5U5W9hGxde8YRsCrn3GOAt\noBbwE3C5Ktuzvj9vIx+Ab7Z8wwUjL+Dn23+mVLFSeWrDyMjPP0Pnzi5Fw1NPQUo+lmUOHjpIn3f6\nsPfgXt65/B2KpxSPnNB88Oj8R/l1x6883yWhf18ZRo7EauQjQVmtAW2Ql3v9jHz2iFAcWCbC4yLc\n4fM+gJFAx0xl9wIfqnIq8DGQ//DEWXBqlVNt02mEqVXLheFZvhwuvxz25jGYxMFDB7nqvavYuX8n\nE3pOSBjDAy699jtr3iHtUBw3OhlG8vC5BCVqxucqr94twG7cPh9fM/+qLAD+zFR8MfC6d/46cIkv\npXlgULNBDFk0xDadRpBjjoHp06FkSWjbFrZsyd39aYfS6D+pP1v2bOHdy9+lRNES0RGaR+pWrkv1\nstVZ+MvCeEsxjGTgNWCRBOUbCcoKCcpKCYqv8Gt+Yrv9rMpfquxQJajKIFW+y4fYY1XZ5LW9ETg2\nH23lSGrtVEoULWGbTiNMiRIwejSkpjoPuO98fhoO6SGun3I963esZ1LvSQk7HdqjQQ/Gr0qwQHeG\nkZj8D+gLXMiR9Z6L/NyYCC49OS7qDB48+PB5amoqqampvhsWEe5odgdPL3qaTnU75VWfkQVFisAj\nj7ipuAsugIkToWnT7Osf0kMMmDKAH/78gal9plK6WOnYic0lPRv0pPXrrRnaaWhMN7oaRhKyWQM6\nOS83Rn2fjwi1gCkhDgdrgFRVNolQHZitymlZ35t3h4N0bNNp9Jk6Ffr1g5dfhkuymERVVQZOHcjy\nTcuZfuV0ypVIyAAZGTjjxTMY3mU4LWq2iLcUw8g1MXQ4eAGoCEwBDuejj4irtQg9/ZTl1AQZM59O\nBvp559cAk3LRVq4pnlKcW869xTKdRpHOnd1eoIEDYdiwjI+pKrdNv42lvy1l2pXTksLwAPQ4rQfj\nV9vUm1FwkKC8IkHZlN2ajASllQRlmwRlqXfc76PZUjij04EouFovVaVxuLJs7h0LpAKVgU1AAJgI\njMc5LvyMc7XelvX9+R/5gG06jRU//eT2BHXuDE88ASLKnTPvZP66+Xx41YdUKFkh3hJ9s3rzajqO\n6cjPt/9sU29G0pHVyEeCcj6wCxilAT1qGkiC0gq4UwPazVcfQUkBbtWA5umXfU6BRTuJMAw4QYRn\nQ47XgIN+GleljyrHq1JClZpeWoY/VWmnyqmqdMjO8ESSyqUr0/v03rbpNMrUru3SMnzxBfTqrdwz\n617m/DTG+/ywAAAgAElEQVSHmX1nJpXhAWhQtQHlipfj8/Wfx1uKYUQEDWhW3seZ8T1VpwFNwwUg\nyBM5ORxsAL4AugFLQsp3AnfktcN4cXuz27lg5AXcd/59CetlVRA45hiYMUM59e8PsHDRdFbe9THH\nlDom3rLyRM8GPRm/ajzNajSLtxTDiBXNJSjLgPXA3RrQ1WHqL5SgPIcLHLA7vVADujRcR9kaH1WW\nA8tFeA8Xyy0NQIQUILE2Z/ggdNPpjU1ujLecAouqMnjBfZQ6cyo7n/iIzZdXpnL9eKvKGz0a9KDr\nm115ssOTMcsrZBh5Yc6cOcyZMye/zSwBampA90hQOuGWSMLFsz/L+/tgSJnikpDmiJ81n0VAO1V2\neddlgZmqRCnGcWjfkVnzSWf2j7MZOHUgX/39K5vHjwKqyh0z7mD+uvnM7DuTcSMrM2aMi4qQn1A8\n8UJVqf98fUZfOppzTzg33nIMwzfZebtJUJz3cRZrPlnU/RFoogHdGg2Nfr6BS6YbHgDvPHE3aeRA\nau1UiqcUt02nUeCQHmLg1IEs+nURH139EZVLV+bmm92G1KFD460ub4iIZTg1ChqZvY+PPBCUaiHn\n5wISzvBIUCpIUJ6WoHzhHU9JUHwt8PoxPrtFjni2idAEyGNUr/giIgxqbplOI03aoTRumHwDK39f\nycyrZlKxZEXAbUR95RW3GfXbb+MsMo/0aNCDCasnkMh5rwzDDxKUscAnQD0JyjoJSn8JygAJSvo6\nRA8JylcSlC+BIUAvH82+ivMDuNw7duBieobX42Pa7RxgHM4BQYDqQC/VDE4IUSHS025gm04jzcFD\nB+k3sR8bdm5g8hWTKVu87FF1nn0W3n4b5s5Nvuk3VaXec/UY130cTY5vEm85huGLGG4yXaYBPStc\nWVb4ie22GKgP3AzcBJwWC8MTLdI3nQ5ZFMHENIWUA2kH6PNOHzbv2cz7fd7P0vAA3HILiMBzz8VY\nYAQQEXqc1sOm3gwja/Z6+4cAkKC0wOfMmJ+Rz9VZlasyKjcK80I0Rj5gm04jwb6D++g1oRdpmsb4\nnuMpWbRkjvXXroXmzWHRIjjllBiJjBBLNiyh9zu9+faWb83rzUgKYjjyORMYBVTAzYxtBfppQJeH\nvdeH8QkNmFISaAssVaVHnhX7JFrGB+Dm92+mapmqPNj6wfCVjQzsPbCX7m93p1SxUrzZ/U3f+Xie\necYFIJ09260HJQuqysnPnsy7vd7lrOphZxMMI+7EMo02gASlPIAGdIfve3L75S5CRWCcKhfmTl7u\niabxsUyneWP3/t1c8tYlVCldhdGXjqZoEf+B0dPSXATsPn3cVFwycc+seyieUpz/tPlPvKUYRlhi\nOPIpgcvvVpuQfaMa0LC/6vPy+3M3cFIe7kso0jedjlkxJt5Skoad+3bSeWxnTih3AmMuHZMrwwPO\n2WDkSBg8GH74IToao0XPBj0Zv3q8eb0ZRkYm4RKEHsTZhvQjLGG/PUSYwpGcO0WABsDbeZKZYNzR\n7A5umXoL1zW+zjadhmH7X9vp9EYnGh7bkOFdh+f59Tr1VPi//4PrroOPPkqe6bezjz+bvw7+xVe/\nf8UZ1c6ItxzDSBRqaEDzNAvm51//SeAp73gUaKnKvXnpLNFoXbu1bTr1wda9W2k3uh1NjmvCS11f\nyrehHjQI9u6Fl16KkMAYYF5vhpEln0hQ8vRrLOrJ5PJDNNd80hm1fBSjV4xm1lWzotpPsrJ592ba\nj25PuzrteKL9ExHz9lqzBlq2hMWLXTTsZGDRr4u4dtK1rB4YLtaiYcSXGK75rAZOAX7E5fURQP2E\n78kppcJOEXZkd0RMfZzp3bA3q35fxYpNWeZXKtRs3LWR1NdT6Vqva0QND8Bpp8Gdd8L110MC//7J\nwLknnMvO/TtZvdmMj2F4dALqciSZXFfvb1j8uFo/BPwGjMZZtSuB41T5dz4E+yIWIx+Ah+c9zPd/\nfs+rF78a9b6ShfU71tNmVBv6ntGXB1o9EJU+Dh50e39uuAFuTJJA47dPv51KpSrx71ZR//gbRp6J\ntat1XvBjfJarcma4smgQK+OzZc8W6g6ra5tOPX7e9jNtRrVhQJMB3NPinqj29dVX0Lo1LFkCNWtG\ntauIsHDdQm764CZW3rwy3lIMI1uSwfj4DSx6pQgpIhQR4Up8utIlC1VKV7FMpx7fb/2eVq+14ram\nt0Xd8AA0bAi33+5GP8kw/db8xOZs3buVr7d8HW8phpHU+DE+fXDRSjcBvwM9vbICxe3Nbmf4F8PZ\neyApA3ZHhG+2fEPq66ncd/593Nr01pj1e889sHmz2wOU6BSRInQ/rbt5vRlGPombt5sI9wF9gTRg\nJdBflf0Z68Rm2i2drmO7cvGpF3NDkxti1meisOr3VXQY04H/tP4P/f/WP+b9r1gBbdvCl19CjRox\n7z5XzPt5HrdOu5VlNy2LtxTDyJJoT7tJUHZyZP9nhodw3m7lw7bhY82nBjAMaOEVzQduU+XX3MnN\n0GYtYDZQX5X9IrwFfJA5WGmsjc/HP37MLVNvKXSZTpdtXMaFYy7kqQ5PcWWjK+Om48EHXeDRDz5w\nUbATlbRDadR4pgbz+s2jbuW68ZZjGEdRUNZ8RgKTgeO9Ywo+kwXlwA5gP1BGhKK4zKgb8tlmvimM\nm06/2PAFHcd0ZFinYXE1PAD33QcbNsCoqMdLzx8pRVK4rP5lNvVmGB4SlGMlKDXTDz/3+DE+VVUZ\nqcpB73gNqJofoar8iYuYsA5YD2xT5cP8tBkJClum009/+ZTOb3RmxEUj6Hl6z3jLoVgxeO01uPtu\nZ4QSmR4NejBhjRkfo3AjQekmQVmL22Q6F/gJmObnXj/G5w8R+nrebiki9AX+yLNaQIQ6wB1ALdxo\nqqxI1k4MgwcPPnzMmTMnP936In3T6cpNBduVdu5Pc+k2rhujLh1Ft1O7xVvOYc46C266CQYMSGzv\nt5a1WvLrjl/54c8ki5BqGJHlIaAZ8K0G9CRcyp1Ffm70s+ZTC7fm0xy3wPQJcKsq6/KqVoTLgfaq\n3OBdXwU0VeWWjPViu+aTTkHfdPrhDx9yxTtXMK77ONrWaRtvOUexfz+cfbbzguvbN95qsuem92+i\nzjF1YuKSbhi5IYbhdb7QgJ4tQVkO/E0DekiCslwDGnYfaI4jHxFSgMtU6aZKVVWOVeWS/Bgej2+A\nZiKUFEFw1nJNPtuMGAPOHsB7X7/Hxl0b4y0l4kxdO5U+7/ThncvfSUjDA1C8uHO7HjQIfvst3mqy\np0cDCzRqFHq2SVDKAvOANyQoQ/G5DzRH46NKGnBF/vUd1e5yXOrVJcBynHvey5HuJ68U1E2nk76e\nRL+J/ZjUexIta7WMt5wcadLEbTy9+ebEnX5LrZ3Kj9t+5KdtP8VbimHEi4uBvbhllOnA90Qwttsz\nQDHgLUIsmipL8yjWN/GadgO34bLlay356bafCkSm0/GrxnPLtFv4oM8HnH382fGW44t9+6BxY7j/\nfrgi4j+BIsMNk2+gfpX63HnenfGWYhiHSQZXaz/GZ3YWxapKm+hICu07fsYHCs6m0zErxnD3rLuZ\nfuV0zqwe9ZB8EWXxYuja1W1CrVYt3mqOZsZ3Mxg8dzCfXvdpvKUYxmFiuOYTutm0OG6gsjsim0zj\nSbyNT0HYdPrql6/ywOwHmNl3Jqcfe3q85eSJe++FtWthwoTE23x6IO0Axz11HF8O+JITK5wYbzmG\nAcRn5CNBEdw0XDMNaNiEo2G/UUWoLMKzIiwVYYkIQ0WoHAmxiU6ybzp9cfGLBOYEmH3N7KQ1PACD\nB8Pq1TB+fLyVHE2xlGJ0O7Ub76x5J95SDCOuaEBVAzoR6Oinvp9pt1k4T4YxXtGVQKoq7fIj1A/x\nHvkAvL7sdd5Y+QYzr5oZVx25ZciiIQxZNISPrv6IkyudHG85+WbRIrjkEli5Eqrma4tz5Jm2dhoP\nz3+YBdcuiLcUwwBiOu12WchlEeBsoJUGtHm4e/3MJR2nykOq/Ogd/wEScPY9OvRu2Juvfv8qqTad\nPrbgMYZ9Poy5/eYWCMMD0KwZXHUV3HJL+Lqxpm2dtqzevJr1O9bHW4phxJqLQo6OwE7c1FtY/Ix8\nngY+B972inoA56pyV17V+iURRj6QPJtOVZWH5j3EGyvf4OOrP+aE8ifEW1JE2bvXRUB45BHo3j3e\najJyzcRrOPu4s/lH03/EW4phFBhvt51AGVzqA4AUjrhcqyphvRryLC5BjE8yZDpVVe7/+H4mfjOR\nj67+KGF15peFC6FHDzf9VqVKvNUc4f1v3+fxhY8zr/+8eEsxjFikVBhG1ikVANCAhk0IFnbaTZVy\nqhRRpZh3FPHKykXT8CQSVUpXodfpvRJ206mqctfMu/hg7QfMuWZOgTU8AC1auD0/t8Yu150v2tdp\nz8rfV/LbzgQOyWAYkeMLXJCAkkBjYK13nIVzuQ6LuVr75OstX9PqtVYx3XSqqqRpGvsO7mN/2n72\np+1nX9qR8/1p+9l3cB+vL3+dxRsWM6PvDCqVqhQTbfFkzx4480x44gnnhJAoXPXeVTSv0Zy/n/P3\neEsxCjkxdDhYBJyvAT3oXRcD5mtAm4W7t2i0xRUU6lepzznHn8P9H99P4+MaZ2sIsjUSuakb8lgR\nKULxlOIUTylOiaIlDp8XTylOiRR3XbtibT686kMqlKwQ75cpJpQuDa++Cr16QcuWUClB7G2P03ow\n5LMhZnyMwsQxQHlgq3dd1isLi418csHyjct5eP7DFEsp5gxAkewNQjiDkdXjmR8rnlKclCIp8X7a\nCcutt8Kff8Lo0fFW4th7YC/HPXUc39zyDdXKFhqHUCMBieHIpz8wGJeZWoCWwGAN6Oth7/XhcDBa\nlavClUWDRDM+RmKxezc0agRDhsBFvkIZRp8+7/ShVa1WDDh7QLylGIWYWHq7SVCqA029y880oL7S\nAfjZ55Nha7yXZqFJ7uQZRuQpUwZeecUln/vzz3ircfRo0IPxqxMwFINhRBAJSn3vb2NcQtBfvON4\nryws2a75iHAf8E+glAg7cEMqgP0kUPoDo3CTmuqcDgYNcjmA4k2nUzpx7aRr2bx7M1XLJFgoBqNQ\nI0F5BegKbNKANsqmzrNAJ9x2mn4a0GXZNDcIuBF4KovHFMIHnvYz7faoKveFayga2LSb4Yddu+CM\nM+D556Fz53irgV4TetHupHZJHw3dSF6ymnaToJwP7AJGZWV8JCidgFs0oF0kKE2BoX681vKKn30+\n94nQTYQnvaNrtMQYRl4oWxb+9z8YMAC2b4+3Guf1NmGNZTg1EgsN6AIgpwnqi3FJPtGAfgZUkKDk\n6DkjQekpQSnnnd8vQXlXgvI3P3r8RLV+FLgNWO0dt4nwiJ/GDSNWtG0LXbrAnQmQ061z3c4s+nUR\nf+z5I95SDCM3nIBbt0lnvVeWEw9oQHd6o6p2wCvAcD+d+XE46AK0V+VVVV4FLgQb/RiJx+OPw6xZ\nMCPOGTDKFC9D+zrtmfTNpPgKMYzokx52rQvwsgb0A3xGOPC7ybQiRzYRFY6djEbSUb48jBgBN9zg\nYr+Vj2Pwpx4NejBq+Siu/du18RNhFBrmzJnDnDlz8tvMeiA0I2INryzHeyQoLwHtgcckKCXwN6jx\n5XBwBfBfMm4iuleVt/x0kEO7FYD/AQ2BQ8C1qnyWsY45HBi554YboEgReOml+GnYuW8nNZ6pwU+3\n/cQxpXxt+DaMiJHdPh8JSm1gigb0jCwe6wwM9BwOmgFDwjkcSFBK42bDVmpA10pQjgPO0ICGTYDm\nK8KBCMcB53iXn6viaxNRmDZfA+aqMlKEokBpVXZkrGPGx8g927dDw4bO9bpd1FMeZs+lb13KJade\nwjVnXRM/EUahJBtvt7FAKlAZ2AQEcFNkqgF92avzHM6Y7Ab6a0CXhu3LrffU1YCOlKBUBcpqQH8M\ne192X+4i5LhRSJWworLtVCgPfKlKjpnOzPgYeWXaNLj5Zjf9Vq5cfDSMXTmWN796kylXTImPAKPQ\nEsPwOgFc9tJTNaD1JCjHA+M1oC3C3ZvT3NxT3vE88BluY+kI7/z5fGo+CdgiwkgRlorwsgixCRVt\nFAo6dYLWreH//i9+GrrW68rcn+ay/a8E8P82jOhwKdANL8ebBnQD4OvnXrbGR5XWqrQGfgMaq3K2\nKk2AvxF+ESocRXE5IJ5XpTGwB7g3q4qDBw8+fERgQc0oRDz9NEyeDLNnx6f/8iXKk1o7lSnf2sjH\nKLDs14AqXmI5CUoZvzf6cThYpXpUfLejynKDCNWAT1Wp412fD/yfKhdlrGfTbkb++OAD+Mc/YMUK\ntxk11oxePpp31rzDxN4TY9+5UWiJ4bTbXUBdnLfbo8C1wJsa0GfD3uvD+LyJG1KN8YquBMqqckW+\nRAtzgRtU+VaEAM7h4P8y1jHjY+Sfq6+GChVg2LDY973tr23UGlKLX+74hfIlCkXiXyMBiHFU6/ZA\nB5w39AwN6Cxf9/kwPiWBm3Eu1gDzgBdV+SvvckGEM3Gu1sWAH4D+qmzPWMeMj5F/tm51sd/GjoVW\nrWLff9exXbnyjCu54ox8/V4zDN/E0vgc1XdQemlAw27FsWRyRqFg0iQXemf5cpeKIZa8tuw1pnw7\nhXcufye2HRuFlmgbH29tZwBwMrAKF1KnG/AIsFYDenHYNhL5y92MjxFJrr8efv3VGaISJWLX79a9\nWzlp6EmsH7SessXjsPBkFDpiYHzeAXYCn+LWe04E/gJuyyENQwZ8hUEwjILA8OFQujRccQUcPBi7\nfiuVqsR5J57H1LVTY9epYUSXuhrQfhrQl4BeuO0zHf0aHghjfERIEeHJfIo0jISgaFF4803Yuxeu\nvRYOHYpd3z1OswynRoHi8M83DWga8KsGNFd+AH4cDhapErWEQjn3bdNuRuTZs8dtQm3QAF54ASQG\ny7J/7PmDOs/WYcOgDZQpHuNFJ6PQEYNptzS8jaU4L7dSuP2aggvXE9a104/xeRGX02F8SGeo8m7e\nZPvHjI8RLXbscHHfWrVyqRhiYYA6jO7AgCYD6N6ge/Q7Mwo18fR284ufNZ+SwB+4nNwXeYfl8zGS\nmvLlXfy36dPhoYdi02ePBjb1ZhjpmLebUajZuBFatnRBSO+4I7p9bd69mbrD6vLbnb9RqpiFMjSi\nR4EY+YhQQ4T3RPjdO94RoUYsxBlGtKleHT78EIYOdYnooknVMlVpcnwTZnwf51SrhpEA+Jl2GwlM\nBo73jilemWEUCGrWdAZo8GAXBSGamNebYTj8OBwsU+WscGXRwKbdjFjy1VfOCWH4cLjkkuj0sWnX\nJuo/X5/f7vyNkkVLRqcTo9BTIKbdgD9E6Ovt+UkRoS/OAcEwChQNG7oo2DfeCLN8hUbMPdXKVuPM\namcy8/uwWYYNo0Djx/hcC1wObMTl9ukB9I+mKMOIF02awLvvQp8+sGBBdPro2aAnE1ZPiE7jhpEk\n5DjtJkIKcKsqz8ROUmj/Nu1mxIdZs+DKK507dpMmkW17w84NNHyhIb/d+RslisYwyJxRaEj6aTdV\n0iB/eXsMIxlp3955v3XpAqtWRbbt48sdz+nHns6HP3wY2YYNI4nwM+22UITnRLhAhMbpR9SVGUac\nufhil4q7Qwf47rvItt2zQU8mrLGpN6Pw4sfbbXYWxapKm+hICu3bpt2M+DNiBDz8MMyb59yyI8Gv\nO37lzOFn8tudv1E8pXhkGjUMj2SYdiua04MiFMFlLX07RnoMI+G44QbYtcu5Yc+b5zam5pca5WtQ\nv0p9Pv7xYy485cL8N2gYSUa4NZ9DwD0x0mIYCcsdd0Dfvm4KbuvWyLTZ47Qe5vVmFFr8TLv9F9gC\nvEXGqNYR+hfMqW+bdjMSB1W45x6YO9dFRCgfNmh8zqzbvo7GLzXmtzt/o1hKsciINAySY9rNj8NB\nL2AgMA9Y4h1fRKJzEYqIsFSEyZFozzCiiYhLv9CkCVx0kcsLlB9qVqjJKZVOMa83o1AS1vioclIW\nR50I9X8bsDpCbRlG1BGB55+HWrXgsstg3778tXdPi3voN6kfM76zYKNG4cJPVOvSItwvwsvedV2R\n/Ofz8SJjdwb+l9+2DCOWFCkCr74KZcrAFVfAwYPh78mOy067jPE9x9NvUj+e+uQpbJrZKCz4jWq9\nHzjPu14P/CcCfT8D3A3Yf5uRdBQt6iJg790L114Lhw7lva2WtVry2fWf8cbKN7h64tXsPbA3ckIN\nI0HxY3xOVuVx4ACA6uE83XlGhC7AJlWWeW1l297gwYMPH3PmzMlPt4YRUUqUgHfegZ9/hoEDnUNC\nXqlZoSYLrl3AwUMHaflaS37d8WvkhBpGAuLH2+0ToC2wUJXGIpwMvKnKuXnuVHgE6AscBEoB5YB3\nVbk6Yz3zdjMSnx073B6gVq2cQ4Lk46eZqvL4wsd59vNnGd9zPOedeF74mwwjE8ng7ebH+LQH7gca\nADOBFkA/VeZERIDQCrhTlW5HP2bGx0gOtm6F1FTo0QP+/e/8tzd17VT6TezHo20f5brG1+W/QaNQ\nUSCMD4AIlYFmuOmxRapsiZgAMz5GAWHTJrjgArj5ZrcpNb98s+Ubuo3rRoc6HXi649O2F8jwTdIb\nHxFuU2WoCP9QZVgMdXn9m/Exkot166BlS/jXv1xYnvyy7a9tXPnulew5sIfxPcdTpXSV/DdqFHiS\nwfiEczjYJcLdhEQ2MAwje2rWdNEPBg923nD5pWLJikzuPZlmJzTjnBHnsHzj8vw3ahgJQLYjHxEC\nQGncRtChwG5VHoyhNhv5GEnLV185J4SXXnKpGSLBmyvf5Nbpt/Jilxfp0aBHZBo1CiTJMPIJN+32\nH2AG0FGV+2Om6nD/ZnyM5GXJEujUCd54wyWniwRLf1vKpW9dytWNribYOkgR8bNbwihsZGV8JCgX\nAkNwM16vaEAfy/R4K2AS8INX9K4GNBJ7OrPWGMb4dFBlpgjtVIl5ACozPkays2ABXHopvPcenH9+\nZNr8fffvdH+7O5VKVWL0paMpXyKfEU6NAkdm4yNBKQJ8i9s2swFYDPTWgH4dUsc5fwX0KOevaJDt\nzyYvW+kW7+/W0CymlsnUMPxx/vlu7eeyy9xIKBIcW+ZYPrr6I44rexzNX2nOd1sjnGbVKIicC6zV\ngP6sAT0AjAOymhCO2VRdTsnknsrhMYXoZzI1jIJA+/YuG2qXLvDRR3D66flvs3hKcYZ3Hc7wL4bT\n4tUWjL50NB1O7pD/ho2CygnALyHXv0KWgQKaS1CW4cKo3a0BjVrg52yNjyqto9WpYRQ2Lr4Ydu92\nyejmzoVTTolMuzedfRMNqjag94Te3HXeXdzR7A4kPyEWjKRkzpw5kQg/tgSoqQHdI0HpBEwE6uW3\n0ezwu8m0IS7CQcn0MlVGRUvUkX5tzccoWIwYAY884gxQzZqRa3fd9nVcMu4SGh7bkJe6vkSpYqUi\n17iRdGSx5tMMGKwBvdC7vhfQzE4HGdoIyo9AEw1oVBKH+kmpEACGeUdr4HE4OhqBYRjhueEGuPVW\n54a9cWPk2k0PTHrg0AELTGpkxWLgFAlKLQlKcaA3ZEziKUGpFnJ+LiDRMjzgL6p1D5yHxEZV+gNn\nAhWiJcgwCjp33AF9+7opuK0R/NcuXaw0Yy8bS4/TetD0f0355JdPIte4kdRoQNOAW3DxOVcB4zSg\nayQoAyQoN3rVekhQvpKgfIlzye4VTU1+Aot+rsq5IizBjXx2AmtUqR9NYa5vm3YzCiaqcM89bvpt\n+nSoVCmy7Vtg0sJNMmwy9TPy+UKEisAI3ILUUuDTqKoyjAKOiEu/cMEFLiV3q1bw8MPwxRf5S0yX\nTue6nZnffz6Pf/I4/5j6Dw6kHch/o4YRQXw5HByuLNQGyquyImqKMvRnIx+j4LN7txsBzZwJM2bA\nli1uTahjR+emfcIJeW/bApMWTpJh5ONn2q1lVuWqzIuKogx9m/ExCh/r1jlDNHOmC1J6wglufahj\nRzdSKpVLR7a0Q2nc//H9jFs1jom9JnJm9TOjI9xIGAqK8ZkSclkStzFpiWr0N5ma8TEKO2lpbipu\nxgxnjJYvh/POO2KMTj/df+ZUC0xaeCgQxueoG4QTgSGqdI+OpNC+zPgYRijbtsHs2c4YzZgB+/c7\nQ9Shg5uiqxJmVs0CkxYOCqrxEWCVKg2iIym0LzM+hpEdqvDdd0fWiubOhbp13YioY0do1gyKFz/6\nPgtMWvApEMZHhGG4WG7gvOPOAn5SpW+UtZnxMYxcsH8/fPrpEWO0di2kph6Zojv55CNTdPvT9nPb\ntNuYt24ek3pP4pRKEYr3YyQEBcX4XBNyeRBneBZGVdXhvs34GEZe2bzZOSykG6OSJZ0R6tAB2rSB\nChVg+BfDCcwJJFRg0v1p+/l99+9s3LWRTbs2ub+7N7Fz306qlK5CtbLVqFam2uG/VUpXIaVISrxl\nJxQFwvhEpVOhBjAKqAYcAkao8uzR9cz4GEYkUIVVq444LnzyCZx5pjNGx549n+DqXlENTHog7QC/\n7/6dTbs3HWVUMpft3L+TY8scS7Uy1ahetvphI1O+RHm27Nly5L5d7t5tf22jUqlKGQxShnPvb/Wy\n1alapipFi+QUzL9gkNTGR4SVHJluOwpVGuW5U6E6UF2VZSKUxW1evViVrzPWM+NjGNFg716YP/+I\nMfplxzrkiks4rXJDXu/xEnVPCu/PffDQQTbv3nzYGKQbkMPnIWXb922naumqGQxBumFIP083NJVK\nVcqVI0S6jlCDdPhvprI/9v5BxZIVwxqqamWrcWyZYymeksWiWRKQ7Manlnc60Ps72vvbF1BV7o2Y\nCGEiMEyVjzKWm/ExjFiwfj28P2MPj6y6jvV7v6P6l0Op23Anx9fdRIUTNlK04ia27stoVLb9tY3K\npSpnGJ1kNiTp55VLV04Iz7q0Q2ls2bPFl6HavGcz5YqXy2AgMxuomhVq0qhann+HR42kNj6HKwhf\nqq5eWWEAABB6SURBVPK3TGVLVSOTzdSLmjAHaKjKroyPmfExjFiiqjy+8AlGfjGWovuO5cCf1di+\noRp/rKvOceWqcXqt6pxzWjVaNq7G+Y2rULJEwV1rOaSH2Lp3awaDFDpVuGnXJr76/Sua1mjK852f\n59gyx8Zb8mEKivFZBgxMdzIQ4TzgBVXOynfnbsptDvCQKpOOflw0EAgcvk5NTSU1NTW/3RqGkUv2\n7XMbXBcvdsfnn7tIDI0awTnnHDnq1oUi8R/gxIy/Dv7F4DmDeW3ZazzT8Rl6N+ydEMn8CorxaQK8\nikujIMCfwLWqLM1Xx0JR4H1gmipDs65jIx/DSFR27IClSzMapG3b4Oyzjxijc8914YES4Ps4qny+\n/nP6T+pPvcr1eLHLi1QvWz2uegqE8TlcUVwOH1W2R6RjYRSwRZVB2dcx42MYycTvv7twQKEGKSUl\n4+jonHOgcuV4K408+w7u46F5DzFi6QiebP8kfRv1jdsoKKmNjwh9VRkjkrVxUOXpPHcqtADmwWGP\nOgX+qcr0jPXM+BhGMqPqpufSjdHixbBkiQsDFGqMGjeGsmXjrTYyLNmwhP6T+lOrYi2GdxnOCeXz\nEZY8jyS78RmgykteGu2jUCUYVWWY8TGMgsihQ/DNNxkN0sqVUKdOxum6M87IOjxQMrA/bT+PzH+E\nFxa/wGPtHqPfWf1iOgpKauOTCJjxMYzCwf79zgCFTtf98AM0bHjEIDVv7hwakmn9aPnG5fSf1J9q\nZavxcteXObHCiTHpt0AYHxEeB/4D7AWmA42AO1QZE3VxZnwMo9Cyaxd8+eURg7RgAZQoAV26QNeu\n0LKlu050DqQd4LGFjzH0s6E80uYRrm98fdRHQQXF+CxT5SwRLgW6AoOAeapEPSOVGR/DMNJRdaOj\n9993x+rV0LatM0SdO0O1avFWmDNf/f4V/Sf1p2LJioy4aAS1K9aOWl/JYHz8eOSnB0LqAoyPlLeb\nYRhGbhBx+4r++U8Xm27tWrj4Ypg2DerXh6ZN4cEHnft3Iv5mbXhsQz697lPantSWs18+mxcXv8gh\nPRRvWXHDz8jnv8AluGm3c4GKwPuqNI26OBv5GIbhgwMH3LRc+qho164j03Nt20KZMvFWmJE1m9fQ\nf1J/ShUrxSvdXqHOMXUi2n4yjHx8ORyIUAnYrkqaCGWAcqpsjLo4Mz6GYeSBb7+FDz5whmjxYmjR\nwhmiLl2gdu14q3OkHUrjmUXP8N8F/yXQKsDAcwdGLP5dgTA+IpTGrfPUVOVGEeoCp6ryftTFmfEx\nDCOfbN8Os2Y5QzR1qlsbSjdEzZpB0ThnWPj2j2+5dtK1FJEi/H975x5lZVWG8d+DkoKkmWWUKKlg\nGEkODQQOOnhB0cqplpe0VmqzEJcWLG3hpYxxNE00V5m6NFcwQWmatwTCvHIxNCRE8gIaRBNIA1Kg\nkC4U5u2PvY9zmDkMZ3Tm7G/G97fWWed8l/N9z/lmzn7Ofr+933fyyZPpv0//933MrmI+dxNKHnzb\njM9FM3qqPXK77VScm4/jOO3Itm2hJ5QLz61eDaNHBzM64QTYe+9Euhq3cdMzN/HjeT/mh0f+kHFf\nHPe+CuR1FfP5qxnl+dmtJZb4aDfHcTo7q1aF3tDMmTB3bsi0kOsVDRhQ+jlFy/+7nOrp1byz7R2m\nVE1hwMcGvKfjdAbzKSbA+LZED2JhOYmDgS0dqspxHKcE7L8/jB0LM2ZAQwNMmAArVoRS4/36wfjx\nIWS3pUQtXr+P9mP2WbM587AzGTFlBNfNv46tjVtLc/ISU0zPZxRwOfBZ4BGgAjjbjDkdLs57Po7j\nJCALc4pWblhJ9fRqNr+9mbqqOgbuO7Do93aGnk+r5iMhoA/wJjCMUFLhL2asL4k4Nx/HcTLAa6+F\n+UQzZ4ay44cc0hSeKyvruBpGjdbI7Ytu50ezf8SFwy7k4oqL2bXbzkdIdHrzAZB43ozDSqSn2bnd\nfBzHyRb5c4pmzYK1a2HECBg5Eior4fDDQxmJ9qR+Yz1jZozhP2/9h7qqup2W7u4q5jMVuNmMhaWR\nlH9uNx/HcbJNQ0MYrJB7vPpqMKPKyvAYPLh9hnObGVMWT+HSxy/le0O/x2UjLqP7Lt0L7ttVzGcZ\n0A+oB/5HCL2ZGa1bb3uIc/NxHKeTsW4dzJvXZEb19XDEEU1mVF4O3Qt7RlGsfmM15844lzWb1lBX\nVUfZJ8ta7NNVzKdvofVm1HeIou3O7ebjOE7nZv16ePLJJjNasSJMbq2sDKG6IUPaXrfIzJi2ZBoT\nHp3AeeXncflRl/OhXZoO0iXMJyVuPo7jdDU2bGgyozlzQiqgoUObzGjoUNh99+KOtWbTGs6beR4r\nN66krqqO8k+VA24+7xs3H8dxujqvvx4GMOTM6KWXQmguN4Bh2DDo0WPH7zcz7nz+Ti565CKqy6qZ\nWDmRHt17uPns8MRiNPBzwkTXyWZMarmPm4/jOB8sNm2C+fODEc2dG+YblZU1mdHw4YWzdDdsbuD8\nP57PsvXLWPrdpS3MR7Xavs2tsZZtbq1+AZxIuL9/ttXYc+3/CQMdNDq9dSS6ATcDJwADgTMk3lse\niRIzZ86c1BJakEVNkE1drqk4XFPxtLeuD3845Ju79lp4+ukwmm7iRGhshCuuCBNcKypCXaOHHw7l\nIwB69+rNfafdx5VHX9nimKpVyza3VgOa7XMicLDVWH9gLHBbu36wZiQxH0JdoL+bUW/GO8BdQFUi\nLW0ii1+ALGqCbOpyTcXhmoqno3X16gWjRsHVV4fw3Lp1cNVVYfj2NddA794hNHfJJfDQQ+L4PqcU\nOkxoc2us3mpsR21uFTANwGpsAbCXatVhuRxSJRPfD1iVt7yacHEcx3GcVujZE445JjwA3noLFiwI\nIbqf/hROP73g24ppc5vv82pct7adpG9H4koWjuM4zvuhR49wP2jkyLC8ZUvxo+VSkmTAgcQw4Aoz\nRsflSwmDNiZtv598tIHjOM57IH/AgWoV2twaGx2XQ5ubN+hAtboNmG01dndcXgZUWo11qZ7PQqBf\nnMD6b+AbwBnNd8r6UEHHcZxOQmhza9VamzsduAC4O5rVxo4yHkg04MCMbcB3CSUaXgTuMmNpCi2O\n4zhdHauxlm1ujS1VrcaqVufGfWYBK1Wr5cAvgfM7UlOmJ5k6juM4XZNUQ61bRdJkSWsl/S21lhyS\n+kh6QtKLkp6XNC4DmnaTtEDS4qjrmtSackjqJulZSdNTawGQ9E9JS+K1eia1nhyS9pJ0j6Sl8W/4\nxcR6DonX6Nn4/HpG/tcvi9fnb5LukNTGbGgdoml8bAuStQeF2kpJe0t6RNLLkh6WtFcKbTsjk+YD\n1BEmQ2WJrcBFZjYQGA5cICnpxFgz2wIcbWZlwCDgGEkVKTXlMR54KbWIPBqBkWZWZmZZGtZ/IzDL\nzA4FPg9pw89m9kq8RoOBLxBmuj+QUpOkvsAYoMzMBhHuVX8jsaaBQDVQDhwOfFnSQQmkFGorLwUe\nM7PPAE8Al5VcVRFk0nzM7M/AhtQ68jGzBrOQasLMNhMaif3SqgIzezO+3I3w90x+3ST1AU4CfpVa\nSx4iY//vkvYEjjSzOgAz22pmbySWlc9xwAozW7XTPTuWN4C3gT0k7Qr0BNaklcShwAIz22Jm24B5\nwNdLLWIHbWUVMDW+ngp8taSiiiRTX8bOgqRPE37tLEir5N3w1mKgAZhjZlnobfwMmABk6YaiAY9K\nWihpTGoxkQOB9ZLqYpjrdkmtpJAsOacDv0stwsw2ADcA/yJMfNxoZo+lVcULwJExxNWT8GNr/8Sa\ncuxrFkapmVkDsG9iPQVx82kjknoB9wLjYw8oKWbWGMNufYCjJFWm1CPpS8Da2EtUfGSBihhKOokQ\nMh2RWhAhfDQYuCVqe5MQMkmOpO7AycA9GdByEHAh0Bf4FNBL0pkpNZnZMmAS8CgwC1gMbEupqRWy\n9CPwXdx82kDs8t8L/MbMHkytJ58YrvkjIQadkgrgZEn/IPxqPlrStMSaMLN/x+fXCPcwsnDfZzWw\nysz+GpfvJZhRFjgRWBSvV2rKgflm9t8Y4rofOCKxJsyszszKzWwksBF4JbGkHGulkJNNUm9gXWI9\nBcmy+WTpV3OOKcBLZnZjaiEAkj6WG8kSwzWjgA5LgV4MZvYDMzvAzA4i3BR+wsy+nVKTpJ6xx4qk\nPYDjCWGTpMTQyCpJh8RVx5KdQRpnkIGQW+RlYJik3SWJcJ2SzwuU9PH4fADwNeDOVFLYvq2cDpwd\nX58FZOqHco5M5naTdCcwEthH0r+AmtxN2YSaKoBvAs/HeywG/MDM/pRQ1ieBqfEL2Y3QI3s8oZ6s\n8gnggZiuaVfgDjN7JLGmHOOAO2KY6x/AOYn1EO9hHAecm1oLgJktib3nRYTQ1mLg9rSqALhP0keB\nd4DzUwwWKdRWAtcC90j6DlAPnFZqXcXgk0wdx3GckpPlsJvjOI7TRXHzcRzHcUqOm4/jOI5Tctx8\nHMdxnJLj5uM4juOUHDcfx3Ecp+S4+TiZR1KjpOvzlr8vaWJ8XSfpPSd0lHR9TInfvIT7VyRdHF9X\ntWcGc0mfl3RioXM5zgcFNx+nM7AF+Hqc0NfejAEGmdkl+SvNbIaZXRcXvwoMbMtBJe3SyubDCTnm\nCp3LcT4QuPk4nYGthBntF+1g+6iYrXpZTGzagrwezhJJp8Z1DwK9gEW5dXn7nyXpJknDCQk2r4uZ\npw+UdJCkh+I55+bS48Re2K2S/gJMkjRE0lOSFkn6s6T+MZPBlcBp8Xin5s4Vj9FX0uOSnpP0aCxP\nkTv2jZLmS1qe6+1J6h01PKtQaC0r9Zwcp1UymV7HcZphwC2E1EaTCmzva2ZDJPUDZks62Mzezm2M\nDfUgMztM0r7AQknzzKxK0hsxo3TB85rZ0wrVWGeY2f3xeI8BY81shaShwK2EfGMA+5nZsLhfL2CE\nmTVKOhb4iZmdEkOGXzCzcXG/s2jKPHwTUGdmv5V0Tlz+WtzW28wqJB1KyN91P3Am8Ccz+0lMs9Sz\nLRfWcVLh5uN0Csxss6SphAqpbzXb/Pu4z3JJK4ABQH4J9hHEJJlmtk7SHGAIMJM2Jq+NiUmPIOTO\nyr23e94u+SUIPgJMk9SfYC7FfN+G02Q2vyGk7c/xh/gZlkYTBVgITI49qgfNbElbPo/jpMLDbk5n\n4kZC6eI9mq3PT1Aodl6/JN9w2prcsBuwwcwGx3LTZWb2ubzt/8t7fRUhq/dhwFeA3Ys4fmt6tuS9\nFoCZPQkcRSiy9mtJ3yrmQzhOatx8nM5ArqHdQOjlVDfbfqoCBxOqg77cbPuTwOkKVV8/DhxJUxXa\nYno+m4A9o4ZNwEpJp7wrThq0g/ftSTAF2D5b9bvHK8BThHIGAN+K2guheO4DgHVmNplQtjwr9YAc\np1XcfJzOQH5v4AZgn7x1Riiv/AyhmN7Y/Ps9AGb2ACEMtwR4DJiQVyStmJ7PXcCEOHDgQEJpjeo4\nKOAFwoCEQse6HrhW0iK2/67NBj6bG3DQ7D3jgHMkPRfPM34Hx84tjwSWSHqWkDo/E7WmHGdneEkF\nx3Ecp+R4z8dxHMcpOW4+juM4Tslx83Ecx3FKjpuP4ziOU3LcfBzHcZyS4+bjOI7jlBw3H8dxHKfk\nuPk4juM4Jef/dsyG+gsV2cMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x122a26d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Implementation of the conjugate gradient algorithm\n",
    "\"\"\"\n",
    "\n",
    "size = 10\n",
    "M = np.random.rand(size,size)\n",
    "M = np.dot(M.T,M) #Symmetric\n",
    "y = np.random.rand(size)\n",
    "xref = np.dot(np.linalg.inv(M),y)\n",
    "\n",
    "#Allocate matrices and vectors\n",
    "xk = np.zeros(size)\n",
    "rj = np.zeros(size)\n",
    "rj1 = np.zeros(size)\n",
    "errorM = [np.dot(xk-xref,np.dot(M,xk)-y)]\n",
    "residual = [np.linalg.norm(np.dot(M,xk)-y)]\n",
    "\n",
    "Wp = np.zeros((size,1))\n",
    "\n",
    "#Initialize\n",
    "rj = -y\n",
    "Wp[:,0] = rj\n",
    "wj = rj.copy()\n",
    "\n",
    "#NB iter can be different from size\n",
    "nbIter = size\n",
    "\n",
    "for i in np.arange(1,nbIter):\n",
    "\n",
    "    #denominator = rjt M rj\n",
    "    Mwj = np.dot(M,wj)\n",
    "    den = np.dot(wj,Mwj)\n",
    "    \n",
    "    #Update xk\n",
    "    ak = np.dot(-rj,wj)/den\n",
    "    xk = xk+ak*wj\n",
    "    \n",
    "    #Current rj\n",
    "    rj1 = rj+ak*Mwj\n",
    "    \n",
    "    #Compute gammaj to update Wp\n",
    "    gammaj = np.dot(rj1,Mwj)/den\n",
    "    \n",
    "    #Update Wp\n",
    "    Wp=np.pad(Wp,((0,0),(0,1)), mode='constant')\n",
    "    wj = Wp[:,i] = rj1-gammaj*wj\n",
    "    \n",
    "    #update metric\n",
    "    errorM.append(np.dot(xk-xref,rj1))\n",
    "    residual.append(np.linalg.norm(rj1))\n",
    "    \n",
    "    #update rj\n",
    "    rj=rj1.copy()\n",
    "\n",
    "#Check visually if WptMWp is diagonal\n",
    "D = np.dot(Wp.T,np.dot(M,Wp))\n",
    "\n",
    "#Show matrix Hpp, L and D\n",
    "fig,ax = plt.subplots(1,1,figsize=(3,3))\n",
    "ax.set_title(\"Diagonal factorization of M\")\n",
    "ax.imshow(D,interpolation='none')\n",
    "ax.get_xaxis().set_visible(False)\n",
    "ax.get_yaxis().set_visible(False)\n",
    "    \n",
    "fig, ax1 = plt.subplots()\n",
    "fig.suptitle(\"Lanczos / Krylov projection based method convergence\")\n",
    "ax1.set_xlabel(\"Nb of iterations\")\n",
    "#axe 1 : Error\n",
    "ax1.set_ylabel('residual*error dot product absolute value', color='b')\n",
    "for tl in ax1.get_yticklabels():\n",
    "    tl.set_color('b')\n",
    "ax1.plot(range(1,nbIter+1),np.abs(errorM),color='b',label='Error')\n",
    "#axe 2 : Residual\n",
    "ax2 = ax1.twinx()\n",
    "ax2.set_ylabel('Residual norm', color='g')\n",
    "for tl in ax2.get_yticklabels():\n",
    "    tl.set_color('g')\n",
    "ax2.plot(range(1,nbIter+1),residual,color='g',label='Residual')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation\n",
    "We can see that the norm of the residual is not monotically decreasing along iterations, which was not a constraint of our algorithm. However, as expected, the absolute value of the dot product of the error with the residual is monotonically decreasing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## A spectral analysis of the conjugate gradient algorithm\n",
    "\n",
    "Here, we will assume that, as $M$ is symmetric, it can be diagonalized in an othornomal basis $Q$ such that:\n",
    "\n",
    "\\begin{equation}\n",
    "  M = Q \\Lambda Q^\\intercal\n",
    "\\end{equation}\n",
    "\n",
    "With $\\Lambda = diag( \\lambda_1, \\lambda_2, \\dots, \\lambda_{n-1})$\n",
    "Now, if we define $\\bar{x} = Q^\\intercal x$ and $\\bar{y}=Q^\\intercal y$ we can rewrite the objective minimized by the conjugate gradient method:\n",
    "\n",
    "\\begin{align*}\n",
    "  Lcz(x) &= \\frac{1}{2} x^\\intercal M x - x ^\\intercal M \\hat{x} \\\\\n",
    "  &= \\frac{1}{2} x^\\intercal Q \\Lambda Q^\\intercal x - x^\\intercal y \\\\\n",
    "  &= \\frac{1}{2} x^\\intercal Q \\Lambda Q^\\intercal x - x^\\intercal Q Q^\\intercal y \\\\\n",
    "  \\bar{Lcz}(\\bar{x}) &= \\frac{1}{2} \\bar{x}^\\intercal \\Lambda \\bar{x} - \\bar{x}^\\intercal \\bar{y} \\\\\n",
    "  &= \\sum_{i=0}^{n-1} \\frac{1}{2} \\lambda_i \\bar{x}_i^2 -\\bar{x}_i \\bar{y}_i\n",
    "\\end{align*}\n",
    "\n",
    "The minimization of this simple quadratic form gives us\n",
    "\\begin{align*}\n",
    "  \\bar{x}_i^\\star = \\frac{\\bar{y}_i}{\\lambda_i} \\\\\n",
    "  Lcz^\\star = -\\frac{1}{2}\\frac{\\bar{y}_i^2}{\\lambda_i}\n",
    "\\end{align*}\n",
    "\n",
    "From this we can redefine the Krylov sequence, which is the sequence of solutions $\\bar{x}^k$ generated by each iteration of the conjugate gradient:\n",
    "\n",
    "\\begin{equation}\n",
    "    \\bar{x}^k = \\underset{\\bar{x}\\in\\bar{\\mathcal{K}_k}}{argmin} \\; Lcz(\\bar{x})\n",
    "\\end{equation}\n",
    "\n",
    "with $\\bar{\\mathcal{K}_k} = Span\\{ \\bar{y} , \\Lambda\\bar{y} , \\Lambda^2\\bar{y} , \\dots , \\Lambda^{k-1}\\bar{y} \\}$\n",
    "\n",
    "From the previous definition of $\\bar{\\mathcal{K}_k}$, we can derive the following expression of $\\bar{x}^k$:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\bar{x}^k &= \\begin{pmatrix}\\bar{y}\\;\\Lambda\\bar{y}\\;\\Lambda^2\\bar{y}\\;\\dots\\;\\Lambda^{k-1}\\bar{y}\\end{pmatrix} \n",
    "    \\begin{pmatrix} p_0 \\\\ p_1 \\\\ p_2 \\\\ \\vdots \\\\ p_{k-1} \\end{pmatrix} \\\\\n",
    "    \\implies \\bar{x}^k_i &= p_k(\\lambda_i) \\bar{y}_i\n",
    "\\end{align*}\n",
    "\n",
    "Where $p_k()$ is a degree $k-1$ at most polynomial that satisfies the following optimality property:\n",
    "\n",
    "\\begin{equation}\n",
    "    p_k = \\underset{p_k\\in X^p, p<k}{argmin} \\; \\sum_{i=0}^{n-1} \\frac{1}{2} \\lambda_i (p_k(\\lambda_i) \\bar{y}_i)^2 -p_k(\\lambda_i) \\bar{y}_i^2\n",
    "\\end{equation}\n",
    "\n",
    "Here we see that the $p_k$ polynomial has an important role in finding the best approximation of the solution in the Krylov basis.\n",
    "We can rewrite the $p_k$ problem using the optimal value gap thanks to the optimal value of the objective $Lcz^\\star$ we obtained earlier:\n",
    "\n",
    "\\begin{align*}\n",
    "  \\bar{Lcz}(\\bar{x_k}) - Lcz^\\star &= \\underset{p_k\\in X^p,p<k}{min} \\; \\sum_{i=0}^{n-1} \\frac{1}{2} \\lambda_i (p_k(\\lambda_i) \\bar{y}_i)^2 -p_k(\\lambda_i) \\bar{y}_i^2 + \\frac{1}{2}\\frac{\\bar{y}_i^2}{\\lambda_i} \\\\\n",
    "  &= \\underset{p_k\\in X^p,p<k}{min} \\; \\frac{1}{2} \\sum_{i=0}^{n-1} \\bar{y}_i^2 \\frac{ \\lambda_i^2 p_k(\\lambda_i)^2 -2\\lambda_i p_k(\\lambda_i) + 1 }{\\lambda_i} \\\\\n",
    "  &= \\underset{p_k\\in X^p,p<k}{min} \\; \\frac{1}{2} \\sum_{i=0}^{n-1} \\bar{y}_i^2 \\frac{ \\left( 1 - \\lambda_i p_k(\\lambda_i) \\right)^2 }{\\lambda_i} \\\\\n",
    "  &= \\underset{p_k\\in X^p,p<k}{min} \\; \\frac{1}{2} \\sum_{i=0}^{n-1} \\frac{\\bar{y}_i^2}{\\lambda_i} \\left( 1 - \\lambda_i p_k(\\lambda_i) \\right)^2\\\\\n",
    "  &= \\underset{p_k\\in X^p,p<k}{min} \\; \\frac{1}{2} \\sum_{i=0}^{n-1} \\frac{\\bar{y}_i^2}{\\lambda_i} \\left( \\underbrace{ 1 - \\lambda_i p_k(\\lambda_i) }_{=q_k(\\lambda_i)} \\right)^2\\\\\n",
    "  &= \\underset{q_k\\in X^p,p\\leq k, q_k(0)=1}{min} \\; \\frac{1}{2} \\sum_{i=0}^{n-1} \\frac{\\bar{y}_i^2}{\\lambda_i} q_k(\\lambda_i)^2\\\\\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Best possible polynomial approximation using Chebyshev polynomial\n",
    "\n",
    "We exposed earlier that the minimization of the $Lcz$ objective at each step can be recasted as an optimization problem over the set of monic polynomials of degree $p < k$.\n",
    "\n",
    "In a first approach, we can simplify the objective function $\\bar{Lcz}(\\bar{x_k}) - Lcz^\\star$, that is divided into a sum of $n$ polynomials by bounding the value of its expectation along the $n$ terms of the sequence by the maximum term, which reads:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\underset{q_k\\in X^p,p\\leq k, q_k(0)=1}{min} \\; \\frac{ \\sum_{i=0}^{n-1} \\frac{\\bar{y}_i^2}{\\lambda_i} q_k(\\lambda_i)^2}{\\sum_{i=0}^{n-1}\\frac{\\bar{y}_i^2}{\\lambda_i}} \\leq \\underset{q_k\\in X^p,p\\leq k, q_k(0)=1}{min} \\left( \\underset{i=0,1,\\dots,n-1}{max} q_k(\\lambda_i)^2 \\right) \\\\\n",
    "\\end{align*}\n",
    "\n",
    "We can see that the optimal monic polynomial of degree at most $k$ for estimating the coordinates of the linear system solution in the Krylov basis should be the one that has the lowest values over the $n$ eigen values of the matrix $M$.\n",
    "\n",
    "This remind us of the optimality property of the Chebyshev polynomial, which is the monic polynomial that provides the best possible approximation of the null function in terms of maximum norm over the interval$[-1,1]$. See the notebook called ChebyshevPolynomialAcceleration for more informations on that topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}