6 Cluster and CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Assignment 6\n",
    "\n",
    "This assignment has weighting $3.5$.\n",
    "The first question about clustering has 35%, and the second question about tiny image classification has 65%.\n",
    "\n",
    "This is a challenging assignment, so I recommend you start early."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Clustering for handwritten digits\n",
    "\n",
    "Supervised learning requires labeled data, which can be expensive to acquire.\n",
    "For example, a dataset with $N$ samples for classification will require manual labeling $N$ times.\n",
    "\n",
    "One way to ameliorate this issue is to perform clustering of the raw data samples first, followed by manual inspection and labeling of only a few samples.\n",
    "Recall that clustering is a form of non-supervised learning, so it does not require any class labels.\n",
    "\n",
    "For example, say we are given a set of scanned hand-written digit images.\n",
    "We can cluster them into 10 groups first, manually inspect and label a few images in each cluster, and propagate the label towards the rest of all (unlabeled) samples in each cluster.\n",
    "\n",
    "The accuracy of such semi-automatic labeling depends on the accuracy of the clustering.\n",
    "If each cluster (0 to 9) corresponds exactly to hand-written digits 0-9, we are fine.\n",
    "Otherwise, we have some mis-labeled data.\n",
    "\n",
    "The goal of this question is to exercise clustering of the scikit-learn digits dataset which has labels, so that we can verify our clustering accuracy.\n",
    "The specifics are as follows.\n",
    "\n",
    "You will be judged by the test accuracy of your code, and quality of descriptions of your method.\n",
    "As a reference, a simple code I (Li-Yi) wrote can achieve about 78% accuracy. Try to beat it as much as you can."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Training and test data split\n",
    "\n",
    "We will split the original dataset into training and test datasets\n",
    "* training for building our clusters\n",
    "* testing to see if the clusters can predict future data\n",
    "\n",
    "## Accuracy\n",
    "What is your clustering accuracy (comparing cluster labels with the ground truth labels), and what are the properties of mis-clustered samples?\n",
    "\n",
    "## Data preprocessing\n",
    "Would the original features (pixels) work well, or we need further processing like scaling/standardization or dimensionality-reduction, before clustering?\n",
    "\n",
    "## Models and hyper-parameters\n",
    "\n",
    "Let's focus on k-means clustering, as hierarchical and density-based clustering do not provide the predict() method under scikit-learn.\n",
    "\n",
    "What is the best test performance you can achieve with which hyper-parameters (for k-means, standard scalar, and dimensionality reduction)?\n",
    "\n",
    "### Hint\n",
    "We have learned Pipeline and GridSearchCV for cross validation and hyper-parameter tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last updated: 2016-12-14 \n",
      "\n",
      "CPython 3.5.2\n",
      "IPython 5.1.0\n",
      "\n",
      "numpy 1.11.2\n",
      "pandas 0.19.1\n",
      "matplotlib 1.5.3\n",
      "scipy 0.18.1\n",
      "sklearn 0.18.1\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a '' -u -d -v -p numpy,pandas,matplotlib,scipy,sklearn\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Added version check for recent scikit-learn 0.18 checks\n",
    "from distutils.version import LooseVersion as Version\n",
    "from sklearn import __version__ as sklearn_version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1797, 64)\n",
      "(1797,)\n",
      "[0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "from sklearn.datasets import load_digits\n",
    "digits = load_digits()\n",
    "\n",
    "X = digits.data # data in pixels\n",
    "y = digits.target # digit labels\n",
    "\n",
    "print(X.shape)\n",
    "print(y.shape)\n",
    "\n",
    "print(np.unique(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Visualize data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xu0FOWZ7/HfE1E2inL1Bo4YzAySeDkiEkVPdIIJcbxA\nJvGGHiOOIdGYWREzx2QZgoqYpWcgRiOTi9k4UbzOjKJRQpSAMVwCEY2EbA2CeMEoysUICEZ4zx9V\naNPueuvd1XvTVdXfz1q1YNfTT/Xb/ezu/XRd3jbnnAAAAJDuI/UeAAAAQFHQOAEAAASicQIAAAhE\n4wQAABCIxgkAACAQjRMAAEAgGicAAIBANE4AAACBaJwAAAACFaZxMrOrzGxb1bqVZtaccXtzzGx2\n+4wOWVDT8qGm5UNNy4ea1qYwjZMkFy+VtrWyri3be/8Xx8z2N7PxZnZ46AbMbDczu97MVpnZJjNb\nYGYnZRxPI8pVTc1sDzO72sxmmNkaM9tmZudnHEujyltNB5vZD83sj2a2wcxeNLN7zOzvM46nEeWt\nph83s3vNbLmZbTSzN8zscTM7NeN4GlGualrNzK6M33+fyTieDtWp3gOo0QBVFKuNPlP1cx9J4yW9\nICm0WP8p6Z8lfV/S85IukPSImZ3onJuXcVyNrp417S1pnKQXJT0t6cSM48CO6lnTKyQNlXRffPv9\nJH1d0mIz+6Rz7k8Zx9Xo6lnTfpK6SrpN0quSdpf0BUkPmtkY59ytGcfV6Or991SSZGZ9JX1b0oaM\nY+lwhW6cnHN/qyH3vapV1pZ8Mxsi6SxJlzvnvh+vu13SHyXdIOn4rGNrZPWsqaI34f2cc6vN7ChJ\ni7KOBR+oc00nSTqncjtmdq+kJZK+JYk9ihnUs6bOuRmSZuywAbMfSlosaawkGqcM6vw6rTRJ0nxF\n/UmvGrbTYXJ5qM7MjjezRWb2jpktM7MxCbf70DFZMzs83m27ycxejnf5jY53+x1Ycbs5Zvbr+P8n\nSFqoaHfjbfFtt6YcpvmipPck/XT7CufcFkk/k3Rs3DUjVoSaOuf+5pxb3S4PuAEUpKYLqt/UnXPP\nS1oqaWDmB19SRahpa5xzTtLLkrq38SGXXpFqamafUnQU5xs1POQOl7s9TmZ2qKSZklZL+q6kXSVd\nFf9cbYfjsWbWR9JsSVslTZS0SdJFkt6tvm3Vzy3xfV0j6ceSnojX+w63/S9Jf3bOVe9OXFgRX+XJ\nbxgFqikClaCm+yraO4xY0WpqZrtL6iKpm6QRkk6WdFdaXiMpUk3N7COSbpL0U+fcUrNadlp1rNw1\nTpImxP8e75xbJUlm9t8Ke5P7lqIX0ZHOuSVx7lRF5x8lig/NzFBU6PnOuTsD7mt/SX9pZf1fFO2m\n7BOwjUZRlJoiXGFrambnSeor6TtZ8kusaDWdJOkr8f+3SfpvReev4QNFqunFkg6U9OnA29dNrg7V\nxR3nZyXdv73IkuSce05R15xmuKJCLanIXS9pWnuPVdEnnS2trN9cEW94BaspAhS5pmZ2iKQfSpor\n6ecdfX9FUdCafl/SSYrOU3tE0i6SOnfg/RVKkWpqZj0lXS3pGufc2vbefnvLVeMkaW9FDUdrHe1z\nAfn9EnK9HXJG76j1F2lTRRzFqinCFLKmZravpIclrZN0RnxeDCKFq6lz7s/OuV875+5wzp0uaU9J\nD3bU/RVQkWo6UdIaRR9qci9vjVOR/EXR4bpq29e9uhPHAsDDzPaS9EtJe0n6nHPutToPCe3vvyQd\nbczRVShm9jFJX1Z0flNfM+tnZgcp2gmxa/xzjzoO8UPy1ji9oWhPTWu/+IcE5L8o6WOtrA95IbX1\n0+fTkv7BzLpWrT8m3tbTbdxeWRWppghTqJqaWWdJv4jv85T4UAV2VKiaJth+ekS3dtpe0RWlpn0V\nnRd8k6J5n16QtELSJxXNLbVC0fx6uZGrxsk5t03RsdeRZnbA9vVmNlDRsdo0MxVNBfD+bKXxsdNR\nAbkb439DL2f9L0Un179/aaeZ7aZoEswFlceUG1nBaooARappfJ7HvYrehL/onFuYktKQClbTvVtZ\n10nSlxQ1CkxqqkLV9I+SPh8vIyuWpYqat5GKpvnJjTxeVTde0uck/dbMpii6fPJSRU9u2vTtN0g6\nT9JjZnazouJdpOjJ7yF/F7xc0npJXzWzDXHu75xzK1u7sXNuoZndJ+l78bkT22cO7ydpdPrDbCiF\nqKkkmdnXFL3Yt8/DdbqZ/V38/5ucc2+njLdRFKWmkyWdpujcl95mdm5l0DnHRQYfKEpNfxwfev2N\noilf9pN0rqK9E2Odc5tSxtpIcl9T59watXJumpldFoXdQynj3Pmcc7lbFM26vVDRp4dlio5/jpe0\ntep2KyT9rGrd4ZLmKJpz4iVFU7d/XdFcFHtX3G62pFlVuacqmlF4S3z781PGuZuk6xW9eDdJWiDp\npHo/f3lcClTTF+LbtbYcWO/nMU9LEWqqD+ahaXWp93OYt6UgNT1T0d6QV+Pbvxn/fEq9n788LkWo\nacK4Z0v6Q72fv9YWiwdYamZ2o6Jflq6uER5wA6Cm5UNNy4ealg81zdk5Tu3BzJqqfu6laHfjE41a\n5KKjpuVDTcuHmpYPNW1dHs9xqtV8M5ujaNr3/SRdqGh+jwm+JOQaNS0falo+1LR8qGkrytg4Pazo\nC3i/rOjktScljXbOza3rqFALalo+1LR8qGn5UNNWNMQ5TgAAAO2hdOc4AQAAdJSgQ3XxCWHDJa3U\nB19ii/prknSQpJkumgsjGDXNLWpaPtS0fKhp+YTXNHA+hVGKjm+y5HMZlWFuD2qa74Walm+hpuVb\nqGn5ltSahp4cvlKS7rjjDg0cODAwpWM988wz3vjVV1+dGPv0pz/tzb3ooou88c6dO3vjO0tLS4vO\nO+88Ka5PG62U8lXTNJdffnlibO3atd7cb37zm974Jz7xiUxjam+NVtOVK1cmxi644AJv7lFHHeWN\nT5o0KcOI2l/ZavrQQ/6JnK+66qrE2EEHHeTNvfPOO71x3nvrY8uWLYmx733ve95c3+9DnrSlpqGN\n02ZJGjhwoAYNGpR9ZO3IV0hJampqSoztt99+3twjjzwy87brJMvu3tzVNE337slfe/Tuu+96cw85\nxP+dljl8DhqipnvssUdirFMn/9uT7/dBoqYdZcmSJZlz0947ee/Np82bkx9m7969vblFeYwVUmvK\nyeEAAACBaJwAAAAC0TgBAAAEonECAAAIVNivXLnwwgu98WeffTYxlnYFVpcuXbzxefPmJcaOPfZY\nby6y69GjR2LsgQce8ObOnDnTGz/66KMzjQl+q1at8sZ9J+376i3VdpIy/HxXJN56663e3Icffjgx\ndsopp3hzV6xY4Y1//OMf98bRMaZPn54YGzx48E4cST6wxwkAACAQjRMAAEAgGicAAIBANE4AAACB\naJwAAAAC0TgBAAAEyvV0BC+//HJizDfdgOSfciDtMue06QqYjqBjpF26njblgA91qQ/fZcySNHTo\n0MTYueee68392te+lmlMSOeb7iXtefd931zad0Yy3UB9+L6LTpJuuummxNg111zjzV2/fn2mMUnp\n30dZL+xxAgAACETjBAAAEIjGCQAAIBCNEwAAQCAaJwAAgEA0TgAAAIFonAAAAALleh6nt99+OzF2\n4oknenPT5mryGTJkSOZc+N1zzz2JsYsvvtibu27dusz3e9RRR2XORXa++YAkacCAAYmxM844w5s7\nevToTGNCOt/7Z9rr0DfH3plnnunNTZtPqKmpyRtHNmnzrbW0tCTGhg0b5s299tprvfGePXsmxi65\n5BJvbr2wxwkAACAQjRMAAEAgGicAAIBANE4AAACBaJwAAAAC0TgBAAAEonECAAAIlOt5nN56663E\n2Kmnntph97t27Vpv3DfvBPzOOuusxNiIESO8uV26dMl8vxs3bvTGu3fvnnnbjc43905zc7M3d9q0\naZnvd8qUKZlzkV3aHHnvvPNOYuzkk0/25qbFZ8yYkRhjjie/RYsWJcbOPvtsb+7YsWMz3++4ceO8\n8cceeyzztuuFPU4AAACBaJwAAAAC0TgBAAAEonECAAAIROMEAAAQiMYJAAAgUK6nI+jWrVtibOHC\nhZm367t8WpLmzZvnjV9wwQWZ7xv18eyzz3rjffv23UkjKZ9///d/T4ylXYrsk/Ya5/LzfPLVxTed\ngCRddtll3vgtt9ySGLv88sv9A2twe+21V2IsbYqJyZMnJ8YWLFiQeUySdNxxx9WUXw/scQIAAAhE\n4wQAABCIxgkAACAQjRMAAEAgGicAAIBANE4AAACBaJwAAAAC5Xoep/333z8xNmvWLG/u/PnzE2M/\n//nPM49Jkr70pS/VlA+UyejRoxNjafP2+OZMGzJkSOb7laSLL744MXb00Ud7c5Fs0qRJ3vjJJ5+c\nGHvrrbe8uffdd583/pWvfMUbR7IBAwYkxtauXevNXbVqVWLssMMO8+aOHTvWGy/ifGzscQIAAAhE\n4wQAABCIxgkAACAQjRMAAEAgGicAAIBANE4AAACBcj0dQY8ePRJjaVMKXHjhhYmxE0880Zs7e/Zs\nbxwdI+2yVN/l51OnTvXmPvLII974sGHDvHEk69u3b2Js7ty53lzfZc7jxo3z5qbVvH///okxpiPI\nrnfv3t74F77whczbTptuYOLEiZm3jez22GOPxNi6deu8uWPGjGnv4dQde5wAAAAC0TgBAAAEonEC\nAAAIROMEAAAQiMYJAAAgEI0TAABAoNDpCJokqaWlpQOH0jbPP/+8N7558+bE2BtvvOHNXbx4caYx\n7WwV9cjy9dK5q2maN998M3Pu66+/7o3npeaNVtPVq1cnxmqptyS9+uqribGdWe+y1XTlypXeuO+9\nN81rr73mjfM6rY+33347c+7SpUu98Y0bN2bedntqU02dc6mLpFGSHEtul1EhdaSmhVqoafkWalq+\nhZqWb0mtqcWF9DKzXpKGS1opKfvHCbS3JkkHSZrpnFvTlkRqmlvUtHyoaflQ0/IJrmlQ4wQAAABO\nDgcAAAhG4wQAABCIxgkAACAQjRMAAEAgGicAAIBANE4AAACBaJwAAAAC0TgBAAAEonECAAAIROME\nAAAQiMYJAAAgEI0TAABAoMI0TmZ2lZltq1q30syaM25vjpnNbp/RIQtqWj7UtHyoaflQ09oUpnGS\n5OKl0rZW1rVle+//4pjZ/mY23swOD0k2sxPMbFsry1YzG5JxTI0mVzWtyBtkZg+a2Roz22hmS8zs\n0oxjajS5qqmZTU14nW5/re6fcVyNJFc1jXM+ZmZ3m9nL8Wu0xczGmVmXjGNqNHms6VFm9ksze8vM\n/mpmM83siIzj6VCd6j2AGg1QRbHa6DNVP/eRNF7SC5KeacN2bpT0+6p1z2ccE+pcUzP7rKQHJS2W\ndI2kDZIOlnRAxjGhvjX9kaRHq9aZpB9LWuGc+0vGcTW6utXUzA6QtEjSOkk3S1or6VhJV0saJOnz\nGcfV6OpZ00GSnpD0Upy3i6RLJM0xsyHOuWUZx9UhCt04Oef+VkPue1WrLOOmfuuc+5+s48CO6llT\nM9tT0n9Kesg5d0bWcWBH9aypc+53kn63wwbMjpO0u6RpWcfV6Or83nu+pL0kHeucezZed6uZ7SLp\n/5hZN+fcW1nH16jqXNMJkjZJOsY5t16SzGyapD9Luk5Srt6Pc3mozsyON7NFZvaOmS0zszEJt/vQ\nMVkzO9zMHjezTfFu3CvNbHS8a/7AitvNMbNfx/8/QdJCRbsbb6vYjX9+4Hi7xi9aJChITc+VtI+k\nK+Nt7G5mWRvq0itITVtzrqJP1ne1Ma/0ClLTPeN/V1etf01RXd9t26Mut4LU9HhJj21vmiTJOfea\npMclnWpmu2d+AjpA7vY4mdmhkmYqelF8V9Kukq7Sh18kUtXxWDPrI2m2pK2SJirqYC9S9EKqPnZb\n+XNLfF/XKNqF/0S8fl7AkKcqeiFvNbMnJP2bc+7JgLyGUaCaDpP0V0l/Z2YPSvoHSRvN7HZJlznn\ntvgeZyMpUE2rx91J0afXuc65l0LzGkGBajpH0hWSms1svKQ1ko6T9FVJP3DOvePJbSgFqmlnSa3V\nbZOk3SQdqqgZywfnXK4WSfdL2iipb8W6AZL+Jmlr1W1fkNRc8fNNkt6TdFjFuu6S3lRU/AMr1s+W\n9OuKn49S9Gnl/MBxHivpXkkXSDpV0v9V9Mu4UdIR9X4e87QUqKZPKzqnaYOk70saqegctm2SptX7\neczTUpSatjLuU+P8MfV+DvO2FKmmivYKb4zztsX3cU29n8O8LUWpqaQ/KGq4rGLdrpJWxvf1+Xo/\nl5VLrg7VmdlHJH1W0v3OuVXb1zvnnlPUNacZLmm+c25JRe56dcC5DM65+c65M51ztznnfuGcu0FR\nMyVJ32vv+yuqItVUUldJXSTd5py7zDn3gHPuG4o+NZ1tZgd3wH0WTsFqWm2Uok/M9+2E+yqMAtZ0\npaLDOBdJ+mdJzZKuNLNLOuj+CqdgNZ2iaA9/s5kNjPeU3S5pvzieq6slc9U4Sdpb0RPU2lVpzwXk\n90vI3SlXuTnnlkuaLukfOTfmfUWq6fZdxXdXrb9T0cmOxwpSsWr6PjPbQ9Lpkn7pnFvXkfdVQIWp\nqZmdLeknkv7FOdccf8D5sqILO643sx7tfZ8FVZiaOud+rOgk8HMkLVW0B+qjkm6Ib7Khve+zFnlr\nnMrgZUXHZPeo90DQZq/G/75etX77+QC8IRfb5xX9IeFqumK7WNJi9+GpJB5UdLXkkTt/SKiVc26c\npH0VnSh+uHPuk4qmJZCiq+tyI2+N0xuKPvX/fSuxQwLyX5T0sVbWt7a9alkn/qp2sKTNzrlcdch1\nVKSabj+pv2/V+j7xv2+0cXtlVaSaVjpX0SfXh2rYRlkVqab76oM/qJV2jf/N3UVPdVKkmkZJzr3l\nnJvnnFsar/qMpFfcB9NO5EKuGifn3DZFx15HWjTJmSTJzAYqOlabZqakY61itlIz66novIY0G+N/\nu4eM1cx6t7LuCEmnKez4cUMoUk0Vnexvkv6lav2XFZ1MOSdwO6VWsJpu335vRVdN/o9zbnNbchtB\nwWr6Z0lHmln1H/VRik5IbssExqVVsJp+iJmdJWmwogt1ciWPnfl4SZ+T9Fszm6LoU8Slkv4oKW36\n9hsknSfpMTO7WVHxLlLUOfeQvwteLmm9pK+a2YY493fOuZUJt7/HzN5RdInlakmfUPQHdoOkb6eM\ns9EUoqbOuafjeUxGm9muik4+/UdJX5B0nYvmFUGkEDWtcLaivRQcpktWlJr+v4px/lDRdASnKTqZ\n+ae8TndQiJqa2f9WNIXBrxTV81hFV6w/oujqvnyp92V9rS2KjnEuVLSbcZmihmS8Pnz55ApJP6ta\nd7iiPQObFE3f/m1JX1d0SePeFbebLWlWVe6pkpZI2hLfPvFSSkW/fPMV7Q7dIukVSbdJ6l/v5y+P\nSxFqGt9+F0nj4nFsVnQS5dfr/fzlcSlKTeOceYrOYbMsj7VRlqLUVNGeiF9IWhW/TlsUze30kXo/\nh3lbilBTSf0lzVB0fukmRSeI/5ukTvV+/lpbLB50qZnZjYp+Wbq6RnjADYCalg81LR9qWj7UNGfn\nOLUHM2uq+rmXot2NTzRqkYuOmpYPNS0falo+1LR1eTzHqVbzzWyOol23+0m6UNFXokyo56BQE2pa\nPtS0fKhp+VDTVpSxcXpY0hcV7Up0ii4xH+2cm1vXUaEW1LR8qGn5UNPyoaataIhznAAAANpD6c5x\nAgAA6ChBh+riE8KGK/piRSaPy48mSQdJmumcW9OWRGqaW9S0fKhp+VDT8gmvaeA8EKMUHd9kyecy\nKsPcHtQ03ws1Ld9CTcu3UNPyLak1DT05fKUk3XHHHRo4cGBgSse6/PLLvfG+fau/buwDY8eObe/h\n1EVLS4vOO+88Ka5PG62U8lXTNL6ar1271ps7derU9h5OhyhbTX/1q1954+vXr0+MzZgxw5v7zDP+\nb9bYc889E2MzZ/q/FWm33XaTmXlvE6psNb311lu98YceSv4qwHPPPdebO2LECG+8c+fO3vjOUraa\nXnXVVd7422+/nRibNGlSO4+mPtpS09DGabMkDRw4UIMGDco+snbUvbv/K3D23XffxFheHkM7yrK7\nN3c1TeOr+bvvvuvNLcpjrFCKmi5btswbX7MmeY94165da7rvTp2S396OPPJIb27nzp3brXGqUIqa\n9unTxxv3NTcHHnigNzetLk1NTd54HZSipr17f+irV3ewyy6tfadyJC+PoR2l1pSTwwEAAALROAEA\nAASicQIAAAhU2JnDlyxZ4o0/8MADibHJkyd7cw8++GBv/Pnnn/fGkc2iRYu8cV9Nb7nllvYeDnaC\nXr16Jcaam5u9uddff703vm7dusRYDs+VKYwnn3wyc27ae++jjz7qjd9///2Z77vR+S7EqOXimbRz\nAYcOHeqNz51bvEnI2eMEAAAQiMYJAAAgEI0TAABAIBonAACAQDROAAAAgWicAAAAAtE4AQAABCrs\nPE6+76KTpOXLlyfGevTo4c1N+6LJzZuTv8qG+WGy+8Y3vpE5N61mqI+zzjorc+6UKVO88eeee84b\nnzVrVub7RrKjjjrKG+/fv39iLO0LYXv27OmN+2o+YMAAb26j27hxY+bckSNHJsZ89Zak6dOnZ77f\nvGKPEwAAQCAaJwAAgEA0TgAAAIFonAAAAALROAEAAASicQIAAAhU2OkI0i49nTdvXmJs3bp13twh\nQ4Z440w50DFef/11b3zo0KGJsb59+7b3cBDId4l4LVMCfOc738mcK0lz585NjA0bNqymbTey0aNH\ne+MHHHBAYmzFihXe3LTpCNKmoUGyXr16Zc696667EmPnnHOON3ft2rWZ7zev2OMEAAAQiMYJAAAg\nEI0TAABAIBonAACAQDROAAAAgWicAAAAAtE4AQAABCrsPE7Nzc3e+BVXXJEYe/rpp725Z599dqYx\nSdJZZ52VObfRpc33cdhhhyXG7rnnHm/u8OHDvfHu3bt740jmm1vn97//vTf3gQceyHy/8+fP98bT\n5npDNhs2bMicm1bvtDn2eJ1m55t/0DdHniR16dIlMTZhwgRv7uOPP+6Nr1+/PjGW13qzxwkAACAQ\njRMAAEAgGicAAIBANE4AAACBaJwAAAAC0TgBAAAEonECAAAIVNh5nNJ05Bwuy5Yt67BtN7KBAwd6\n4745YFavXu3NTZub65VXXkmM9e3b15vb6HxzraTNtzZ16tTE2MKFC725zNPUcVatWpUYO+SQQ7y5\nt9xyS2Js+fLl3txTTjnFG3/44YcTY3md86cI5s6d6437fh9qfX8cO3ZsYizt/aNe2OMEAAAQiMYJ\nAAAgEI0TAABAIBonAACAQDROAAAAgWicAAAAAtE4AQAABCrsPE6LFi3yxvfaa6/E2Le+9a2a7vuM\nM86oKR+t+9d//VdvfN68eYmxtDl9WlpavPHp06cnxi655BJvLpJde+213niPHj0SY4cddlh7DweB\nevXqlRjz1UySLrzwwsTYmjVrvLkHHHCAN37nnXcmxniddhzfXE1pr/HJkyd74/Pnz880pnpijxMA\nAEAgGicAAIBANE4AAACBaJwAAAAC0TgBAAAEonECAAAIVNjpCGbOnOmNjxs3LvO2x44d642nXfqO\nbEaMGOGNT5gwITGWdsnryJEja7pvZDNjxgxv3Pc6bmpqau/hIJDvuU97LXXp0iUxljaVwejRo71x\n31QHyC5tSoEnn3wyMbZ69Wpv7pIlS7xx31QHecUeJwAAgEA0TgAAAIFonAAAAALROAEAAASicQIA\nAAhE4wQAABCIxgkAACBQ6DxOTZLU0tLSgUNpm3/6p3+qKV6LxYsXd9i226KiHlkmvMldTdP4alpr\nvV9//fVMsfZWtprefPPNmXPz8jqrVdlqeumll9YUr8Wf/vSnDtt2W5Stph359zTt/XNnvr/6tKmm\nzrnURdIoSY4lt8uokDpS00It1LR8CzUt30JNy7ek1tTiQnqZWS9JwyWtlLQ5NQE7S5OkgyTNdM6t\naUsiNc0talo+1LR8qGn5BNc0qHECAAAAJ4cDAAAEo3ECAAAIROMEAAAQiMYJAAAgEI0TAABAIBon\nAACAQDROAAAAgWicAAAAAtE4AQAABKJxAgAACETjBAAAEIjGCQAAIFBhGiczu8rMtlWtW2lmzRm3\nN8fMZrfP6JAFNS0falo+1LR8qGltCtM4SXLxUmlbK+vasr33f3HMbH8zG29mh4duwMx2M7PrzWyV\nmW0yswVmdlLG8TSiXNXUzPYws6vNbIaZrTGzbWZ2fsaxNKq81XSwmf3QzP5oZhvM7EUzu8fM/j7j\neBpR3mr6cTO718yWm9lGM3vDzB43s1MzjqcR5aqm1czsyvj995mM4+lQneo9gBoNUEWx2ugzVT/3\nkTRe0guSQov1n5L+WdL3JT0v6QJJj5jZic65eRnH1ejqWdPeksZJelHS05JOzDgO7KieNb1C0lBJ\n98W330/S1yUtNrNPOuf+lHFcja6eNe0nqauk2yS9Kml3SV+Q9KCZjXHO3ZpxXI2u3n9PJUlm1lfS\ntyVtyDiWDlfoxsk597cact+rWmVtyTezIZLOknS5c+778brbJf1R0g2Sjs86tkZWz5oqehPezzm3\n2syOkrQo61jwgTrXdJKkcyq3Y2b3Sloi6VuS2KOYQT1r6pybIWnGDhsw+6GkxZLGSqJxyqDOr9NK\nkyTNV9Sf9KphOx0ml4fqzOx4M1tkZu+Y2TIzG5Nwuw8dkzWzw+PdtpvM7OV4l9/oeLffgRW3m2Nm\nv47/f4KkhYp2N94W33ZrymGaL0p6T9JPt69wzm2R9DNJx8ZdM2JFqKlz7m/OudXt8oAbQEFquqD6\nTd0597ykpZIGZn7wJVWEmrbGOeckvSypexsfcukVqaZm9ilFR3G+UcND7nC52+NkZodKmilptaTv\nStpV0lXxz9V2OB5rZn0kzZa0VdJESZskXSTp3erbVv3cEt/XNZJ+LOmJeL3vcNv/kvRn51z17sSF\nFfFVnvzqJh+tAAAaYUlEQVSGUaCaIlAJarqvor3DiBWtpma2u6QukrpJGiHpZEl3peU1kiLV1Mw+\nIukmST91zi01q2WnVcfKXeMkaUL87/HOuVWSZGb/rbA3uW8pehEd6ZxbEudOVXT+UaL40MwMRYWe\n75y7M+C+9pf0l1bW/0XRbso+AdtoFEWpKcIVtqZmdp6kvpK+kyW/xIpW00mSvhL/f5uk/1Z0/ho+\nUKSaXizpQEmfDrx93eTqUF3ccX5W0v3biyxJzrnnFHXNaYYrKtSSitz1kqa191gVfdLZ0sr6zRXx\nhlewmiJAkWtqZodI+qGkuZJ+3tH3VxQFren3JZ2k6Dy1RyTtIqlzB95foRSppmbWU9LVkq5xzq1t\n7+23t1w1TpL2VtRwtNbRPheQ3y8h19shZ/SOWn+RNlXEUayaIkwha2pm+0p6WNI6SWfE58UgUria\nOuf+7Jz7tXPuDufc6ZL2lPRgR91fARWpphMlrVH0oSb38tY4FclfFB2uq7Z93as7cSwAPMxsL0m/\nlLSXpM85516r85DQ/v5L0tHGHF2FYmYfk/RlRec39TWzfmZ2kKKdELvGP/eo4xA/JG+N0xuK9tS0\n9ot/SED+i5I+1sr6kBdSWz99Pi3pH8ysa9X6Y+JtPd3G7ZVVkWqKMIWqqZl1lvSL+D5PiQ9VYEeF\nqmmC7adHdGun7RVdUWraV9F5wTcpmvfpBUkrJH1S0dxSKxTNr5cbuWqcnHPbFB17HWlmB2xfb2YD\nFR2rTTNT0VQA789WGh87HRWQuzH+N/Ry1v9SdHL9+5d2mtluiibBXFB5TLmRFaymCFCkmsbnedyr\n6E34i865hSkpDalgNd27lXWdJH1JUaPApKYqVE3/KOnz8TKyYlmqqHkbqWian9zI41V14yV9TtJv\nzWyKossnL1X05KZN336DpPMkPWZmNysq3kWKnvwe8nfByyWtl/RVM9sQ5/7OObeytRs75xaa2X2S\nvhefO7F95vB+kkanP8yGUoiaSpKZfU3Ri337PFynm9nfxf+/yTn3dsp4G0VRajpZ0mmKzn3pbWbn\nVgadc1xk8IGi1PTH8aHX3yia8mU/Secq2jsx1jm3KWWsjST3NXXOrVEr56aZ2WVR2D2UMs6dzzmX\nu0XRrNsLFX16WKbo+Od4SVurbrdC0s+q1h0uaY6iOSdeUjR1+9cVzUWxd8XtZkuaVZV7qqIZhbfE\ntz8/ZZy7Sbpe0Yt3k6QFkk6q9/OXx6VANX0hvl1ry4H1fh7ztBShpvpgHppWl3o/h3lbClLTMxXt\nDXk1vv2b8c+n1Pv5y+NShJomjHu2pD/U+/lrbbF4gKVmZjcq+mXp6hrhATcAalo+1LR8qGn5UNOc\nnePUHsysqernXop2Nz7RqEUuOmpaPtS0fKhp+VDT1uXxHKdazTezOYqmfd9P0oWK5veY4EtCrlHT\n8qGm5UNNy4eatqKMjdPDir6A98uKTl57UtJo59zcuo4KtaCm5UNNy4ealg81bUVDnOMEAADQHoL2\nOMXHNYdLWqkPvosN9dck6SBJM110SWcwappb1LR8qGn5UNPyCa9p4GWBoxTtpmPJ5zIqwyWq1DTf\nCzUt30JNy7dQ0/ItqTUNPcdppSTdcccdGjhwYGBKx9qyZYs3fvvttyfG7rjjDm/uiSee6I1fddVV\n3vjO0tLSovPOO0+K69NGK6V81bQWI0aM8MZ79uzpjf/oRz9KjHXuvPO+cL1sNV26dKk33tzcnBi7\n7rrrvLk7sy61KGJN3347eZ7Xe+65x5vre3/t1s3/bSinnXaaN3766acnxvbZZx9vbnsqYk1rce+9\n9ybGpkyZ4s2dOXOmN56X13FbahraOG2WpIEDB2rQoEHZR9aONm/27+GcNWtWYqxTJ//D7t27tzee\nl+egQpbdvbmraS3SXnxdu1Z/peCOjjzyyMRYU1NTYqwDlaKmW7du9ca7d0/+RgZfTaS61aUWhanp\n+vXrE2NPPPGEN9f3/pr2Ou3Tp483fthhhyXG+vbtmxjrQIWpaS0WLFiQGEv7e1rA13FqTUs3jxMA\nAEBHoXECAAAIROMEAAAQiMYJAAAgUGFnDr/kkku88alTpybGbrnlFm/u5MmTvXHfiefDhg3z5iK7\nRYsWJcaWL1/uzU2L+y42yOHJi4UxfPhwb9x3teP06dO9uWeddVamMSHd66+/nhibMWOGN/faa69N\njK1du9abO27cOG/c9/uS9jcBydIutvL9Taz1ysAivveyxwkAACAQjRMAAEAgGicAAIBANE4AAACB\naJwAAAAC0TgBAAAEonECAAAIlOt5nHxfNOmbp0mSxo4dmxhLm+8jba6R+fPnJ8aYx6njnHPOOZlz\nR44c6Y37vmwW2aXN8eKbEy2t3szj1HEGDBiQGJs7d64311fTr3zlK97cHj16eOMjRozwxpHNlVde\n6Y37/iY+/vjj3ty0L272vTc3Nzd7c+uFPU4AAACBaJwAAAAC0TgBAAAEonECAAAIROMEAAAQiMYJ\nAAAgUK6nI2hqasqcO2bMmMy5PXv2zJwLv82bNyfG0i6JXb58eXsPB+3AN23IMccc4831vcaXLFmS\neUyon2nTpmXOXbFihTfOtCHZ3XPPPYmxyZMne3PvvvvuxFivXr28uevWrfPGBw8e7I3nEXucAAAA\nAtE4AQAABKJxAgAACETjBAAAEIjGCQAAIBCNEwAAQCAaJwAAgEC5nsfpxRdfrPcQ0M7WrFmTGEub\nw+Xggw9OjKXN8XTUUUf5B4bMfHPrjBs3LvN202rqmxNMqm0eOGTnmxOof//+3tyxY8d6483NzZnG\nBGnZsmWZc2+66abEWNr8e2mOPvromvLrgT1OAAAAgWicAAAAAtE4AQAABKJxAgAACETjBAAAEIjG\nCQAAIBCNEwAAQKBcz+PUr1+/zLl//etfE2Np87/8/ve/98YnTJiQaUyQ+vbtmxi7//77vbmLFi1K\njA0ZMsSb65tbRpK+853veOPIxjfHkyTNmjUrMdajRw9vLvM05ZOv5mlztaXN83TFFVckxgYMGOAf\nWIP75je/mRhbt26dN3fq1KmZc33z70nM4wQAAFBqNE4AAACBaJwAAAAC0TgBAAAEonECAAAIROME\nAAAQiMYJAAAgUK7ncfLN0zJy5Ehv7nXXXZcYS5srJG3+GN9cROg4e+21V+bcnj17tuNIEOraa6/1\nxseNG5cYS3sdpm3bV/NRo0Z5c7t16yYz896mzHxz3S1ZssSb65tD77vf/a43N21OoFdeeSUxxjxO\nfr6/p5MmTfLmTpw4MTHWpUsXb+6IESP8Aysg9jgBAAAEonECAAAIROMEAAAQiMYJAAAgEI0TAABA\nIBonAACAQLmejsDnrrvu8savvPLKxNiCBQu8uffee2+mMaFj9evXLzE2dOhQb+68efO8cd/l177L\neOE3evRob3zFihWJscGDB3tzp02b5o3vs88+ibFhw4Z5c7t16+aNl53v9eCb6qVWab8vaXVDx/D9\nPU2bNmTMmDHtPZy6Y48TAABAIBonAACAQDROAAAAgWicAAAAAtE4AQAABKJxAgAACBQ6HUGTJLW0\ntHTgUNpmy5Yt3vjrr7+eGNuwYYM3N+3bv33b3pkq6pHlevnc1TSNr+ZpNU3z1FNPJcY6d+5c07bb\nomw1Xb16tTf+5ptvJsZeeuklb25azXfbbbfE2NKlS1O3bWbe24QqYk3ffvvtxNj69eszbzetZr7f\nB0lavHhx5vtuT0WsaS18f/Pee+89b27aa23jxo2ZxtTe2lRT51zqImmUJMeS22VUSB2paaEWalq+\nhZqWb6Gm5VtSa2pxIb3MrJek4ZJWSkqeGQ07W5OkgyTNdM6taUsiNc0talo+1LR8qGn5BNc0qHEC\nAAAAJ4cDAAAEo3ECAAAIROMEAAAQiMYJAAAgEI0TAABAIBonAACAQDROAAAAgWicAAAAAtE4AQAA\nBKJxAgAACETjBAAAEIjGCQAAIFBhGiczu8rMtlWtW2lmzRm3N8fMZrfP6JAFNS0falo+1LR8qGlt\nCtM4SXLxUmlbK+vasr33f3HMbH8zG29mh4ckm9kJZratlWWrmQ3JOKZGk6uaVuQNMrMHzWyNmW00\nsyVmdmnGMTWaXNXUzKYmvE63v1b3zziuRpKrmsY5HzOzu83s5fg12mJm48ysS8YxNZo81vQoM/ul\nmb1lZn81s5lmdkTG8XSoTvUeQI0GqKJYbfSZqp/7SBov6QVJz7RhOzdK+n3Vuuczjgl1rqmZfVbS\ng5IWS7pG0gZJB0s6IOOYUN+a/kjSo1XrTNKPJa1wzv0l47gaXd1qamYHSFokaZ2kmyWtlXSspKsl\nDZL0+YzjanT1rOkgSU9IeinO20XSJZLmmNkQ59yyjOPqEIVunJxzf6sh972qVZZxU791zv1P1nFg\nR/WsqZntKek/JT3knDsj6ziwo3rW1Dn3O0m/22EDZsdJ2l3StKzjanR1fu89X9Jeko51zj0br7vV\nzHaR9H/MrJtz7q2s42tUda7pBEmbJB3jnFsvSWY2TdKfJV0nKVfvx7k8VGdmx5vZIjN7x8yWmdmY\nhNt96JismR1uZo+b2aZ4N+6VZjY63jV/YMXt5pjZr+P/nyBpoaLdjbdV7MY/P3C8XeMXLRIUpKbn\nStpH0pXxNnY3s6wNdekVpKatOVfRJ+u72phXegWp6Z7xv6ur1r+mqK7vtu1Rl1tBanq8pMe2N02S\n5Jx7TdLjkk41s90zPwEdIHd7nMzsUEkzFb0ovitpV0lX6cMvEqnqeKyZ9ZE0W9JWSRMVdbAXKXoh\nVR+7rfy5Jb6vaxTtwn8iXj8vYMhTFb2Qt5rZE5L+zTn3ZEBewyhQTYdJ+qukvzOzByX9g6SNZna7\npMucc1t8j7ORFKim1ePupOjT61zn3EuheY2gQDWdI+kKSc1mNl7SGknHSfqqpB84597x5DaUAtW0\ns6TW6rZJ0m6SDlXUjOWDcy5Xi6T7JW2U1Ldi3QBJf5O0teq2L0hqrvj5JknvSTqsYl13SW8qKv6B\nFetnS/p1xc9HKfq0cn7gOI+VdK+kCySdKun/Kvpl3CjpiHo/j3laClTTpxWd07RB0vcljVR0Dts2\nSdPq/TzmaSlKTVsZ96lx/ph6P4d5W4pUU0V7hTfGedvi+7im3s9h3pai1FTSHxQ1XFaxbldJK+P7\n+ny9n8vKJVeH6szsI5I+K+l+59yq7eudc88p6prTDJc03zm3pCJ3vTrgXAbn3Hzn3JnOuducc79w\nzt2gqJmSpO+19/0VVZFqKqmrpC6SbnPOXeace8A59w1Fn5rONrODO+A+C6dgNa02StEn5vt2wn0V\nRgFrulLRYZyLJP2zpGZJV5rZJR10f4VTsJpOUbSHv9nMBsZ7ym6XtF8cz9XVkrlqnCTtregJau2q\ntOcC8vsl5O6Uq9ycc8slTZf0j5wb874i1XT7ruK7q9bfqehkx2MFqVg1fZ+Z7SHpdEm/dM6t68j7\nKqDC1NTMzpb0E0n/4pxrjj/gfFnRhR3Xm1mP9r7PgipMTZ1zP1Z0Evg5kpYq2gP1UUk3xDfZ0N73\nWYu8NU5l8LKiY7J71HsgaLNX439fr1q//XwA3pCL7fOK/pBwNV2xXSxpsfvwVBIPKrpa8sidPyTU\nyjk3TtK+ik4UP9w590lF0xJI0dV1uZG3xukNRZ/6/76V2CEB+S9K+lgr61vbXrWsE39VO1jSZudc\nrjrkOipSTbef1N+3an2f+N832ri9sipSTSudq+iT60M1bKOsilTTffXBH9RKu8b/5u6ipzopUk2j\nJOfecs7Nc84tjVd9RtIr7oNpJ3IhV42Tc26bomOvIy2a5EySZGYDFR2rTTNT0rFWMVupmfVUdF5D\nmo3xv91DxmpmvVtZd4Sk0xR2/LghFKmmik72N0n/UrX+y4pOppwTuJ1SK1hNt2+/t6KrJv/HObe5\nLbmNoGA1/bOkI82s+o/6KEUnJLdlAuPSKlhNP8TMzpI0WNGFOrmSx858vKTPSfqtmU1R9CniUkl/\nlJQ2ffsNks6T9JiZ3ayoeBcp6px7yN8FL5e0XtJXzWxDnPs759zKhNvfY2bvKLrEcrWkTyj6A7tB\n0rdTxtloClFT59zT8Twmo81sV0Unn/6jpC9Ius5F84ogUoiaVjhb0V4KDtMlK0pN/1/FOH+oaDqC\n0xSdzPxTXqc7KERNzex/K5rC4FeK6nmsoivWH1F0dV++1PuyvtYWRcc4FyrazbhMUUMyXh++fHKF\npJ9VrTtc0Z6BTYqmb/+2pK8ruqRx74rbzZY0qyr3VElLJG2Jb594KaWiX775inaHbpH0iqTbJPWv\n9/OXx6UINY1vv4ukcfE4Nis6ifLr9X7+8rgUpaZxzjxF57BZlsfaKEtRaqpoT8QvJK2KX6ctiuZ2\n+ki9n8O8LUWoqaT+kmYoOr90k6ITxP9NUqd6P3+tLRYPutTM7EZFvyxdXSM84AZATcuHmpYPNS0f\napqzc5zag5k1Vf3cS9HuxicatchFR03Lh5qWDzUtH2raujye41Sr+WY2R9Gu2/0kXajoK1Em1HNQ\nqAk1LR9qWj7UtHyoaSvK2Dg9LOmLinYlOkWXmI92zs2t66hQC2paPtS0fKhp+VDTVjTEOU4AAADt\noXTnOAEAAHSUoEN18QlhwxV9sSKTx+VHk6SDJM10zq1pSyI1zS1qWj7UtHyoafmE1zRwHohRio5v\nsuRzGZVhbg9qmu+FmpZvoablW6hp+ZbUmoaeHL5Sku644w4NHDgwMKV2S5cuTYw1Nzd7c9euXZsY\ne+aZ2mbknzNnTmJszz33rGnbbdHS0qLzzjtPiuvTRiulnV/TWtx7772JsSlTpnhzZ870fwtO586d\nM42pvZWtplu2bPHGp0+fnhhLq+npp5/ujY8dO9Yb31nKVtPJkyd744ceemhi7K677vLmHnfccd74\nRRdd5I3vLGWr6cKFC73xiRMnJsZ+8IMfeHMPOuigLEPa6dpS09DGabMkDRw4UIMGDco+sjbaunVr\nYqx7d/9X4Lz77rvtPZz3HXHEEYmxtHF1kCy7e+tS01osWLAgMdapk/9X+cgj/V+Y3tTU5I3XQSlq\nunmz/2E89dRTibG0mu67777eeF6egwqlqGna8/7Rj340Mda1a1dvbp8+fbzxvDwHFUpR03Xr1nnj\nvg+Wn/jEJ7y5AwYMyDSmOkqtKSeHAwAABKJxAgAACETjBAAAEIjGCQAAIFCuv3LlP/7jPxJjDzzw\ngDe3R48eibFbbrnFmzts2DBvvE4ngDe8Rx99NDHWs2dPb24OT/4ujVWrViXGzjzzTG9uS0tLYiyt\npr4r8iRp0qRJ3jiy8b23Sv4rtPbZZx9vbtoVe5deemlijPfl7KZNm+aNL1++PDH2k5/8xJtbxtch\ne5wAAAAC0TgBAAAEonECAAAIROMEAAAQiMYJAAAgEI0TAABAoFxPRzB48ODE2G9+8xtv7qc+9anE\n2IUXXujN5dL1+vBd1i75p6C4++6723s4CPTqq68mxo455hhv7ty5cxNjl19+uTd3xYoV/oGhQ5xx\nxhne+PXXX58Y69+/vzc3baoDphzoGL6/tZL/723aFBLjxo3zxotYU/Y4AQAABKJxAgAACETjBAAA\nEIjGCQAAIBCNEwAAQCAaJwAAgEA0TgAAAIFyPY+Tz/LlyzPH0+aAev755zONCbV59tlnM+cOHz68\nHUeCtjj66KMTY3369PHmLlq0KDE2depUb+7o0aO98fXr1yfGijh3TF7069fPG/fVbezYsd7cKVOm\nZBoTapM2t+Gjjz6aGDvssMO8uWk1b25u9sbziD1OAAAAgWicAAAAAtE4AQAABKJxAgAACETjBAAA\nEIjGCQAAIBCNEwAAQKBcz+Pkm1tiwIABmbd70kknZc5Fx3nzzTcz5/bo0cMbHzp0qDd+4403JsZ8\n8xTB74ADDuiwbU+ePNkbX7FiRWLs/vvvb+/hNIxzzjnHG/e91saMGePNbWpqyjQm1Cbtea/l9ZI2\nR9SqVasSY3379s18vx2JPU4AAACBaJwAAAAC0TgBAAAEonECAAAIROMEAAAQiMYJAAAgUK6nI/Bd\nIjls2DBv7qJFizLfr+/ySCm/l0gW3cUXX5w5d8KECTXdt+8S6+eff76mbZfd5s2bE2O33HKLN/fR\nRx9NjC1ZssSbO3bsWG98xIgR3jg6xqxZsxJjaVMZME1E+aS9r48bNy4x1tzc3N7DaRfscQIAAAhE\n4wQAABCIxgkAACAQjRMAAEAgGicAAIBANE4AAACBaJwAAAAC5XoeJ9/8MGlzvAwfPjwxNnToUG8u\n8zTVR1pNTzjhhMzbvvTSS71x31wi69ev9+Z269ZNZpZpXGXgm2/tkksu8eYuX748MbZ69Wpvbtq2\nkZ3vvbd///6Zc9Ne46gPX80k6cUXX8y87RUrVnjjU6dOTYxNnjzZm1uv9172OAEAAASicQIAAAhE\n4wQAABCIxgkAACAQjRMAAEAgGicAAIBAuZ6OwHcJpG+6AUlat25dYuzhhx/OPCZ0nLRpICZOnJgY\nu/jii725vukGJGn06NGJse7du3tzkZ3vdXryySfvxJGgkm+KCV/NJGnw4MGJsbvuuivzmNBxpk+f\n7o2fffbZmbedNv2P773X93tYT+xxAgAACETjBAAAEIjGCQAAIBCNEwAAQCAaJwAAgEA0TgAAAIFC\npyNokqSWlpYOHMqHrVy5MjH23nvvZd7uH/7wB298zz33zLztnamiHlmu2axLTWvxwgsvJMZq+X2Q\npDfffDMxtnjx4pq23RaNVlPf8552KfLOrEstylZTX80kacuWLYmxZ5991pu7yy67ZBrTzla2mvre\nW2u1YcMGb9z3+/TUU095c3fbbTeZWaZxVWtTTZ1zqYukUZIcS26XUSF1pKaFWqhp+RZqWr6FmpZv\nSa2pxYX0MrNekoZLWilpc2oCdpYmSQdJmumcW9OWRGqaW9S0fKhp+VDT8gmuaVDjBAAAAE4OBwAA\nCEbjBAAAEIjGCQAAIBCNEwAAQCAaJwAAgEA0TgAAAIFonAAAAAL9fyAcYwWYh51HAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114eb56a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pylab as pl\n",
    "\n",
    "num_rows = 4\n",
    "num_cols = 5\n",
    "\n",
    "fig, ax = plt.subplots(nrows=num_rows, ncols=num_cols, sharex=True, sharey=True)\n",
    "ax = ax.flatten()\n",
    "for index in range(num_rows*num_cols):\n",
    "    img = digits.images[index]\n",
    "    label = digits.target[index]\n",
    "    ax[index].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "    ax[index].set_title('digit ' + str(label))\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Data sets: training versus test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training: 1257, test: 540\n"
     ]
    }
   ],
   "source": [
    "if Version(sklearn_version) < '0.18':\n",
    "    from sklearn.cross_validation import train_test_split\n",
    "else:\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    \n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=1)\n",
    "\n",
    "num_training = y_train.shape[0]\n",
    "num_test = y_test.shape[0]\n",
    "print('training: ' + str(num_training) + ', test: ' + str(num_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[119 133 128 119 120 135 130 122 128 123]\n",
      "[59 49 49 64 61 47 51 57 46 57]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# check to see if the data are well distributed among digits\n",
    "for y in [y_train, y_test]:\n",
    "    print(np.bincount(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We first write a scoring function for clustering so that we can use for GridSearchCV.\n",
    "Take a look at use_scorer under scikit learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "## Note: We do not guarantee that there is a one-to-one correspondence, and therefore the toy result is different.\n",
    "##       See Explanation for more information\n",
    "def clustering_accuracy_score(y_true, y_pred):\n",
    "    n_labels = len(list(set(y_true)))\n",
    "    n_clusters = len(list(set(y_pred)))\n",
    "    Pre = np.zeros((n_clusters, n_labels))\n",
    "    Rec = np.zeros((n_clusters, n_labels))\n",
    "    F = np.zeros((n_clusters, n_labels))\n",
    "    w = np.zeros((n_clusters))\n",
    "    F_i = np.zeros((n_clusters))\n",
    "    P = np.zeros((n_labels))\n",
    "    C = np.zeros((n_clusters))\n",
    "    \n",
    "    for i in range(n_clusters):\n",
    "        C[i] = sum(y_pred == i)\n",
    "    for j in range(n_labels):\n",
    "        P[j] = sum(y_true == j)\n",
    "    \n",
    "    for i in range(n_clusters):\n",
    "        F_i_max = 0\n",
    "        for j in range(n_labels):\n",
    "            if (C[i]):\n",
    "                Pre[i][j] = sum(y_pred[y_true == j] == i) / C[i]\n",
    "            if (P[j]):\n",
    "                Rec[i][j] = sum(y_true[y_pred == i] == j) / P[j]\n",
    "            if (Pre[i][j]+Rec[i][j]):\n",
    "                F[i][j] = 2*Pre[i][j]*Rec[i][j]/(Pre[i][j]+Rec[i][j])    \n",
    "            F_i_max = max(F_i_max, F[i][j])\n",
    "        F_i[i] = F_i_max\n",
    "        w[i] = sum(y_pred == i) / len(y_pred)\n",
    "    return F_i.dot(w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 1.0 , should be 1\n",
      "accuracy 0.833333333333 , should be 0.8333333333333334\n",
      "accuracy 0.75 , should be 0.6666666666666666 , this will be explained in the following content\n"
     ]
    }
   ],
   "source": [
    "# toy case demonstrating the clustering accuracy\n",
    "# this is just a reference to illustrate what this score function is trying to achieve\n",
    "# feel free to design your own as long as you can justify\n",
    "\n",
    "# ground truth class label for samples\n",
    "toy_y_true = np.array([0, 0, 0, 1, 1, 2])\n",
    "\n",
    "# clustering id for samples\n",
    "toy_y_pred_true = np.array([1, 1, 1, 2, 2, 0])\n",
    "toy_y_pred_bad1 = np.array([0, 0, 1, 1, 1, 2])\n",
    "toy_y_pred_bad2 = np.array([2, 2, 1, 0, 0, 0])\n",
    "\n",
    "toy_accuracy = clustering_accuracy_score(toy_y_true, toy_y_pred_true)\n",
    "print('accuracy', toy_accuracy, ', should be 1')\n",
    "\n",
    "toy_accuracy = clustering_accuracy_score(toy_y_true, toy_y_pred_bad1)\n",
    "print('accuracy', toy_accuracy, ', should be', 5.0/6.0)\n",
    "\n",
    "toy_accuracy = clustering_accuracy_score(toy_y_true, toy_y_pred_bad2)\n",
    "print('accuracy', toy_accuracy, ', should be', 4.0/6.0, ', this will be explained in the following content')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explanation\n",
    "\n",
    "I adopt a modified version of F-value selection, that is, for each cluster, select the best label class with highest F-score. This accuracy calculating method supports the condition that number of clusters not equal to number of labels, which supports GridSearchCV on number of clusters.\n",
    "\n",
    "Formula: Let $C[i]$ denotes cluster $i$ and $P[j]$ denotes label $j$. Then for each $(i, j)$, we have<br><br>\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{Precision}[i][j] & = \\frac{\\left|C[i]\\cap P[j]\\right|}{|C[i]|} \\\\\n",
    "\\text{Recall}[i][j] & = \\frac{\\left|C[i]\\cap P[j]\\right|}{|P[j]|} \\\\\n",
    "\\text{F-value}[i][j] & = \\frac{ 2 \\times \\text{Precision}[i][j] \\times \\text{Recall}[i][j]}{\\text{Precision}[i][j] + \\text{Recall}[i][j]}.\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Then for each cluster, we search the best F-value for each label, that is,\n",
    "$$\\text{F-value}[i] = \\max\\limits_{j} \\text{F-value}[i][j].$$\n",
    "We also store the weight $w$ for each cluster, that is,\n",
    "$$w[i] = \\frac{|C[i]|}{n}$$\n",
    "\n",
    "Hence the final score is simply the dot product of $\\text{F-value}$ and $w$, which is the weighted F-score.\n",
    "\n",
    "Again, since this accuracy calculating method supports the condition that number of clusters not equal to number of labels, we __do not__ guarantee that there is a bijection between clusters and labels, therefore there are cases that some labels have no corresponding clusters, as the second toy example shows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Build a pipeline with standard scaler, PCA, and clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.decomposition import KernelPCA\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.metrics import make_scorer\n",
    "from scipy.stats import mode\n",
    "\n",
    "pipe = Pipeline([('scl', StandardScaler()),\n",
    "                 ('pca', KernelPCA()),\n",
    "                 ('km', KMeans(random_state=1))])\n",
    "\n",
    "# map cluster number to acutal label \n",
    "def cluster_mapping(y_true, y_pred):\n",
    "    mapping = {}\n",
    "    n_labels = len(list(set(y_true)))\n",
    "    n_clusters = len(list(set(y_pred)))\n",
    "    Pre = np.zeros((n_clusters, n_labels))\n",
    "    Rec = np.zeros((n_clusters, n_labels))\n",
    "    F = np.zeros((n_clusters, n_labels))\n",
    "    P = np.zeros((n_labels))\n",
    "    C = np.zeros((n_clusters))\n",
    "    \n",
    "    for i in range(n_clusters):\n",
    "        C[i] = sum(y_pred == i)\n",
    "    for j in range(n_labels):\n",
    "        P[j] = sum(y_true == j)\n",
    "    \n",
    "    for i in range(n_clusters):\n",
    "        F_i_max = 0\n",
    "        F_i_max_label = 0\n",
    "        for j in range(n_labels):\n",
    "            if (C[i]):\n",
    "                Pre[i][j] = sum(y_pred[y_true == j] == i) / C[i]\n",
    "            if (P[j]):\n",
    "                Rec[i][j] = sum(y_true[y_pred == i] == j) / P[j]\n",
    "            if (Pre[i][j]+Rec[i][j]):\n",
    "                F[i][j] = 2*Pre[i][j]*Rec[i][j]/(Pre[i][j]+Rec[i][j])    \n",
    "            if (F_i_max < F[i][j]):\n",
    "                F_i_max_label = j\n",
    "                F_i_max = F[i][j]\n",
    "        mapping[i] = F_i_max_label\n",
    "    return mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Use GridSearchCV to tune hyper-parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.824823005662\n",
      "{'pca__n_components': 15, 'scl__with_mean': True, 'pca__kernel': 'cosine', 'km__init': 'k-means++', 'km__n_clusters': 11, 'scl__with_std': False}\n"
     ]
    }
   ],
   "source": [
    "if Version(sklearn_version) < '0.18':\n",
    "    from sklearn.grid_search import GridSearchCV\n",
    "else:\n",
    "    from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "pcs = list(range(1, 60))\n",
    "kernels = ['linear', 'rbf', 'cosine']\n",
    "initTypes = ['random', 'k-means++']\n",
    "clusters = list(range(10, 20))\n",
    "tfs = [True, False]\n",
    "\n",
    "param_grid = [{'pca__n_components': pcs,\n",
    "               'pca__kernel': kernels,\n",
    "               'km__init' : initTypes,\n",
    "               'km__n_clusters' : clusters,\n",
    "               'scl__with_std' : tfs,\n",
    "               'scl__with_mean' : tfs}]\n",
    "\n",
    "gs = GridSearchCV(estimator=pipe, \n",
    "                  param_grid=param_grid, \n",
    "                  scoring=make_scorer(clustering_accuracy_score), \n",
    "                  cv=10,\n",
    "                  n_jobs=-1,\n",
    "                  verbose=False)\n",
    "\n",
    "gs = gs.fit(X_train, y_train)\n",
    "print(gs.best_score_)\n",
    "print(gs.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.844\n"
     ]
    }
   ],
   "source": [
    "best_model = gs.best_estimator_\n",
    "\n",
    "print('Test accuracy: %.3f' % clustering_accuracy_score(y_test, best_model.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Visualize mis-clustered samples, and provide your explanation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkoAAAGGCAYAAACE4a7LAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xm8FMW5//HPEyCgorIoGtCgoiLX7ece0QRvcEk0iUvi\nBsYtK6gxl2hM3JdIrlExGtxuIprEBTCJolFDEiIat7gHF0QFwR0EVBaFCNbvj6qj43T1OT1nZrqH\nOd/36zWvA8+prqkzz3RPTXVVtznnEBEREZGkTxXdABEREZFGpY6SiIiISAp1lERERERSqKMkIiIi\nkkIdJREREZEU6iiJiIiIpFBHSURERCSFOkoiIiIiKdRREhEREUmRa0fJzB40swvyfE6pL+W0+Sin\nzUc5bT7KaX6q7iiZ2Rpmdo6Z3WVmC8zsQzM7MqX4BcBxZtYnQ72rmdlZZvaFGrRxLzO7z8yWmtlC\nM7vZzPpXW2+VbRphZhPNbE54zcYV2Z5SZrajmY01s6fNbElo4wQz2yxSXDn17Tkq5DHtcXhRbQvt\n+6/wfpsZXrO3zOweM/tKpHhROd3BzP5sZm+Y2WIz+7eZnWBmhY18N/J+Ws7MTgttnBb5dSE5Lav3\n16F9t9Wy3na0o2FzamZDUo4fK81s57Liuee0lePcyiztqBcz62Nm15rZXDN7z8weM7Nv1Kr+zjWo\nYx3gDGAO8CSwRytlJwGLgJHA2W3UuzpwFuCAe9vbuPBBcCvwKHAKsBbwQ+CfZradc25Be+uu0o+B\n7sDDwPoFtSHNKcBg4GZgGr59JwCPm9kuzrlnS8oqp949wBGR+ChgG2BKvs1J6I9/v10HvI7PxdeB\n28zsu86535SULSKn2wP3A88D/wu8B3wZuBTYBPif9tZdpUbeTz9iZv2AnwJLUorkntOy9u0IHAW8\nX4v6qrQq5PSX+ONbqRfL/l9UTh3+M392WfydKuttFzNbE3/sWBf/us0FDgEmmtkw59z4qp/EOVfV\nA+gC9An/3gH4EDiylfKXAbMy1LtOqOvMKtv3DDAD6FQS2wZYAVxY7d9fRbs2LPn3YmBcUW2JtO1z\nQOey2Kb4g9zvlNPM7ewGvAvcVXRbUtpnwBPAsw2Q0/8L76+1y+JTgbcLfI0adj8ta+d44G/A3cC0\nlDK55rSszvuBXwMvAbcV/Fo1bE6BIeG1Pyhj+bz306OAlcD2Rb9WJW06ObRpSEnMgH8Br5V/lrXn\nUfWQtnPuA+fcvAo2+RvQ38y2TSsQTqHMw/dczy4Z3jsz/L6zmQ00s1a/DZhZT2AQcItzbmVJm6cB\n04HD2mqsmU01s2lmtr2Z3R+G9WaZ2fciZTc0s4Ft1Rna8EqWckVwzj3knFtRFnsR30EZFNlEOY37\nGrAmcEM7t68r548orwA9Ir/OLafBmsAy59y7ZfE3yTAK0RH30xbhdMpB+FHV1uSd05Y6jwS2BE7L\nuk3YrsPmFMDMuptZpzaKFZLTkvZV1IeoU053B95yzt3TEgjHton4EcMhlbQxpohz/4/he3u7tVLm\nLeD7odyf8Kc0jgj/BuiH/1Ac3cZzdQ0/Ywfa94C+Gc6rOqAXcAd+KPRk/IfLlWZ2dFnZ34d2Nav1\ngPmRuHIaNzy06ZZ2bl9zZra6mfU2s03M7H/wp7f+HimaZ07BjxytZWb/Z2ZbmNlnzez7wAEZt++Q\n+2n4oLoM+LVz7pk2iuedU8ysO/5U6vkVfqGGDprT4Fr8abVlZvYPM9shpVzuOQ31TA3te8/MJpnZ\nphm3rUdOu5L+eWD4M13VqfEQWJun3kK5ZcDYNsr0JmWoED/fYiVwTRt1GLAQ+Guk7sWhju3aqOPu\nUO7EklgX4HHgDT55+uduYEU7XreGGv5NaeMRIR9HKaeZXq+e4TW5sejclbXrypCDD/GnKidQdror\n75yGsi0f+MtL2vcf4LsZ/64OuZ8Cx4X9oVfJ3xY99ZZ3TkP5C/Fza7qE/2c+9dYRcwrsih8JORr4\nCn4+1TxgKbBt0TkFDgauwX8efA04Bz8vbi7Qr4ic4ucxfkDJKdUQvyk816XV5qWo1SRv48+Ztotz\nbo5zrpNz7lttlHPA1cBQMxttZpuGnvkEfHIAVsvwlCvwcyha6v0g1NuHkt6qc+6/nXO1mCDfUMxs\nC2Asfp7B71KKKaefdHBoT6OddrsE2BM4ErgT6MTHo3TlcslpKPshMBP4C/BN/GTM24GxZva1jE/Z\nofZTM+uF/6A61zm3MONmueXUzDYHfgCcFHLRHh0qp865B51zhzjnrnPO/dk59wt85wng5ymb5bmf\n3uyc+5Zz7nrn3G3OubOAfcLzZz21Wuuc/gbfCbzZzHYNo+U/xY9GQ7bPg1YV1VEy/BBcHs7E94BP\nxq+oeRjf+2xZEpq2SqTU68658qG95/F/x0a1aWZjMrP18MOkbwMHh45KtCjKaanh+G/6f6lBXTXj\nnHveOfePcKBrmUOVtlw7t5ya2U/w354Pd87d4Jz7g3Pu68B9wOUZ50J0tP30fGAB/ktMVnnup5cC\n9znnbq2ijo6W0wTn3Ez8Crf/NjOLFMkzpwnOufvxE6f3zLhJTXPqnHsKOBy/OvY+/Ajm8cCJoc4s\nnwetKqqj1IP4XJeac36y+XeBvsDngYHOuS+HNnxIcsmlBGa2Fv6Dfi3gS865N1sprpwGZrYhfoLh\nRFcy4bxB/QHYyeLXyMotp8AI4B/OuffK4rfh87xRTu1YJYQ5Id/Bn67sZ2b9zWwj/ErLLuH/PSOb\n5pJTM/sifqThstCWlvZ1BlYL/1+z3u1oIq8AnwbWiPwuz/00zSv4uUeFcM79CX+c2Bm/ars//jQv\n+E5YVXIfqjSzvviEtzVJq6Y9ZOfcW/hJbS0TIIcAD0UOzDF9zWy1sl7wwNDG2bVsZ6Mws67An/GX\nBRjqnJvRSlnl9JOGhZ+NdtotpmVYeu3SYAE5XQ9/GrBcy+nULMeqjrSf9sN/W74M+FXk97PwIzqj\nWgI553TDUE/5QgaHb/ss/LWxLmujno6U09YMwK8K/cToSFHH3ohNCMfiDOqSU+dXaj/W8n8z2yvU\nGVusUpEiRpR2wDf+gTbKtXzYJZYut3c5Y4mT8csGL85YvjN+1UDL83cBvod/Y5Qmppql5A0jdDom\nArsA33DOPdzGJsrpJx0OvOyca+v1yI2ZrRuJdebjiwA+W/brvHP6PLBX6ShIeB8eip9wOzNDHR1p\nP30aODA8Dih5PIO/+O8B+NPTpfLM6ZRI2w7Aj3w8Ev59ext1QMfKKWaWmGsUlv5/FZgc2STX/TSl\nffuGdtzV1vZB3XMaRsi/B9zu/KVtqlKTESUzOw6fgH4h9LVw+gHgMufc4pLie+M/RJ5srU7n3DIz\nexY41MxewM/3eNr5JbAtyxmvA45to23D8Vcgvhd/rnIv4Bv45bRZz52/Dvw4DB0/j79WzzbAd8pO\nrfwe+AIZOqDmry69Lf5bYRdgWzNrmQw3yTn3dMa21cMY/I55G7BOeA0/4pwrHylRTj9u21ahnqxL\nbfNydTiVei/+Imzr4+dRDQRGRUbhcs0pfgn574GHzazl4pPDgO2A0zKewuww+6nzV59PzC0Ll3xw\nzrlYJyS3nDrnXgVejbTvUmBuSvtiOkxOgwlm9j6+4zMPf/2p7+CPcz+NlM97P33AzJ7AL+1/F99B\nOgbfOU+bbF6uHjl9Bn8niZfxo1vfx3fKR2RsU+vKl8G154E/F7gy5fHZknKGP0ifnbHeXfATdd8P\ndZ3pKl/OuBN+meF8/BLLx4FvV/C33Y2/jcd2+FVfS/HDxt9PKZtpiSr+Ohlpr1mrl1eo94OPl3BG\nH2VlldNPlh8d/o4ti8xhpF2H4L+Rvo5fgj8//H+/SNnccxrK7wX8A7/U+H38LZEy5bUj7qetvA7/\nbpScRuqdhe+MKKfxth0PPIgfXVmO72xeB2zSCDkFzsWP+izEX5bgJfyp33ULzukN+NN27+PnS40F\n1qlVXiw8SS7M7ADgemCAc25ubk9cBTO7G+jtnNum6LY0IuW0+SinzUc5bT7KaX7ynqP0Y+BXq0pS\nJRPltPkop81HOW0+ymlOcl315pwbnOfzSf0pp81HOW0+ymnzUU7zU9R1lFY1hV3MS+pGOW0+ymnz\nUU6bzyqX01znKImIiIisSjSiJCIiIpKi6jlKZtYbf6n62fjlglKdbvjbNUx2/lopuVNOa045bT7K\nafNRTptPTXJai8nc+7Bq3KphVTMcuLGg51ZO60M5bT7KafNRTptPVTmtRUdpNsD111/PoEGDalBd\nbcybNy8RO+WUU6Jlp02bloh99atfTcTOPvvsqtvVlunTp3PEEUdAsfcxmg3F5jSWvyuuuCIRmzp1\nanT7jTfeOBG77LLkbaXWXLP+9+VUTr3ly5cnYj//efJivk888UR0+3fffTcRmzhxYiLWp0+fdrSu\nMsppumeeeSYRu/nmm6Nl8zimZqWcesccc0xV2/fqlbw37gknnJCIbbTRRlU9Txa1ymktOkrLAAYN\nGsT2229fg+pq47XXXkvEunfvnnn7ddZJ3NIm77+vyGHXwnMay18sJ507x9/CsVxvu+22iViPHolb\nH9VTh87psmXJPz+W065du0a3j+V66623TsT69euXiNVRh85pzMqVybvNxPIMuR9Ts+rQOa3kczIm\ndkzdcsstE7GBA3O9NV9VOdVkbhEREZEUuV5wsl5mzJiRiO26666J2AEHHBDdfvjw4YnYcccdl4id\nd955idjaa69ddQ+8o3vnnXcSsdhIQSx/P/vZz6J1jhkzJhGbPDl58+1DDz00SxOlQrHRo6FDhyZi\nDzyQvOl52n56663J+x1PmjQpERs5cmSWJkqdjB8/PhGLHaOleLH9NLZPPvzww5nrjB1nV3UaURIR\nERFJoY6SiIiISAp1lERERERSqKMkIiIikkIdJREREZEUq9yqt9gKqdgKt1GjRmUqB3DmmWcmYj17\n9mxH66Q9brwxecHU2EXLxo0bl4jF3g8AN9yQvLjtPvvs047WSXvEchVbTfPcc88lYq+++mq0ztiq\nt5122qkdrZN6iq04TTNlypRELLY6Uupjzpw5idjgwYMTsdh+lnbsja1Y7tatWzta1zg0oiQiIiKS\nQh0lERERkRTqKImIiIikUEdJREREJMUqN5l77ty5idjbb7+diJ1xxhlVPU/sDso532yzw9h///0T\nsdhkbDPLXGdsMv6qPqFwVRKb/BnLyQUXXJCIpd3uYsCAAZmeR/ITu4VQ7BY0sYn4AGuttVbN2yTZ\nLVq0KBGLfcYee+yxidi1114brTM2GfyOO+5IxHK+IXlVNKIkIiIikkIdJREREZEU6iiJiIiIpFBH\nSURERCTFKjeZe+DAgYnY+PHjE7EFCxZk2hZgzz33TMQqubqsVCc2Sf7+++/PtO2mm24ajd90002J\nmCZz5yc2yXrWrFmJWGzRROwK3hCfJCr5iV2JObZoJna19bTJ3LH3hCboF2vhwoWJ2PDhwxOx2NX3\nIT7BP3b3hZEjR7ajdcXQiJKIiIhICnWURERERFKooyQiIiKSQh0lERERkRSr3GTumEMPPTRTuSlT\npkTjsSv+rkpXDe0oJkyYkLmsJoQ2ntg+NW/evMzbxyZ5x/bpoUOHVtYwyWTs2LGZyu23336Z6zzs\nsMMSsd133z0R010R6iN2nHzqqacSsUpe/9idMnr16lVZwxqMRpREREREUqijJCIiIpJCHSURERGR\nFOooiYiIiKRQR0lEREQkRVOsesvqe9/7XjR+9NFH59sQadOyZcsSsREjRiRiDz74YB7NkTpJu11J\nVvPnz69RS6QtsdvNxFYzPfroo4nYzJkzo3VefvnliVjv3r3b0Tppj9deey0RGzJkSCIWuyVU7PYz\nANdee20i9vrrr7ejdY1DI0oiIiIiKdRREhEREUmhjpKIiIhICnWURERERFI07WTu2CS1tAmFBx98\ncL2bIxWKTeaOXRq/f//+eTRH6iQ2QXjGjBnRsn369EnE9t9//5q3SeJit7EYOXJkIvbII48kYmk5\njW0v+YlNnI8tbjr88MMTsYULF0brjC2w6datW+WNayAaURIRERFJoY6SiIiISAp1lERERERS1GKO\nUjeA6dOn16Cq2pk3b17mss8880witnTp0lo2J7OS17HIk7qF53Tx4sWZyj3xxBPReNeuXWvZnKoo\np+liF4xcsmRJtOynP/3pRCyW/zxyr5yme+655xKxtJw+/vjj9W5OZh0xp8uXL0/EYheHjJVbsWJF\ntM6m/Dx1zlX1AIYBTo+aP4ZVmxvltOEeymnzPZTT5nsop833qCqnFpLTbmbWG9gHmA0klypJpboB\nGwGTnXMLimiAclpzymnzUU6bj3LafGqS06o7SiIiIiLNSpO5RURERFKooyQiIiKSQh0lERERkRTq\nKImIiIikUEdJREREJIU6SiIiIiIp1FESERERSaGOkoiIiEgKdZREREREUqijJCIiIpJCHSURERGR\nFOooiYiIiKTItaNkZg+a2QV5PqfUl3LafJTT5qOcdgxmNsjMPjCz/yq6Lc2kZh0lM9vezG4zswVm\nttTMnjKz48uKXQAcZ2Z9MtS3mpmdZWZfqLJdnzezSWb2spm9b2ZvmNldZja4mnqrbNNRZvZhK4/D\ni2pbaN+1rbRtpZl9pqR47jmN1Pvr0LbballvO9vyLTN7NrzXno/sA4Uxs03NbLyZvRL20elmdoaZ\nrVZWtJCcmtleZnZfaNtCM7vZzPpXW2+VbRphZhPNbE54j40rsj3lzGwHM/uLmb1rZovMbLKZbRsp\nWsSx94tmdo2ZzQg5nRn21fWrqbdajZJTM1vDzM4Jn0cLQluObKX8FiHXi0P535nZOqVlnHPTgTuA\nczO2YdeQ67Wq/Fv2Crl+ysxWmNmsauqrhVq+/zrXqEF7A7cBj+MTtAQYAGxQVnQSsAgYCZzdRrWr\nA2cBDri3iuZtDqwErgTeBHoCRwD3mtm+zrm/VlF3e90T2lBuFLANMCXf5iRcBfytLGbA1cAs59wb\nJfEicvpxo8x2BI4C3q9FfVW25Xv499nNwMXA54HLzGw159yFBbdtA+AR4G3gV8BCYFfgHGB74MCS\n4rnn1My+AtwKPAqcAqwF/BD4p5lt55xb0N66q/RjoDvwMFDoB3w5M9se+CfwMj4HnfA5m2pmOzvn\nXigpXsR+egH+eHsz8AKwCXACsJ+Z/T/n3Lwq6q5Go+R0HeAMYA7wJLBHWkEz64fP9dvAT4A1gZOB\nrUKuV5QUvwq4w8w2ds691EYbBgNnAtfi3x/tNQw4BN8HeK2Kemqpdu8/51xVD3zC3gBuzlj+MvyH\nbVvl1gE+BM6sto2RulcLbb6z1nVX0aZuwLvAXUW3JaV9u4V8nNJIOQXuB34NvATcVnD+3gImlcV/\njz8ArV1w/k7Ff2HYoix+XYivXRbPNafAM8AMoFNJbBtgBXBhga/bhiX/XgyMKzKPZW27A5gP9CiJ\nrR/eb4njcQE53T0S+3yo+9yOnlOgC9An/HuH8LocmVL2CvwARL+S2NCwzbfLynYGFgBnZ2jDSWH/\n/2yVf8v6LfsucHuW91kOr2/N3n+1aMz3wwu9efj/6oC1Uv6rofy2rZTpH/6YleFny+PMkjfCQGD9\nKto9DXggQ7mpoez2+A/l94BZwPciZTcEBrazPYeEv/GIot9gKe27InxoJXaoonIKHAm8A/Shgo5S\nPXIKfDn8bfuUxT8X/s5hBefv56F9vcri/wt8AKxWVE7x3/o+BP438rungFeKyGlku0brKL0LjI/E\nb8ePsK5eVE7baPd8Mnyx7kg5pe2O0pspuX4O+Gsk/kfgiTae86xIrj/qNAG9Q65Xq/BvqaijVPKe\nG4UfRZ4dcj0V2LKsbG7vv9JHLeYoDcV/g9nQzJ7D93oXmdkVZtY1Uv4x/Gmc3Vqp8y18B8yAP+FP\nUx0R/g3QD5gOjM7aSDNb08x6m9lAMxsNbAn8PcOmDuiF//b2KH648xXgSjM7uqzs70O72mM4/s1x\nSzu3rxsz6wwcDNzvnHs5UiT3nJpZd/yH/Pmu8iH8euR0u/DzsbL4Y/iDwHYUayr+tR9nZtua2QZm\ndig+J5c658pPXeaZ05bjROz06XtA3wxza/LaTxtJV9Jfs08DW5XFCzn2ljKzNfCnveZnKN4Rc5pg\nZn3xXwYfjfz6YeLHlsfwp+W6t1L1H4Gbwr9PxOf5m/j3APjTVNOBndrR7PY4KjznWPz7a0tgipmt\nW1Imz/ffx2rQE34S3zlaAlwCHAD8Ev/hcEPKNsuAsW3U25uU4V98D3QlcE0F7byLj3vNy/AjJJ/O\nsN3d4blOLIl1wZ+LfYNPniq4G1jRjtewZ2jTjdXmox4P4CvhdftuK2VyzSlwIfAi0CX8v5IRpZrn\nFD/v5z8pv5ubti/knMfTgKV88ttj6hB0XjnFfygvpOybcah7cahju7xzGnmOhhh9KGnPv/EfGlYS\n64L/Rr4SOLConLZS9+lh+yEZynaYnNLKiFLJ74ZHfndBeI26lMUPC/Ed23jeH5Fy6g0/4rQS+EKF\nf0t7R5SWUDJShO+gfQhcVMT7r/RRixGl7vg5P9c55/7HOXerc+6H+Im/h5nZgMg2b+PPg7eLc26O\nc66Tc+5bFWx2CrAXcCzwIP4bV5eM264A/q/k+T/A/3198G/ilvh/O+faM0H+4NCWG9qxbR6GAf/B\nT4pLk1tOzWxz4AfASSEX7VHrnK6Gf41iloXfF202fiHBt4GDgHHAaWY2MqV8Ljl1/gh2NTDUzEaH\n1Xk7ABP4eB/N8vrVez9tNFfgF6uMM78sfCv8yErLBOXYa1bEsReAsIruTGCCc+6ejJt1tJzGtORx\neeR3y8rKtHg7/Kwm1+eEXNdk4U0Gtzjn3ix5/keAfwH7lsTyfv8Btbk8QMvQ7/iy+I34b4q7RrYx\n/LBqbpxz05xzU5xz1wF7A7vgZ/pn8bpLnpp4Hv93bFSD5g3Hf6P+Sw3qqqkwVPk14C/OubdbK0p+\nOb0UuM85d2sVddQ6p+/jO98x3Sh4VZ6ZHYb/wPmWc25c+ELzHeC3wAVm1jO2Gfnl9EzgGvzplefx\npxQ+wHfmwH/bbEu999OG4py7Gn8K4nD8ZPh/AxsDvwhFYq9Z7sde8Evb8afvpgHfqWDTDpXTFC1/\nf2wqS7eyMi0s/Mw911V4MRJ7nhrkuYr3H1CbjtLr4efcsnjLvJHYAbgHlZ4jrKHwreQ24KCUeVS5\nMbMNgd2Bic65lUW2JcWB+G8rbY125ZJTM/sisA9+2X3/8NgIP8lvtfD/Nevdjog3gE7l1zUxsy74\nUxmvR7fKzwjgcffJSzuA3w9WJz7PIbf91Dn3gXPuu0Bf/MqUgc65L4c2fEj8INrhOefOANbDH0O2\ncc7tgr9MAPgPmXK5H3vDMe6v+FGO/ZxzS/N8/ibQss9+JvK7zwALIyPrLZ+7hX3ONopavP9q0VFq\nmbzaryzeN/x8qzQYJqZ9mrYn3tW7J7w6vted5UO1ryUvyjcQ38bZVbZjWPjZqKfdhuO/md6eViDn\nnG4Y6rkFPy/pJfxKmL74hQWzgGMy1FPrnD6Jfz/tWBbfCb+fPdmOOmtpPT7+AC3VcmrrE6ctitpP\nnXNvOefud869aGafAoYADznn3suweT3304blnHvXOfeAc+6ZENoLeNU591xpuSJyama98B9SnfEr\nQsu/ULelQ+a0lHPudfznaPmxBWBn4seWjfFfMGKd5U9UX13ramqzSGxzqshzDd5/QG06ShPxHxDl\n5wy/gx86n1oW3wGfnAfaqLflwNij/Bdm1jmsXmvzYmFlM+ZbYj2ArwMvO+ey9Lg741eCtGzfBfge\n/s37WEl8QzMbmKG+UoeHdrT1euQujI4MBf7knFvWStE8czoFP8p1QNljPv6CigfQSqeuRK1z+g/8\n6dMRZfER+AnUd2Soo56eB7Yzs03L4sPwB9RpZfFc99MUJ+Pn21ycsXw999NVQljJuCN+YU25vI+9\nq+MX0XwG2Nc5156rNXf4nAZ/BL4SLjwJgJkNxXckJkbK7wA845xb3Ea9LaMrsVy3rBLPa37lAaEz\n3/L8O+OnyNxZEsv7/QfU4MrczrknzV8C/pjwJr4H+G98R2R06eSsYG98x6DVb9jOuWVm9ixwqJm9\ngP8Qejp8a2pZIngdfnJ2a+4ys1fxk8Lm4WfNH41/8Q7J+Ge+Dvw4nOJ5Hr+iYBvgO2Wny34PfIGM\nHdAw+XIb2rnUMQeH4Uch2hrtyi2nzrlXgVfL42Z2KTDXOZelkwQ1zmn4284AxprZRGBy2G4YcKpz\n7p2M7aqXC4EvAfeZ2Vj8Bem+ij+N+eui91MzG44/ZtyLH8HcC/hGaFvWuWg130/NXzF8W/yXwS7A\ntmZ2Wvj1JOfc0xnbVnNm9nn83K6/4vO5K/7Ydif+4pLl8j723ogfUb0G2NLMtiz53RLn3KQ2tocm\nz6mZHYfvpLR0gL4WThUBXFbS0RmN3x+mhmPdmviLRf4bn4vSOjvjR2LHZmhCyyUjRpvZePzgxm1h\nXtgJ+PfXHrRxhXYz2xo/lxVgU2Dtktf03865P2doy4v449OV+LlXJ+I7xKV3Ncj7/edVskQu7YH/\nMD0Df9pjGf4KuydEyhn+8uZnZ6x3F/ykzvfxS/paLnqWeYkg/hv9Pfg5VMvxF+66BRicsQ13479t\nb4e/6NnS8Hd+P6Vs5iWq+Df/SsouqtUoD/w3z9dp/QKiuec0pd5ZlF0Vu6Ccfgt4Nvx9z8f2gwLz\nuSPw55CvZfgDzinAp4rOKf6Adjd+ZHApfgn4tyv42+qSU/yCj5Upj+jFAXPM5yb4b8xz8aNAz+BH\n4TpHyhaR05daee2yXCG86XPaxmv02bKyg0K+F+M7xr8F1o3U+aWw/SYZ23Aq/jY4H5Q+LxVcHgB/\nDaS0v6PVyy+QfsHJu4GtImVzef+VPixUmAszOwC4Hhjg2nmuMG9mdjfQ2zm3TdFtaUTKafNRTpuP\nctpxmNmt+E7jN4puSxbmb3z9Ev5yL2OKbk9M3ted+DHwq1VlR5VMlNPmo5w2H+W0AwjL4PfFn1qU\nGsm1o+Sprt9xAAAgAElEQVScG5zn80n9KafNRzltPsppx+D8Sse067lJO9Vi1VtH0EhLKKU2lNPm\no5w2H+W0Y3A0cK5znaMkIiIisirRiJKIiIhIiqrnKJlZb/y1WGbz8Q36pP264e9tM9k5t6CIBiin\nNaecNh/ltPkop82nJjmtxWTufWjc22+syobjL5hVBOW0PpTT5qOcNh/ltPlUldNadJRmA1x//fUM\nGjSoBtVVbuLE5BXcb7gh+V579dXEBZ1Trblm8hZwkydPTsQ6depE5861Wzw4ffp0jjjiCCj2Pkaz\nodicxixenLwaf3itEmLvia5di7n/sXKabt68eYnYBRdcEC07derUREz7aePlNOY3v/lNNH7//fcn\nYtdee229mxOlnKY75pjk7TN79eoVLXvxxVnvOFR/tcppLY4cywAGDRrE9ttvX4PqKvfQQw8lYtV+\nKMYOqtttl7zBeufOnWt6AC5R5LBr4TmNeeed5F1A0vIcy1W3bt1q3qYKKadlXnvttUSsR4/EbadS\naT9tvJzG9O3bNxrv3r17ItYAf4tyWiaWp7T9tJHaXaKqnGoyt4iIiEiKvK/MXRcDByZvGn311Vcn\nYmuttVZ0+9GjG/WetFJqxx13TMRGjRoVLdsAo0dSJjZ6tMEGGyRi5513XnT7TTbZJBEbMyZ5xwPl\nvlixkd8zzjgj8/bLliW//CunxZo7N3lB9wceeCBathnzpxElERERkRTqKImIiIikUEdJREREJIU6\nSiIiIiIp1FESERERSdEUq96GDh2aiMVWXqStkIoZO3ZsIraqz9xflVxxxRWJ2MKFCxOxY489No/m\nSA0MGTIkERs8eHAittlmm0W3HzFiRCJWyT4t+XjhhRcSsbSVjFq12Hhiq9Zix96ORCNKIiIiIinU\nURIRERFJoY6SiIiISAp1lERERERSNMVk7kceeSQRO/zwwxOxo48+Orr96aefXusmSZVuuOGGROzm\nm29OxNImfsbeE1deeWUiNm7cuHa0TtoSW0wxc+bMTLG0WyPEVHJrDMnHTjvtlLnsddddV7+GSLvM\nmTMnEXv77bczb79gwYJErF+/flW1qWgaURIRERFJoY6SiIiISAp1lERERERSqKMkIiIikqIpJnNP\nnjw5EYtNEpXG9NprryVi06dPT8RiV2BPM378+ERsk002qaxhUlMHHHBAIrbDDjskYnfddVd0++HD\nhydiPXr0qL5hUlOx/XmfffaJlo0tsJBiLVq0KBHr2bNnIpY2wfvaa69NxFb1BVMaURIRERFJoY6S\niIiISAp1lERERERSqKMkIiIikqIpJnOfdNJJidisWbMSsbSr+G622WaJ2KGHHlp9wyST2OS/QYMG\nJWIzZsxIxJ588slonWPGjEnEXn311Xa0TtojNsn6lltuybRtLHdQ2RWfpfYmTJiQiB122GGZtj3m\nmGOicR1nG09sP1u4cGEiduCBB0a3j+2/sc/otLsqNCKNKImIiIikUEdJREREJIU6SiIiIiIp1FES\nERERSaGOkoiIiEiKplj1tmzZskQsdsn82OoqKV5s1WFsheIWW2yRuc7BgwcnYv369ausYVJ3sdtd\npN0aYeutt653c6QVsRVqsdtH3XvvvYlY2rE3dpzefffdEzHtu/mJfZ6OHDkyEYvdfgjg1ltvzbR9\nbHVct27dGnI1nEaURERERFKooyQiIiKSQh0lERERkRTqKImIiIikaIrJ3GPHjk3E7rrrrkTs4Ycf\njm6vWyMUKzZJNDahc4011kjEevbsGa3z3HPPrb5hUogBAwYU3QTJaNy4cZnK/exnP4vGR4wYkYiN\nGjUqETv99NMra5i0W9bJ1Gm3BIuJTeaPxS655BJ++MMfZq43LxpREhEREUmhjpKIiIhICnWURERE\nRFLUYo5SN4Dp06fXoKr2ef311xOxJUuWJGLPPfdcdPtOnTrVvE3tVfI6FnnVrcJzOm/evERstdVW\ny7z9Cy+8kIilzWeqN+U0XSzPy5cvj5Z94oknErGuXbvWvE1ZKKeViR2jAVasWJGp7OOPP17zNpVT\nTtPNnz8/l+d55ZVXaprrmuXUOVfVAxgGOD1q/hhWbW6U04Z7KKfN91BOm++hnDbfo6qcWkhOu5lZ\nb2AfYDaQvPa5VKobsBEw2Tm3oIgGKKc1p5w2H+W0+SinzacmOa26oyQiIiLSrDSZW0RERCSFOkoi\nIiIiKdRREhEREUmhjpKIiIhICnWURERERFKooyQiIiKSQh0lERERkRTqKImIiIikUEdJREREJIU6\nSiIiIiIp1FESERERSaGOkoiIiEiKwjtKZjbIzD4ws/8qui1SG8pp81FOm49y2nyU0/qoqKNkZmuY\n2TlmdpeZLTCzD83syJSyO5nZFWb2qJn9x8xWxso556YDdwDnZmzDrmZ2lpmtVUnbI/WYmX3fzJ4w\ns8Vm9qaZ3Wlmu1ZTb7XMbC8zu8/MlprZQjO72cz61/H5MuU0vF5Hm9kkM3vZzJaY2VNmdpqZdS0t\nW0ROzWw1MzvOzCab2etmtsjMHg85LuwLgZl9vuQ1e9/M3giv9eA6Pmcl++m3zWxqeP8vM7NZZjau\n/D1X4H76UzN70MzmhdfveTO7xMzWqabeajXqfhrZrrOZPRvKjyr9XYE53cvMrgnHjxVmNqua+mrJ\nzPY0sylm9k44hjxqZgfX6bkq2U+vDb8vfzxbWk45/USbvhjaNCPspzPN7Ndmtn6ldVX6AbIOcAaw\nBfAk4Fopuy9wLPAhMLONeq8CDjSzjTO0YTBwJtAjQ9nWXARcAfwb+J/w/82Be8xsxyrrbhcz+wpw\nF9AZOCW0aQjwTzPrXaenzZrT1YFxofyVwInAv4BzgDsj5fPO6SbAZeHfFwM/Ambhc3xNFfVWa3Ng\nJf41GwlcCKwH3Gtme9fpOSvZT7fDv04XAN8Hfg98GXg4ckApYj/dAXgC+Bn+9bsVOAa438xWq7Lu\ndmnw/bTcD4ANWylfRE6HAYcB7wCvVVlXzZjZMcBk4D/AT4GTgHvwr189VJrTZcBw4IiSx8mRcsqp\ndwF+v/wTcAJwE3AI8LiZ9amoJudc5gfQBegT/r0DvhN0ZErZdYGu4d+/Ala2Um9nYAFwdoY2nIT/\n4PlsJW0vq6MTsBQYXxbfKPxNl7S37moewDPADKBTSWwbYAVwYZ2eM1NOQ7nPReJnhHx8seCc9gYG\nReLXhLo3KSKnKW1dDXgDuLPInLay/fZhmx8XmdNW6j4o1H1IQflr2P20bJs+wNvAaaH8qEiZ3HMK\nrN/y2gG3A7OKyGNZm/qHz4QxOT5nJZ+n1wKLMtarnPp27B6JfT68zudWUldFI0rOuQ+cc/Myln3L\nObc8Y9kVwFRg/9bKmdlZwC/Cf2eHoceVZvbZ8PveZjYwwzfNLvgPq/K/5S38i/heG+3o3zKUbWY/\nNLPZZvZeOIWxZVnZzqFNrQ73mVlPYBBwi3Puo9OUzrlpwHR8b73msuY0lHso8qtbAMO3vbR8rjl1\nzi1wftg51j7K2xdpR81z2kpb38e/16r9FpdWf+b9NMWc8PMT7StgP22tfVbevkg7Otx+WuZ/8W26\noZV6c8+pc+7N0teuEnXcT0fgz7CcFbZdoz3tq0R7cmpmnzKzNduoVzn1bbovEvsnsJA2Pg/KFT6Z\nu8RjwFZm1r2VMn/ED5+BP/VzBPBN/IcO+OG16cBOrT2Rc24Z/rTR0WY2zMw2NLNtgOvwPfFfZ2zz\nUeE5xwKjgS2BKWa2bkmZfqFNo9uoq2Wez/uR370H9K14uDAfnwk/50d+l1tO29m+mFrm9CNmtmbJ\ngael3r9n3b7ezKyXma0bTjtfiz8NMCVStJCchtduPTP7PP4Ua8uHQRYdbj81s52BI4Ef0vYpnUbY\nTytV6/10KPAcsJ+ZvQIsNj9v6Fwzs9o2vd1WBxYB74a2jW2lQ6ecRoTXqzvZPw8AP0TXKGbhO25b\nAI/GCjjnnjazx/Hf2iY5514uL0L2c/fDgYnA9SWxmfjhutkZ6xgAbOqcexPAzCbjO2Cn4Ic0K2nX\nXPz53d1Kg+bnPLSsYOhHchSsaD8G3sXP2SiXd04/wcy64D8oZgGPZNysljktNRHYJ/z7P8DV+Hk3\njeI1Pu4EzAd+4JyLdZRyz6mZrYc/VdniFeBw59zzGavoiPvpr4CbnHMPW9uTzAvdT9up1vvpZvhT\nUOPwc1um4U/xno6fqnFazVrePq/jR38ex+fqS/g5e9uY2R7OuQ/Lyiuncf+DP6M0vpKNGmlE6e3w\ns92rWZxz5zjnOjnn7s1QfAl+rsFY4ED80GtnYJKZ9cr4lLe0JDU8/yP4xO5bEpsT2vStNtru8B+e\nQ81stJltamY7ABPwiQV/urBhmNmpwBeBU5xziyJF8s5pucvxB4rjIweSNDXLaZlTgL3wCxweBD7N\nx3ltBF/CT+IeBbwMpH1TLSKnC4E9ga/g58TNB1o9/VCmQ+2n5iclb4l/z2VR9H7aHrXeT7vjT+We\nGf6WW5xz3wT+ApyYx6m41jjnTnPOneqc+4NzbqJz7lh852034BuRTZTTMmb2BfzE9QnOuXsq2baR\nOkotw5t178GaWSf8aY93nHM/cM5Ncs5djf8gG0B8JUHMi5HY8/hJ4e1xJn7y8cmhnoeBD/DfcsB3\n7hqCmR0KnAf8xjn3f2nFws88v5X4JzY7Gfg2cLpzbnIFm9Y6p4Cfw+Kcm+Kcuw7YG9gFf4qrITjn\n7nHOTXbO/RK/MuRsMxsZKZp7TsNcjn845+50zp0PHA+MM7N929o26DD7aZi/Mhr4hXPu9aybhZ+5\n76dVqHVOW06llo803ITv+G7Xznrr6RJ8zvaM/E45LWFmW+BXv00DvlPp9o3UUeoZflZ07rCdvgBs\nBdxWGnTOvYg//7lbbKN6Cx8I3wX64mfnD3TOfRn/TedD4m+k3JnZXsBv8asbRrRSNM+cfsTMjsZP\nZL3COffzPJ87C+fcB/j33kFWdg2qRuCcm4Vfkj888utCclrKOfcg/lRcrH15PH8j76cn40e2JoZJ\nsv35eHl7zxArH8ksPKcNoKVTObcsPg/f6ehJgwlzbRcAsTMgymlgZhsCf8WPsu3nnFtaaR2NNEdp\nY/xBpq15B7XoIa8X6ukU+V0Xsr8um0VimwOz29cszzn3FmFCnfmLJQ4BHnLOtboaLw9mtgu+Z/4w\ncGgbp7TyzCkAZrY/fjL+H5xzx7ejirrkNGJ1/AF4TSDT6tCcrYY/PVgu95ym6AasnbFsR9pPN8R/\nSD5bFnf4UzWn4kdHppX8rlFyWola5/QxYFP8/LLSOvrh/+63ItsUKkzUXod425RT/CIVfCepM7CH\nc668I5xJI40o7QA845xb3Ea5lt5gYmlwBcsZn8d/SH1iKa+ZbQ8MxE+Yy+IAM+tbsv3O+FMqd5bE\nqlpKjv+GuD7+IoqFMrNBwJ/xEwW/6tq+/EOeOW05B30TfjXUEW2VT1HTnJat2GiJ9QC+DrzsnCvs\nG5+ZdQptKY/vDGxNfAJ8bjk1s9VjZczs6/jOQNYJ+h1pP70UP+fygJLHd/HHu2vD/18q2ybX/bRG\nap3TCfjX6KO5L2G12zH4OXKP1arhlTKzrimr184MP2MLaTp8Ts1sdfxr8xlg3zBS3i4VjyiZ2XH4\nF7VfCH0tDG0BXNaSGPPXYvhmiO8YYi0rB+Y45z5abWZmnfHfxsZmaMJj+Df0aDMbj58bcJvz16Y5\nAf/m2QNInYDmnHvczP4GHGVma+N7nH3xcx+W4g82WbwI3GdmV+K/4Z6I791fWFKmZTnjdfiJvKnM\nbDj+A/Re/DyHvfAT9X7tnLs1Y5sqliWnYUedHMr9AvhK2arZma7kOkt55zS8327Df4v6E3BIWfum\nOeeeytCWmuYUuMvMXsVPSpyHv7Dd0fid95AM7WmXjPtpd+AVM5uAX9iwFH/hxKPxw9Q/K6sz15zi\nv2H+PbTvOXxud8KfcpvFx1dib0uH2U+dc0/ir/Jcul3LqrdnnHO3l/0u75xiZlsDXwv/3RRYu+Sz\n4d/OuT9naEtNc+qcm2RmU4Cfhi83/8Z3OAcD3w2ny2su4366PvCEmd2E3w/g48UXdzrnbiurUzn1\nbsQfL64BtrRPXpNpiXNuUoY2ea7yq12+hF9GGXt8tqTcEPyBLVbuH2V1fokKrp6MHz5+GZ/Uj54X\nf7GwlcAXMtTRFT8U/RT+YLcQf3uEbTJs2z/8baPwy89n46+hcjewVaTsSuCaDPXuFOqYj//Qehz4\ndqU5qkdOS/6OtMe4InMa3m+tte/MgnI6An8bhLn4U2xv4i+CObgBctoFGIOfj/Q2/hYJs/CruhJX\n6i0gp73xt355Bn/9mPfxHxQXAb20n8aPvSmvw0riV+Yu4th7VCt/x7g2tq1LTkP51cP+8Fp4rz0J\nHFZ0TvGnmH+Lvxr84vD3TsNfmqVTpE7ltO3XtqIrh1uosFBmdiuwwjkXW+bYcMI3tJeAk5xzY4pu\nTyNSTpuPctp8lNPmo5zWXuGTucOyvX2BbYtui9SGctp8lNPmo5w2H+W0PgrvKDnnniO+ukZWUcpp\n81FOm49y2nyU0/popFVvq5q8L+8u9aecNh/ltPkop82noXPaEHOURERERBqRRpREREREUlQ9R8n8\nXbP3wS/pW1ZtfUI3/L1tJjvnFhTRAOW05pTT5qOcNh/ltPnUJKe1mMy9D3BDDeqRTxqOv2BWEZTT\n+lBOm49y2nyU0+ZTVU5r0VGaDXD99dczaNCgGlTXjgbMnp2InXPOOYnYtGnTEjGADTbYIBG7/vrr\nE7E111yz8sZVaPr06RxxxBFQ+3uLVWI2FJvTMWOSl9O44Ybk8SOWO4DTTjstEdt5552rb1g7KKfe\n8uXJO978/ve/T8SuvPLK6PaxuHJabE4XL07eIWOPPfZIxH73u99Ft99yyy2j8SIop17s2PvUU8mb\nGlx11VXR7bt2bZz7fNcqp7XoKC0DGDRoENtvv30NqqvcGmuskYh17x67NU5cLLHbbpu8DEWPHonb\n4dRTkcOuhed0vfXWy1QubafcbLPk/RWL+ltKdOicLluW/POnTJmSeXvlNP7cReb0nXfeyVRuiy22\niMYbIH8xHTqnsWPvSy+V3x4Qtttuu+j23bp1q3mbaqCqnGoyt4iIiEiKwi84WanYN5hdd901EZs8\neXIittZaa0Xr3G+//RKxuXPnJmI5jyh1aLHh38svvzwRGzhwYHT7PffcMxF79dVXE7F+/folYlK9\nGTNmJGKx/Wz//fdPxI455phonXfeeWciNnTo0Ha0TmoldpyMGT9+fDS+00471bI5UgOx09lZT8dB\nc+ZUI0oiIiIiKdRREhEREUmhjpKIiIhICnWURERERFKooyQiIiKSYpVb9Xbjjdkurrlo0aJEbOut\nt46WXbhwYSLWv3//yhomNXXeeeclYrHVFGkrLAYPHpyIPffcc4mYVr3Vx09+8pNELLbC7eKLL07E\ndtttt2id5557bvUNk5qKHWdjirowqFRunXXWyVQua+6bgUaURERERFKooyQiIiKSQh0lERERkRTq\nKImIiIikWOUmc8duWXDcccclYrFbWMQm+EL8lgkNemO/DuP000+vavvYrRVit9XQLTDq4/jjj0/E\n5s+fn4jFcvLAAw9E60y7BZEU55FHHslU7oUXXqhzSyRvZ555ZjTejMdUjSiJiIiIpFBHSURERCSF\nOkoiIiIiKdRREhEREUmxyk3mHjhwYCL2/vvvJ2IXXXRRInbGGWdE6/zc5z5XfcOkELHJwAAzZ85M\nxIYNG1bv5khQzYTO8ePHR+OjR49OxG666aZETAsx8pN10cV1110Xjccm/ffo0aOaJkmVYvvu3//+\n90QstmAK4J133knEVvWcakRJREREJIU6SiIiIiIp1FESERERSaGOkoiIiEiKVW4yd8ycOXMSsdjk\nwVdffTW6/ZAhQxKxfffdNxFrxiuOruqOPfbYaDx2tfVVfUJhR3HooYdG4wsWLEjETjvttETs4osv\nrnmbJO7KK69MxGJ5GjNmTHT78847LxFT/hrPFltskbnsjTfemIiNHDmyls3JnUaURERERFKooyQi\nIiKSQh0lERERkRTqKImIiIikUEdJREREJEVTrHq7+eabE7H9998/8/brrbdeIha7NYZWveUndhn8\nsWPHJmIPPPBAdPvYbWmmTJmSiO22226JWOfOnencuSl2jYZyxRVXJGI33HBDIjZx4sTo9rGVM5tu\numkiNmrUqERs7bXXpnv37lmaKRWI3W7mnnvuScR69eoV3T62Gm7SpEmJ2NFHH52I7bXXXuyyyy4Z\nWimVmDBhQiI2efLkzNvHcrpw4cJE7OCDD07E1l133dT3SpE0oiQiIiKSQh0lERERkRTqKImIiIik\nUEdJREREJEVTzFiN3a5i6623TsTSLqMf2z7t1hiSj1hObr311kRswIAB0e0feuihRCyW/8svvzwR\n22233dh2222zNFMqkPU2BhtssEHmOgcPHtze5kgNXHvttYlY7LYksf0RYObMmYlYbCFO7Hiw9tpr\nZ2miVGidddZJxGLH3rR9b+7cuYnYGWeckYjFjsfnn38+I0aMyNLMXGlESURERCSFOkoiIiIiKdRR\nEhEREUlRizlK3QCmT59eg6raZ968eYnYihUrMm8/f/78ROyJJ55IxLp27VpZw9qh5HXsVvcnS1d4\nTmMXnIxZvnx5NL5kyZJM27/88suJWK9evVi5cmWm7bNQTtPFXv9KxPL81FNPJWLdu3dn9dVXr+q5\nSimn3uLFixOx2ByVrPtj2vbKaX5eeOGFRCz2eZqW07RjcpY6X375ZR5//PFM22dRs5w656p6AMMA\np0fNH8OqzY1y2nAP5bT5Hspp8z2U0+Z7VJVTC8lpNzPrDewDzAaWVVWZgO/5bgRMds4tKKIBymnN\nKafNRzltPspp86lJTqvuKImIiIg0K03mFhEREUmhjpKIiIhICnWURERERFKooyQiIiKSQh0lERER\nkRTqKImIiIikUEdJREREJIU6SiIiIiIp1FESERERSaGOkoiIiEgKdZREREREUqijJCIiIpKi8I6S\nmQ0ysw/M7L+KbovUhnLafJTT5qOcNh/ltD4q6iiZ2Rpmdo6Z3WVmC8zsQzM7spXyZmYjzOwJM3vP\nzOab2RQz27qljHNuOnAHcG7GNuxqZmeZ2VqVtD2lbd8PbVtsZm+a2Z1mtms19VbZpvXN7H/N7B9m\ntii8vl+o83Nmzmn4Xdpjcku5AnM6NaVtd1ZTb7XMbC8zu8/MlprZQjO72cz61/H5Kt1PDzGzB83s\n7bCPTjWzfUvLKKeJdjV6To83s2fNbJmZvWpmF5vZ6qVlCsxp51DPzNC+mWZ2mpl1qqbeKttUxLF3\nRzMba2ZPm9kSM5tjZhPMbLOU8luY2V/C59UCM/udma1TWqbAnP40HEPmmdn7Zva8mV1S3r681Ww/\ndc5lfgD9gQ+Bl4ApwErgyFbKXwcsB34NHAucAIwDhpaV+1Koa+MMbfhRKPvZStoeqefi8LdcB3wb\nOAl4EfgPsGM1dVfRpiHhb3sOuC/8+wt1fs7MOQWGRR6XhG1GNUBO7wbmAIeXtXGPIvIZ2vQVYAXw\nEHA8cCowD3gZ6N0AOT0hlJ0EfBf4AfB4iB2gnK6SOb0glB0fcvrLcFy7K1K2iJxOCK/f1aF940J7\nryowp0Uce28GXgv5OTa8j94AFgP/VVa2H/AW8DxwHPATYEHYVzs3QE7/AFwRjh/HAL8A3gFmAKsV\nlNOa7aeVPnEXoE/49w7hzZ22sx4Sfv+1DPV2Dkk/O0PZk6pNLNAJWAqML4tvFNp8SUGJXQPoEf79\n9Zx21sw5Tdn+N+HN2LfInIZ67gamFZG7Vtr0TDhYdCqJbRNeswuLzmlo20NlsTWBRcAtyumqlVNg\nfXyn6Nqy+HEhH/sVmVNgx9D2s8riF4bXb6uCclrEsfdzJDs5mwLvA78ri18BLAH6lcSGhtfy20Xm\ntJW6Dwp1H1JQTmu2n1Z06s0594Fzbl7G4v8D/Ms5d5t5q6cVdM6tAKYC+7dWoZmdhe+pAswOw6Mr\nzeyz4fe9zWygma3WRtu6AKvhe5el3sK/8d5rox39w3OPMrMfmtls86cWp5rZlmVlO4c2rd9Gm3DO\nLXXOvdNWuVqqMKefYGafxu8MU51zr5fVm3dOS+vsZGZrVPCn1CWnZtYTGITvcKxsiTvnpgHTgcMq\naWNWFeZ0Lcr2A+fcYvxB+f2yuHLa+DndFf9FcEJZfDxglLWvgJx+HnAp7fsUcGgb7WimY+9D4fUv\njb2I/4AfVFb8IODPzrnXSspOwY8wHVJWR2H7aZk5+Pdcjzba0fD7aV0mc5vZmsDOwCNmdj7wLrDE\n/Lnog1M2ewzYysy6t1L1H4Gbwr9PBI4Avonv4IA/jTAd2Km19jnnlgH/Ao42s2FmtqGZbYM/DbcA\nf6owi6PCc44FRgNbAlPMbN2SMv1Cm0ZnrHNVsh9+J7gh5fe55bTE5vjRwsVm9oaZnWtmnTNuC7XN\nadfw8/3I794D+ppZnwraVg9TgS+Zn9PSPxyELsd3oH4ZKa+ceo2a07T2tXz52yGyTZ45bU/7Ypr5\n2LseML/lP2bWF+gDPBop+zCwXSRexH7a0rlaz8w+D1yGH72ZmnHzht1PKznYVGIAvid5OPABfnhv\nET4Z483sXefcX8u2mYXvuG1B/A2Bc+5pM3sc3xuc5Jx7ubxIeGQxHJgIXF8Smwns7pybnbGOAcCm\nzrk3AcxPaP4XcAr+b25Pu1Ylw4Fl+B0uJu+cvgj8A3gKP5T+DeB0YDP8ezGLWuZ0Lv48/W6lQTPr\nDbSsSulHcmQzTycA6+APapeF2Fv4eYQPR8orp42d0xn4Y+9uwD0l8ZaJyf0i2+SZ09L2zcnYvpim\nPPaa2RH41+D0kvBnws83Ipu8AfQysy7OuQ9K4nnvp5jZemVtfAU43Dn3fMYqGnc/reL8X2vnyXcP\nvwkwAPsAACAASURBVFtJycRo/IFuHnBvZJt9QvkvtfG8tZp81gf4Lf7DYX/ge/iJks8CvdrYtmVi\n5e8jv3sQeLaatoV6cjlPnjWnkbJr4nvmN7dSJtecptR9dah75yJyCvw8PP9o/PyDHYC/4zuYK4HB\nReY07JNj8RNqD8J/q3sSeB3YRDldJXP6IH4U/+jwGnw5HNuWA/8pMqf4b/ovhffXgcBn8aeO3grt\ne76InJbVk/uxNzzvFvgP938CVhJv+Tz9RmSbc0Jb1yoqpyV1dQG+COwLnIYf1To6w3YNv5/W6zpK\nLcNdLznnPurNOueWArcDO5tZ+XNbS7E6tenjJ/LLUP8OvOOc+4FzbpJz7mpgL3yv9uSMVb0YiT2P\nnxTe7L6BP+ilnXaDHHPaiotDO/bMWL7WOT0TuAb/nnoeP1T+Ab5jAn4uUJH+AGzonDvWOfcn59xv\ngf8GPg2cHymvnDZ+Tg8C/o1v40v4FY0TgCeIty23nDrnluM/SBfg33uz8VMezgHeTmlfTFMde8No\nzB341+BgFz7pg5bP066JDaFbWZmPqgw/c9tPnZ9H9w/n3J3OufPxK83GWdmlRlrRsPtpvTpKLRN7\n50Z+Nw/f8yyfmNkz/JxP/X0B2Aq4rTTo/ES66ZQN10nUcPy31jtaKZNnTtO8En72KuLJw8Hju0Bf\n/ETWgc65L+Pndn1I/OCQCzPbGP/Ns3w/eBu/RDq2HyinDZzT0L43nHNfwM/t+jywgXPuJ8CG+A+M\ncrnm1Dk33Tm3Nf4YvDv+dfwN/hRw1tM0TcP8NYz+gp8X+CUXTj2VaDmd9RmSPgMsdJ887QYNsJ86\n5x7Et314Qc9fs/20LnOUnHNvmNmbxM839wOWOb+yptTG+Ma3taPUooe8XqgndoGzLmR/XWIXBtsc\n/y2paYUVB3sA4yI7aKk8c5pmQPj5VqulPlaXnDrn3mppQxhNHYJflt/qCss6Wy/8rGQ/UE6DBs3p\nR5xzM/HzLjF/pebP8PG36VKF5NT5iyMCEEYdPgX8LePmTXHsNbOuwJ/xp4aGOudmlJdxzr1uZm/h\nL61Qbmf8qfJyjbCfgh/xWjtj2YbdT+t5C5MJwIZmNrQlYP4qnV/DXzCt3A7AM5EOVLml4WdiyWEF\nyxmfJ7JU1sy2BwbiL+KVxQFhRULL9jsDuwB3lsQyL1FdhRyOf/1aO+0GOebUzNY0f7mCcqfjDwaT\nI7+LySOnJ+Ovd3NxO7evlRfxB9NPLMk2sw3w38Bi+4FyGtcoOU0wM8MvA1+Kn99VLs9jb6x9qwHn\n4c9EjM+42Sp/7A0f2hPx7f6Giy+eaPFH4Ctm9tHgQ/hs3TzUUS7P/XT1WBkz+zp+ZOuRNtrQomH3\n04pHlMzsOPyL2pKwr5nZhuHfl5Uk5uf4SXp/NLNL8Kvevhee89SyOjvje3ljMzThMfyH9GgzG48/\n53ibc+59/AqeM/GjHfemVeCce9zM/gYcZWZrA3/FD88dj3/jXJqhHeA/aO4zsyvxPecT8T3XC0vK\ntCxnvA5/9dVWmVnLh8CW4e88Miy1JJz3rbkKctpiOPC6c+4eUuSdU2B74CYzuwmfl9XwczV2Ba52\nzsW+dcXUNKdmNhw/OfRe/DnxvfDzu37tnLs1Y5sqliWnzrn5ZjYO+JaZTQH+hB/+H4H/239eVqdy\nSmPnNJT7Jf7vfBI/MjgcPxpxpHPu1bI6884pZjYB3yl6Fv9+OxY/ArJvmMeaRTMce8cAX8Wf+l4n\nvK8+4pwr/SI6Gv8em2pml+IX05yEn4t2XdnfkXdONwP+HvL6HP7L1074990sPl5N25bG3U/bMZP8\nJfyM8djjs2VlN8JP2GuZpPdXYPtInS2XXE+ssklpw6n4y5B/UPq8wFlkXK2Anxh3Gn7Z8RJgIXAr\nsE2GbVtm6Y8CfogfGnwPfxXhrSJlVwLXZPzbWlYLlj9WVJqrOuV08xD/RRt15prT8F4bjz/VsBR/\nG4CHKbtqbd45xR8w7sbPFViKH6XJ1KY8coofVR6JP2C+Gx5/i73eyukqk9OjQpsW4VdR/TXttc47\np6HsSfiLKi4Nr+GfgK2LzGkon+uxN7Q5LZ8rI+UHAXeF/WABftX2ukXnFOgNXBlyugg/sfw54CLa\nWEFez5zWcj+1UGGhzOxW/JvxG0W3JQvzN9V7CTjJOTem6PY0IuW0+SinzUc5bT7Kae3V64KTmZnZ\nFvjlotsW3RapDeW0+SinzUc5bT7KaX0U3lFyzj2Hv2aLNAnltPkop81HOW0+yml91HPVW7NzrEKX\nxpdMlNPmo5w2H+W0+TR0ThtijpKIiIhII9KIkoiIiEiKqucomb8b7z74JX3Lqq1P6IZfFj3ZObeg\niAYopzWnnDYf5bT5KKfNpyY5rcVk7n1o+wrNUrnhwI0FPbdyWh/KafNRTpuPctp8qsppLTpKswGu\nv/56Bg0aVIPqKrd4cfIq7T/4wQ8yb3/VVVclYl27xm7UXH/Tp0/niCOOgGLvWTQbis3pj370o0Rs\n6tSpVdX585//PBHbe++9q6ozi46Y03nz5iViF1xwQaZtX3wxfq/Kd999NxGbODF594Y+ffpkep5q\ndMScZrV8+fJE7J574hfxv/zyyzPVOWnSpKralIVymu6+++5LxE488cRo2UsvTd7YYvfdd695m7Ko\nVU5r0VFaBjBo0CC23377GlRXuXfeeScR6969e+btt9tuu0SsW7duVbWpBoocdi08pz16JG49VLWN\nN944Ecv57+swOX3ttdcSsaw5TfuS0rlz8nC19dZbJ2L9+sXuxV03HSanWS1blnxJXn755WjZrF9I\ntZ8W680338xcdtNNN03EGuBvqSqnmswtIiIikqLwC07Wwo03Jk89PvDAA4nYgAEDotvPmTMnERs4\ncGD1DZNMYt9An3rqqUSsZ8+eidjNN98crXODDTZIxPr379+O1kl7LFmyJBGLnX558MEHq3qenEeP\npExs5DA2ynfllVdGt3/66acTsa222qr6hkkmsWPvUUcdlYhNmzYtEbvooouidf72t79NxPbdd992\ntK5xaERJREREJIU6SiIiIiIp1FESERERSaGOkoiIiEgKdZREREREUjTFqreddtopU7n9998/Gp8y\nZUqm7bUSrj7GjRuXiM2cOTMRe/vttxOxelxvSaq3aNGiRGzIkCGJWCX7VOx6aZKf2OsfW106atSo\nRGyTTTaJ1hlb4RbbXurjvPPOS8RiK9ymT5+eiJ122mnROnv16lV9wxqMRpREREREUqijJCIiIpJC\nHSURERGRFOooiYiIiKRoisncsUvmx4wZMyYaj00eHDZsWFVtkuwWLlyYqdwxxxyTuc4ddtghETvp\npJMSsQa4+XGHEbuFyYEHHpipXJrJkycnYlkXd0hl5s6dm4jFbisUO87OmjUrWmds0Ybyl5/YxOuD\nDjooEXvllVcSsdGjR0frTLsB8qpMI0oiIiIiKdRREhEREUmhjpKIiIhICnWURERERFKscpO5Y1eH\njV31dfDgwYlYnz59onVefPHF1TdM2i02ybraq7uefvrpiVjsyt7KfX5iV+Y+/vjjE7Fbbrkluv0V\nV1yRiD3yyCOJmCYD10fsKuqxyfR9+/ZNxHr37h2tM1ZW+cvPj370o0zlHnzwwUQsLafNSCNKIiIi\nIinUURIRERFJoY6SiIiISAp1lERERERSrHKTuV944YVELDZJdOzYsYlY2hW8ly1blojpis35ib3W\nI0eOrKrOG264IRFLuzqw1F5sQm7aJO2sYldx3nnnnauqU6qTdeL1jBkzovHYsVuKFbsK96677pqI\n3X777dHt995770TsgQceSMRiV3VvVBpREhEREUmhjpKIiIhICnWURERERFKooyQiIiKSQh0lERER\nkRSr3Kq32CqLp556KtO2gwYNisbnzJmTiMUu1y/5id2uYtiwYYnYY489Ft0+tsri4Ycfrr5hkkls\nJelFF12UiMVuNTNhwoRonZMmTUrEzj///Ha0TvL25JNPRuOx209Jsa666qpELHY8HT16dHT7PfbY\nIxG77bbbErGjjjqq8sYVRCNKIiIiIinUURIRERFJoY6SiIiISAp1lERERERSrHKTuWPuueeeRCx2\nu5K0ydz9+/eveZsku9jE30cffTQRO+644zLXefnllydiWW+3INWL5XTMmDGJ2BlnnJGIDRgwIFpn\nbD/XrYZWbbqtUOOJLZC48847E7FTTz01uv3UqVMTsSOPPLLqdhVJI0oiIiIiKdRREhEREUmhjpKI\niIhIilrMUeoGMH369BpU1T7z5s1LxFasWJGILVmyJLr9E088kYh17dq1+oa1Q8nrWOTki1xz+v/b\nu/9oKcr7juPvr17KRUTkh8HiEVFohCImomAkieSE5GC1OVpNhAiJ0Hps/BG1inqCGFOT2tTk+KMB\nqW0CNP7C2DQQq+0t8UfSRC0EjEkQUeFe8EfF8EsICFF8+sczV5adZ+7O7s7u7F0+r3P2XHjuzLPP\n3c8+s8/OPDOzZ8+eWNmmTZuqqnPDhg2xspUrV1ZVZ6UOxEx37NgRKwv1yZDQ+wHCF5bduHFjeQ3L\nyIGYaTXa29uD5du2bYuVqZ82XqYvv/xyrCzUx5OsWbMmVlaPz9jMMnXOVfUALgCcHpk/Lqg2G2Xa\ncA9l2nwPZdp8D2XafI+qMrUonIqZ2QBgEtABxE91kXK1AkOBNufc5jwaoEwzp0ybjzJtPsq0+WSS\nadUDJREREZFmpcncIiIiIgk0UBIRERFJoIGSiIiISAINlEREREQSaKAkIiIikkADJREREZEEGiiJ\niIiIJNBASURERCSBBkoiIiIiCTRQEhEREUmggZKIiIhIAg2URERERBLkPlAys5Fm9o6Z/WnebZFs\nKNPmo0ybjzJtPsq0NsoaKJnZKWY2x8x+a2a/N7P1Zvagmf1JYNmxZnaXmf3SzP5gZntDdTrnVgOP\nADenbMNpZnaTmR1WTtsD9ZiZfcnMnjWzHWb2hpk9amanVVNvlW260MzeCzz2mtkHavScqTKNXq/p\nZrbEzDZEy/7GzG4ws56Fy+aY6VfM7Gkze9PM3jazF83sdjMbWE29VbbpiYRM3zOzPTV6znL66UVm\n9mT0/t9tZuvMbL6ZHVO4XI6ZtkT1rI3atzZ6zx1cTb1VtulIM/ummT1uZtujLE+v8XOmzrRovRYz\nez5q49WFv1Om+7Wp0fvpgoS2PV+4XF6ZFtXZN9oGv2dm52ZVbwXtyKyftpS5/PXAeOAh4NfAkcCX\ngZVmdqpzrjC0M4G/jJZbC3ywi3r/CXjEzI51zrWXaMN44KvAAmB7me0v9G3gb4DvA3OBw4EvAT81\ns/HOuV9WUXc1HHAj0FFUvq1Gz5c200OA+cDTwDzgTeA04G+BTwITi+rNI9OTgWeBB4AdwEjgYuBM\nM/uwc+7tKuqu1DeAfykq6w3cDbTV6DnL6acnAeuAJcBW4Fj8a3aWmX3IOfdGwbJ5ZHofcB7wPWAF\n8BHg68DR+P6ah+OBa4GX8K9vPb5clZNpoSvwr5VL+L0y9Rq9nwLsBv4KsIKytwL15pFpoa8DrSS/\n5+olu37qnEv9wL+hW4rKhgNvA98vKj8C6Bn9+zvA3i7qbQE2A19L0YaZwF5gSDltL6rjYGAnsKio\nfCjwHnB7pXVX8wAujP62MXV8zlSZAj2AjwTWvzFq8yfzzLSLus+N6j4/j0wT2jQ1ep9NzjPTLtYf\nE7XvujwzBU6J2nFTUfm3gHeBE3LKrzdwePTv86K/8/QaP2fZmQIfwA9+b4hex6sDyyjT5LY2TD8l\nGsikrDe3bS9wAvCH6D23Fzg3x/wy66dlHXpzzj3jnHu3qOxlYBX+23th+e+cc6l2WUZ1Pgmc3dVy\nZnYTcGv03w7bd1hqSPT7AWZ2vJn1KvGUPYBe+L0ihX6H7xi7SrTjmM5d2WZ2lZl1mNmu6BDGqKJl\nW6I2HVmiTcXPcaiZ1XwOWdpMnXPvOOeeCVTxI/w3nOL8651pkvVR+w4v0Y6aZ1pgKvB74McVrt+l\ncvppgvXRz/1esxwy/Tj+W+mDReWL8NMGJpdoR00ydc7tdM7Vag9v0nNWkuk3gdX4PThJ9SrTZA3X\nT83sIDPrU6LePLe9dwI/BH7O/nu+umpHw/fTrD6IBwGbqqxjBXCCmR3axTI/xB9WAbgSmAZ8AT/A\nAb/bcjUwtqsncs7tBv4XmG5mF5jZ0WZ2IrAQPxIv3gWb5MLoOecAtwCjgMfM7IiCZY6K2nRLyjoN\n/ybfDuwyPydoeMp1s5Q20z+OfoaWrVumhaIOPsjMPg78I/6b6pMpV69FpoVtGwh8CviRq/+hwMRM\nzay/mR1hZqfgv7064LHAovXMtHPuW/Hr1PlF5uQS63eqaaY5C2ZqZuOALwJXUfoQiDIt0qD99BD8\n58JbZrbZ/Pym3gl11H3ba2afw+8luy7N8gEN20/LnaMUY2bT8I2fXWVV6/ADtxFAcH6Qc+63ZrYS\nmAIscc5tKF6E9MdFpwI/AO4tKFsLfMw515GyjmHAcBfN4zCzNvwA7Hr8Ls1y27UL/yH1BL5DnAxc\nA/zCzMY4515L2a6qlJnpdfjj5P8Z+F29M8XMBgH/V1D0CvB559yLKavIOtNiU/CHfhO/5ddCikxf\nY9+H2CbgCudcaKBUz0zX4L84fJR9e7kAOidkHpWiDqh9prkokel3gAecc8usaGJ+gDKNa7R++jp+\n789KfFZnAJcCJ5rZJ5xz7xUtX9dtr5m14g+f3uace8XMjkuzXpHG7adVHgMcgZ9k/D+AdbFcl3OU\nomUm4Y8hnlFiuWvI4Jgq/vj9v+L3OJwN/DXQDjwP9C+x7jH4Q3T3BH73NPB8NW0rqu+j0d97V1Z1\nZpFptOysqG0XN0KmUV098JPLz8QfJ18BTE+xXl0yBZ4C3gAOqkeeaTMFJkR5XYXfsF6XsFzdMsUP\n3NrxHxJ/AQwBzsd/490DvJh3ptRpjlI5mQIz8IeMBhe9DrE5Sso08Xkasp8WLf8VEuZf1nvbiz+p\n51XgkOj/E6KcSs5R6g79tOJDb9E390fwkwU/56LWVKHzeGbNR4rmT0P9CbDNOXeFc26Jc+5u4NP4\nUe21Kat6OVD2In5SeCacc7/Aj6o/lVWdScrJ1Mwm489u+K5z7p+TFot+1m307/xcqsedc4865/4O\nuByYb2ZnpqyiZpma2bH4XdOLXPwbYE2kzdQ591PnXJtz7g78B9fXzOzSUJWdq9Skwfu3aQ9+wLsZ\n+Df8maAL8RvlrfjBQBo176f11FWm0fyVW4BbnXOvp60y+qlMaex+WuR2fGahz4Z6fp4Oxe/xmeWc\n63J+bwkN208rGiiZv+bCfwGH4Uesb5RYJY1+0c9q5zqlcTp+dv5+k/Scn0i3Gr8Xp5G8AvSv5ROU\nk6mZfRq/N+5h4JIuqq1npkHOuafxh+Km5tWGAlPxG6776/FklfZT59w6/GUWQq9ZXTN1zq12zo3G\n99ePAYOB7wID8RvRA0qKTK/F71X9QTRJ9hj8afcA/aKyHkXrKNP9dZd+uhs/4Ax9NtQz05vxe5N+\nVvCe65y7ekRUlmpid6Mqe46S+YsL/gf+NMaJzrk1GbXlWPzut1IdJYsR8qContAFznqQ/nUJXezt\ng8SvgVSt49g3wS5z5WRqZqcC/w4sw58229U3rnpm2pVWoG/KZWuZ6eeBtc65ZRnU1aUM+mkv4I8C\n5blk6vyF9ACI9g4eBCxNuXq9+mlNpcz0aPyHZPE1eBz+UPQs/LWzfl3wO2W6v27RT6OJ2gMJfzbU\nM9Oj8e1fF6h7XvSzH6Wv09Sw/bTcK3MfhJ8AfSrw2YzfSCcDq5xzO0ostzP6GTvdu4zTGV/E75qc\nUrT+GPxFqlamazLnmNnggvXH4V+bRwvKUp/OaIErSEcbkJMJT5auWjmZmtlIfKdeB3zGlb78Q90y\nNbNDQsuY2Xn4Trq8RBs6ZZppwTofxp/yW/PJoWkzNbODzSz0mo8DRhN+zerZT0Nt7oU/5Ps6/pTy\nNGqSaT2V0U/vxM/9OafgcTF+e7cg+n970TrKdN86jdhPeyacvfbV6Gfos6Gemd5A/D3XORn9H6Lf\n7Qyvup+G7afl7lG6DfgM/pDVQDPbb9e8c+79N5f5azF8IfrvKVHZDdH/1zvn7i1YtgU/+WtOijas\nwHf6W8xsEfAO8GPnT+H8Mv7N8wngZ0kVOOdWmtlS4EIz6wv8N3737+X4QO9M0Q7wx1R/bmbz8Hst\nrsSP7r9VsEzn6YwL8Vcq78pTZvYsfjLtW/g3+wz82SF/n7JN5UqVadRR2/Ad6lbgz4v2pq51BddZ\nqnem+G8jPzGzB4EX8N+mxuJ3o6/DT9pPI+tMO02jfrvz0/bTQ4FXotdsFf69fyIwHT9X4huF6+WQ\nKVHbXsfvITkM/3ofC5zpnEuz8YUaZGpms/F5jor+zi+avxwF0dy4rKXK1Dn3K+BXRW3tPOttlXPu\n4aLfKdP9NWI/PRJ41swewG/bwJ/19mfAo865/aaQ5PB5+lRxmZm9FdW7vLh9XWjcflrmzPEn8DPH\ng4+iZTtnvYeWfbxo2TOi8uNStmMWsAEf6vsz9oGbSDmzHX/2xQ3Ab/ATCLcAi4ETU6z7/lkk+LOE\nOvCn9j9B0ZVlo2X3At9LUe/N+DfuFvzl6tvxZwweUU5Otci04O9IeszPM1NgAH437yr8Lt638RuV\nb1PiLMZaZhotb/h5ZstqlWOFmfbAb6yfxQ+MduMHlXcTOAsmp346k32DuE34w76jUz5/LTNN2ra9\nm2emXbwOewlfmVuZ7lu+UftpX/yc0DX4WzPtwh86vQ44uBEyDdQ5gZRX5u4O/dSiynJlZoujhn82\n77akEX1DawdmOuduy7s9jUiZNh9l2nyUafNRptmr+oKT1TKzEfjTRT+Ud1skG8q0+SjT5qNMm48y\nrY3cB0rOuRcIn10j3ZQybT7KtPko0+ajTGuj5jddbWKObnS7A0lFmTYfZdp8lGnzaehMG2KOkoiI\niEgj0h4lERERkQRVz1EyswH4G/B14E8vluq04u9t0+ac25xHA5Rp5pRp81GmzUeZNp9MMs1iMvck\n6nAV0wPQVOp0r6EAZVobyrT5KNPmo0ybT1WZZjFQ6gC49957GTlyZAbVlW/GjBmxsi1btsTKhg8f\nHlx/xYoVsbKFCxfGyoYOHVp228q1evVqpk2bBvne36YD8s101apVsbKHHnooVvbwww/HypJcf/31\nsbLzzz+/vIZVQJkm27Mnfhece+65J7jsvHnzYmWXXBK/J/NFF11UfcNKUKZRAzo6YmVXXnllrOzV\nV18Nrt+nT59YWVtbW6ysZ8+e5TeuTMq0PLfdFr7k0dVXX13nliTLKtMsBkq7AUaOHMmYMWMyqK58\nhx4avw3Ozp3xK+EffnjsdjYAtLTEX4ZRo0bFyo4//vgKWlexPHe75p7p3r17Y2UDB8ZuhVeWIUOG\nxMrq/Pcd0JmG7N4df0kee+yx1OsPHjw4VqZM66d3796xsnIGNaFt70knnRQra21tLa9h1TmgM01r\n0KBBwfIGbXdVmWoyt4iIiEiC3C84mYWnnordk49ly5Jurh23ePHiWNn27durapNUZ9KkSbGy/v37\nx8rmzp0bXH/p0qWxstmzZ8fKLr300gpaJ5UI7T0Kvf5r1qwJrj9s2LBYWeiweeh5WlpagnsvJL3l\ny5fHysaNGxcrC/XJiRMnBuscMWJErGzJkiWxssmTJ6dpotTINddcEysL9cckd911V6ysO217tUdJ\nREREJIEGSiIiIiIJNFASERERSaCBkoiIiEgCDZREREREEjTFaSDjx4+Pla1bty5WFro4HcALL7wQ\nK6vzNZOkSOiCoSHbtm0LlocuhtZIF0JrdqEzz0LXPApdVG/+/PnBOkNnSK1duzZWtnlz/E4Fffv2\nDV5vTdILnfUW2vaWcyZjyKJFi2JlOuutfkJZPfPMM7GyG2+8Mbh+aJsc2h7rrDcRERGRJqCBkoiI\niEgCDZREREREEmigJCIiIpKgKSZzhyZeT5kyJVa2devW4PpJN8uVxpc0QXv06NGxspkzZ9a6ORIJ\nTeYO9b/Q7YdCk7YB+vXrFyubMGFCrOyoo45K00Qp04ABA2JlofxCtxpK2vaGJoPPmTOngtZJJUIT\nr88666xYWWh72tbWFqwzdDLF2WefXUHrGof2KImIiIgk0EBJREREJIEGSiIiIiIJNFASERERSdDt\nJnOHJp+FriQamuS7YsWKYJ0TJ06svmFSc6EJwgsWLAguu2zZslhZa2tr5m2SsNAJEqErLocmfib1\nx9Ak01mzZlXQOqlE6OrYAwcOjJWFJmMvXrw4WGfoKuyajF8/oaxCV7sPlSVlGhLq+92J9iiJiIiI\nJNBASURERCSBBkoiIiIiCTRQEhEREUnQ7SZz33///bGyO+64I1YWupLo4MGDg3Vu2bKl+oZJLoYN\nGxYsD101duzYsbVujnQhNBk4ZPny5anrVKb5Ck28v++++2Jl55xzTnD90F0VpH5mz56dqiwkqZ9O\nmjQpVqYrc4uIiIg0KQ2URERERBJooCQiIiKSQAMlERERkQQaKImIiIgk6HZnvYXOcgnNvg+d9bZ1\n69aatEmyF7pdyfr162NlOmOx+SSdTTN9+vT6NkQqErqtUHe/hYXEJfXTCRMmxMq6++2jtEdJRERE\nJIEGSiIiIiIJNFASERERSaCBkoiIiEiCbjeZOzRJOzRRsFevXrGyuXPn1qRNkr0lS5bEyqZMmRIr\nmzFjRnD9yy+/PPM2SX0sXbo0WK5MG89rr72WarnQbS2ke7vsssuC5c34Oas9SiIiIiIJNFASERER\nSaCBkoiIiEiCLOYotQKsXr06g6pK27NnT6xs48aNqdbdsGFDsHzlypVVtSlLBa9jnlfoqmumIe3t\n7amW27RpU7D8ueeei5X16dOnqjZVSpmWZ9u2bcHyl156KVbWr1+/WjcnSJl6b775ZqrlQv0Rjqhp\nQgAAAJ1JREFU8uuTIco0G6HP2bw+YzPL1DlX1QO4AHB6ZP64oNpslGnDPZRp8z2UafM9lGnzParK\n1KJwKmZmA4BJQAcQv++ElKsVGAq0Oec259EAZZo5Zdp8lGnzUabNJ5NMqx4oiYiIiDQrTeYWERER\nSaCBkoiIiEgCDZREREREEmigJCIiIpJAAyURERGRBBooiYiIiCTQQElEREQkwf8D2T0oCtY7oTwA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1191ceef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mapping = cluster_mapping(y_train, best_model.predict(X_train))\n",
    "\n",
    "y_test_pred = np.array(list(map(lambda x: mapping[x], best_model.predict(X_test))))\n",
    "miscl_img = X_test[y_test != y_test_pred][:25]\n",
    "correct_lab = y_test[y_test != y_test_pred][:25]\n",
    "miscl_lab = y_test_pred[y_test != y_test_pred][:25]\n",
    "\n",
    "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True)\n",
    "ax = ax.flatten()\n",
    "for i in range(25):\n",
    "    img = miscl_img[i].reshape(8, 8)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "    ax[i].set_title('%d) t: %d p: %d' % (i+1, correct_lab[i], miscl_lab[i]))\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Explanation\n",
    "Since the accuracy is 84.4%, which means more than 1 digit will be incorrectly clustered in a group of 10 digits, the error is still considered to be high (compared with using neural networks or other methods).\n",
    "The mis-clustered samples, as we can observe from the picture above, are generally two kinds:\n",
    "1. Hard to differentiate: e.g. No. 14, No. 21 and No, 22. It is even difficult for a human to tell which classes they should belong to.\n",
    "2. Blurred to much: e.g. No. 4, No. 18 and No. 19. Those can be differentiated by a human but seems too vague to be correctly clustered. \n",
    "\n",
    "Furthermore, we can find that digit '5' tends to be mis-clustered to digit '9' (e.g. No. 5, No. 17 and No. 18), and digit '9' tends to be mis-clustered to digit '1' and '7' (e.g. No. 1, No. 2, No. 8, No. 19, No. 23 and No. 24). The reason may be that the distance between those digit clusters are not far, and a digit lying around the border can be easily mis-clustered. For example, when the digit is blurred/vague.<br>\n",
    "Also, we can find from No. 10, No. 14, No. 15 that the digit '1' in the dataset are sometimes twisted, tilted or blurred. It's even hard for a human to tell whether it is digit '1' or not. From the examples, we can see that they tend to be clustered as digit '6' or '2' with respect to their shape and level of clarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tiny image classification\n",
    "\n",
    "We will use the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) for image object recognition.\n",
    "The dataset consists of 50000 training samples and 10000 test samples in 10 different classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck; see the link above for more information).\n",
    "The goal is to maximize the accuracy of your classifier on the test dataset after being optimized via the training dataset.\n",
    "\n",
    "You can use any learning models (supervised or unsupervised) or optimization methods (e.g. search methods for hyper-parameters).\n",
    "The only requirement is that your code can run inside an ipynb file, as usual.\n",
    "Please provide a description of your method, in addition to the code.\n",
    "Your answer will be evaluated not only on the test accuracy but also on the creativity of your methodology and the quality of your explanation/description."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Description\n",
    "\n",
    "## 1. Introduction\n",
    "\n",
    "### 1.1. High-level approach\n",
    "I adopted a Wide Residual Network with 28 convolutional layers and 10x width. As stated Zagoruyko et al's paper <sup>[1](#myfootnote1)</sup>, a 96.11% testing accuracy can be obtained.\n",
    "\n",
    "### 1.2. Reason\n",
    "Without cifar-10 data size-augmentation<sup>[2](#myfootnote2)</sup>, one of the best test results - a 93.57% testing accuracy is reported<sup>[3](#myfootnote3)</sup> with the famous deep residual netwook<sup>[4](#myfootnote4)</sup>. Since the desired testing accuracy is high, I decided to start from resnet. However, inspired by Zagoruyko et al, I choose to use a modified version of resent - the Wide Residual Network. <br><br>\n",
    "Wide resent basically increases the channel of each convolutional layer while decreasing the depth of the network. With result data from their work, a structure of 28 convolution layers and 10x width with dropout achieves the best result - 96.11% accuracy.<br><br>\n",
    "When training my network, however, I only found the result of their early version's paper<sup>[5](#myfootnote5)</sup>, which is training without dropout layer achieves better accuracy (which is also modified from 95.61% to 96.00% in the lastest version). Moreover, due to assignment time constraint, I simply use the default adadelta optimizer instead of [sgd + selected learning rate] in the first 100 itertaions, which may accounts for my less accurate result. After finetuning, in the end, I only get a 94.14% accuarcy, which is still higher than the original resnet.\n",
    "\n",
    "\n",
    "## 2. Method\n",
    "\n",
    "### 2.1 Data Preprocessing\n",
    "\n",
    "Using `keras.preprocessing.image.ImageDataGenerator`, the data preprocessing scheme I use is\n",
    "1. `featurewise_center`: Set feature-wise input mean to 0 over the dataset\n",
    "2. `featurewise_std_normalization`: Divide inputs by feature-wise std of the dataset\n",
    "3. `rotation_range`: 10$^{\\circ}$ range for random rotations\n",
    "4. `width_shift_range`: 15.625% random horizontal shifts\n",
    "5. `height_shift_range`: 15.625% random vertical shifts\n",
    "6. `horizontal_flip`: Randomly flip inputs horizontally\n",
    "\n",
    "Since `vertical_flip` looks unrealistic for a real-world picture, I did not use it. Also, `zca_whitening` may results in loss of image info, and other methods may more or less lead to problems in training.<br>\n",
    "\n",
    "Moreover, I did not adopte cifar-10 data size-augmentation<sup>[2](#myfootnote2)</sup>, which requires structural changes to my network, which may results in more time cost in training and finetuning. As the paper suggests, the test accuracy should increase compared with non-size-augmentated data.\n",
    "\n",
    "### 2.2 Model\n",
    "\n",
    "Since we use 28 convolutional layers, by structure of wide resnet, the layer blocks are:\n",
    "1. A initial block with 1 convolutional layers\n",
    "2. 4 conv1 block with 160 channels. All blocks contains 2 convolutional layers. The first block contains one additional residual convolutional layer. In total 3+2+2+2=9 convolutional layers. Each block ends with a residual merge.\n",
    "3. 4 conv2 block with 320 channels. All blocks contains 2 convolutional layers. The first block contains one additional residual convolutional layer. In total 3+2+2+2=9 convolutional layers. Each block ends with a residual merge.\n",
    "4. 4 conv3 block with 640 channels. All blocks contains 2 convolutional layers. The first block contains one additional residual convolutional layer. In total 3+2+2+2=9 convolutional layers. Each block ends with a residual merge.\n",
    "\n",
    "All convolutional layers are followed by a batch normalization layer and a relu layer.<br>\n",
    "After all layer blocks, we use average pooling to 8-by-8, followed by a flatten layer, and finalliy a fully connected layer (with softmax) of 10 outputs, which corresponds to each class.\n",
    "\n",
    "### 2.3 Hyper-parameter\n",
    "\n",
    "As discussed in 1.2, the most important hyper-parameter of wide resnet, also as Zagoruyko et al discribed, is N (the number of layers), K (the width of the convolutional channel), and dropout ratio. I adopted the best result from version 1 of their work, that is, N = 28, K = 10 (times original resnet), dropout ratio = 0. However, as their version 2 points out, this model with dropout can achieve a better result. Since the difference is just 0.11%, we can see that dropout may not influence much in wide resnet.\n",
    "\n",
    "### 2.4 Detailed Model\n",
    "\n",
    "You may view the detailed model in the following code output.\n",
    "\n",
    "## 3. Result\n",
    "\n",
    "| Network | Accuracy |\n",
    "|------------------------------------------|--------|\n",
    "| Kaiming He et al Resent | 93.57% |\n",
    "| My WRN-28-10 w/o dropout (Adadelta only) | 93.69% |\n",
    "| My WRN-28-10 w/o dropout (Fine-tuned) | 94.14% |\n",
    "| Zagoruyko et al WRN-28-10 v1 w/ dropout | 95.61% |\n",
    "| Zagoruyko et al WRN-28-10 v1 w/o dropout | 95.83% |\n",
    "| Zagoruyko et al WRN-28-10 v2 w/o dropout | 96.00% |\n",
    "| Zagoruyko et al WRN-28-10 v2 w/ dropout | 96.11% |\n",
    "\n",
    "With adadelta optimizer only, my training accuracy finally becomes 100%, which denotes an over-fitting and thus making the testing accuracy fail to improve. With SGD fine-tuning on the model, the testing accuracy increases around 0.45%. However, my model fail to achieve a testing accuracy above 95% as the papaer suggestes. See 4.2 Limitations for more analysis.<br>\n",
    "\n",
    "Note that the code I provided below simply uses the fine-tuned result of my network, and the training process is just for showing the result. They are not the original processes.\n",
    "\n",
    "## 4. Conclusion\n",
    "\n",
    "### 4.1 Traning Conclusion\n",
    "1. Wide Residual Network achieves better result than the original Residual Network.\n",
    "2. Optimizer choice, fine-tuning and data augmentation are curtial for improving the testing rate.\n",
    "3. Ideas that worked several years ago may not work nowadays, ideas that work on a specific model may not work on another model (e.g. importance of dropout, improving the depth always work on resnet)\n",
    "4. The importance of keeping track of the latest news and models on machine learning. \n",
    "\n",
    "### 4.2 Limitations\n",
    "1. Optimizer, Regulation and Over-fitting:\n",
    "With more time provided, I would try to use [different optimizer (adagrad, RMSprop, adam, sgd+specific training rate) + different Regulation (l1, l2)] instead of using adadelta only. This may account for the 1.5% accuracy below the paper's result. \n",
    "2. Dropout:\n",
    "As the second version of the paper suggested, with dropout the testing accuracy improved 0.11%. However, this should not be the major contraint of the result.\n",
    "3. Fine-tuning:\n",
    "I only ran 50 iterations of fine-tuning based on the training rate I set. The peak testing accuracy I obtained is 94.49% but was not saved during fine-tuning.\n",
    "4. Data Augmentation:\n",
    "Cifar-10 data size-augmentation<sup>[2](#myfootnote2)</sup> is proved to be useful to improve the testing rate, which I did not try during my training.\n",
    "5. FCN:\n",
    "Fully Convolutional Network<sup>[6](#myfootnote6)</sup> is proved to be useful to improve the testing rate. Also, R-FCN<sup>[7](#myfootnote7)</sup> provides a nice implimentation of Fully Convolutional Resnet. Maybe a wide, fully convolved residual network can provides a better result.\n",
    "6. Use of GPU:\n",
    "AWS refused to provide me with a GPU instance since I am a new user. I trained my network on my Nvidia GeForce GTX 980M, which takes 7 days to run 100 iterations. This heavily limited adjustments to my model.\n",
    "\n",
    "## 5. References\n",
    "<a name=\"myfootnote1\">1</a>: [Zagoruyko, Sergey, and Nikos Komodakis. \"Wide Residual Networks.\" arXiv preprint arXiv:1605.07146v2 (2016).](https://arxiv.org/pdf/1605.07146v2.pdf)<br>\n",
    "<a name=\"myfootnote2\">2</a>: [Graham, Benjamin. \"Fractional max-pooling.\" arXiv preprint arXiv:1412.6071 (2015).](https://arxiv.org/pdf/1412.6071v4.pdf)<br>\n",
    "<a name=\"myfootnote3\">3</a>: [Benenson. What is the class of this image ?\" rodrigob@github](http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html)<br>\n",
    "<a name=\"myfootnote4\">4</a>: [He, Kaiming, et al. \"Deep residual learning for image recognition.\" arXiv preprint arXiv:1512.03385 (2015).](https://arxiv.org/pdf/1512.03385v1.pdf)<br>\n",
    "<a name=\"myfootnote5\">5</a>: [Zagoruyko, Sergey, and Nikos Komodakis. \"Wide Residual Networks.\" arXiv preprint arXiv:1605.07146v1 (2016).](https://arxiv.org/pdf/1605.07146v1.pdf)<br>\n",
    "<a name=\"myfootnote6\">6</a>: [Long, Jonathan, Evan Shelhamer, and Trevor Darrell. \"Fully convolutional networks for semantic segmentation.\" IEEE Conference on CVPR. 2015.](http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf)<br>\n",
    "<a name=\"myfootnote7\">7</a>: [Dai, Jifeng, et al. \"R-FCN: Object Detection via Region-based Fully Convolutional Networks.\" arXiv preprint arXiv:1605.06409 (2016).](https://arxiv.org/pdf/1605.06409.pdf)<br>\n",
    "All codes referenced have been specified in their context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n",
      "Using gpu device 0: GeForce GTX 980M (CNMeM is disabled, cuDNN 5005)\n"
     ]
    }
   ],
   "source": [
    "# Functions to build a user-defined wide resnet for cifa-10\n",
    "# Author: Somshubra Majumdar https://github.com/titu1994/Wide-Residual-Networks\n",
    "# Modified By: Gao Chang, HKU\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, merge, Activation, Dropout, Flatten, Dense\n",
    "from keras.layers.convolutional import Convolution2D, MaxPooling2D, AveragePooling2D\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras import backend as K\n",
    "\n",
    "def initial_conv(input):\n",
    "    x = Convolution2D(16, 3, 3, border_mode='same')(input)\n",
    "\n",
    "    channel_axis = 1 if K.image_dim_ordering() == \"th\" else -1\n",
    "\n",
    "    x = BatchNormalization(axis=channel_axis)(x)\n",
    "    x = Activation('relu')(x)\n",
    "    return x\n",
    "\n",
    "def conv1_block(input, k=1, dropout=0.0):\n",
    "    init = input\n",
    "\n",
    "    channel_axis = 1 if K.image_dim_ordering() == \"th\" else -1\n",
    "\n",
    "    # Check if input number of filters is same as 16 * k, else create convolution2d for this input\n",
    "    if K.image_dim_ordering() == \"th\":\n",
    "        if init._keras_shape[1] != 16 * k:\n",
    "            init = Convolution2D(16 * k, 1, 1, activation='linear', border_mode='same')(init)\n",
    "    else:\n",
    "        if init._keras_shape[-1] != 16 * k:\n",
    "            init = Convolution2D(16 * k, 1, 1, activation='linear', border_mode='same')(init)\n",
    "\n",
    "    x = Convolution2D(16 * k, 3, 3, border_mode='same')(input)\n",
    "    x = BatchNormalization(axis=channel_axis)(x)\n",
    "    x = Activation('relu')(x)\n",
    "\n",
    "    if dropout > 0.0: x = Dropout(dropout)(x)\n",
    "\n",
    "    x = Convolution2D(16 * k, 3, 3, border_mode='same')(x)\n",
    "    x = BatchNormalization(axis=channel_axis)(x)\n",
    "    x = Activation('relu')(x)\n",
    "\n",
    "    m = merge([init, x], mode='sum')\n",
    "    return m\n",
    "\n",
    "def conv2_block(input, k=1, dropout=0.0):\n",
    "    init = input\n",
    "\n",
    "    channel_axis = 1 if K.image_dim_ordering() == \"th\" else -1\n",
    "\n",
    "    # Check if input number of filters is same as 32 * k, else create convolution2d for this input\n",
    "    if K.image_dim_ordering() == \"th\":\n",
    "        if init._keras_shape[1] != 32 * k:\n",
    "            init = Convolution2D(32 * k, 1, 1, activation='linear', border_mode='same')(init)\n",
    "    else:\n",
    "        if init._keras_shape[-1] != 32 * k:\n",
    "            init = Convolution2D(32 * k, 1, 1, activation='linear', border_mode='same')(init)\n",
    "\n",
    "    x = Convolution2D(32 * k, 3, 3, border_mode='same')(input)\n",
    "    x = BatchNormalization(axis=channel_axis)(x)\n",
    "    x = Activation('relu')(x)\n",
    "\n",
    "    if dropout > 0.0: x = Dropout(dropout)(x)\n",
    "\n",
    "    x = Convolution2D(32 * k, 3, 3, border_mode='same')(x)\n",
    "    x = BatchNormalization(axis=channel_axis)(x)\n",
    "    x = Activation('relu')(x)\n",
    "\n",
    "    m = merge([init, x], mode='sum')\n",
    "    return m\n",
    "\n",
    "def conv3_block(input, k=1, dropout=0.0):\n",
    "    init = input\n",
    "\n",
    "    channel_axis = 1 if K.image_dim_ordering() == \"th\" else -1\n",
    "\n",
    "    # Check if input number of filters is same as 64 * k, else create convolution2d for this input\n",
    "    if K.image_dim_ordering() == \"th\":\n",
    "        if init._keras_shape[1] != 64 * k:\n",
    "            init = Convolution2D(64 * k, 1, 1, activation='linear', border_mode='same')(init)\n",
    "    else:\n",
    "        if init._keras_shape[-1] != 64 * k:\n",
    "            init = Convolution2D(64 * k, 1, 1, activation='linear', border_mode='same')(init)\n",
    "\n",
    "    x = Convolution2D(64 * k, 3, 3, border_mode='same')(input)\n",
    "    x = BatchNormalization(axis=channel_axis)(x)\n",
    "    x = Activation('relu')(x)\n",
    "\n",
    "    if dropout > 0.0: x = Dropout(dropout)(x)\n",
    "\n",
    "    x = Convolution2D(64 * k, 3, 3, border_mode='same')(x)\n",
    "    x = BatchNormalization(axis=channel_axis)(x)\n",
    "    x = Activation('relu')(x)\n",
    "\n",
    "    m = merge([init, x], mode='sum')\n",
    "    return m\n",
    "\n",
    "def WRN(nb_classes, N, k, dropout):\n",
    "    \"\"\"\n",
    "    Creates a Wide Residual Network with specified parameters\n",
    "    :param nb_classes: Number of output classes\n",
    "    :param N: Depth of the network. Compute N = (n - 4) / 6.\n",
    "              Example : For a depth of 16, n = 16, N = (16 - 4) / 6 = 2\n",
    "              Example2: For a depth of 28, n = 28, N = (28 - 4) / 6 = 4\n",
    "              Example3: For a depth of 40, n = 40, N = (40 - 4) / 6 = 6\n",
    "    :param k: Width of the network.\n",
    "    :param dropout: Adds dropout if value is greater than 0.0\n",
    "    \"\"\"\n",
    "    init = Input(shape=(3, 32, 32))\n",
    "    \n",
    "    x = initial_conv(init)\n",
    "\n",
    "    for i in range(N):\n",
    "        x = conv1_block(x, k, dropout)\n",
    "        \n",
    "    x = MaxPooling2D((2,2))(x)\n",
    "\n",
    "    for i in range(N):\n",
    "        x = conv2_block(x, k, dropout)\n",
    "\n",
    "    x = MaxPooling2D((2,2))(x)\n",
    "\n",
    "    for i in range(N):\n",
    "        x = conv3_block(x, k, dropout)\n",
    "\n",
    "    x = AveragePooling2D((8,8))(x)\n",
    "    x = Flatten()(x)\n",
    "    x = Dense(nb_classes, activation='softmax')(x)\n",
    "\n",
    "    model = Model(init, x)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "input_1 (InputLayer)             (None, 3, 32, 32)     0                                            \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_1 (Convolution2D)  (None, 16, 32, 32)    448         input_1[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_1 (BatchNormal(None, 16, 32, 32)    32          convolution2d_1[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "activation_1 (Activation)        (None, 16, 32, 32)    0           batchnormalization_1[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_3 (Convolution2D)  (None, 160, 32, 32)   23200       activation_1[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_2 (BatchNormal(None, 160, 32, 32)   320         convolution2d_3[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "activation_2 (Activation)        (None, 160, 32, 32)   0           batchnormalization_2[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_4 (Convolution2D)  (None, 160, 32, 32)   230560      activation_2[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_3 (BatchNormal(None, 160, 32, 32)   320         convolution2d_4[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_2 (Convolution2D)  (None, 160, 32, 32)   2720        activation_1[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "activation_3 (Activation)        (None, 160, 32, 32)   0           batchnormalization_3[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "merge_1 (Merge)                  (None, 160, 32, 32)   0           convolution2d_2[0][0]            \n",
      "                                                                   activation_3[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_5 (Convolution2D)  (None, 160, 32, 32)   230560      merge_1[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_4 (BatchNormal(None, 160, 32, 32)   320         convolution2d_5[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "activation_4 (Activation)        (None, 160, 32, 32)   0           batchnormalization_4[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_6 (Convolution2D)  (None, 160, 32, 32)   230560      activation_4[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_5 (BatchNormal(None, 160, 32, 32)   320         convolution2d_6[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "activation_5 (Activation)        (None, 160, 32, 32)   0           batchnormalization_5[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "merge_2 (Merge)                  (None, 160, 32, 32)   0           merge_1[0][0]                    \n",
      "                                                                   activation_5[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_7 (Convolution2D)  (None, 160, 32, 32)   230560      merge_2[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_6 (BatchNormal(None, 160, 32, 32)   320         convolution2d_7[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "activation_6 (Activation)        (None, 160, 32, 32)   0           batchnormalization_6[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_8 (Convolution2D)  (None, 160, 32, 32)   230560      activation_6[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_7 (BatchNormal(None, 160, 32, 32)   320         convolution2d_8[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "activation_7 (Activation)        (None, 160, 32, 32)   0           batchnormalization_7[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "merge_3 (Merge)                  (None, 160, 32, 32)   0           merge_2[0][0]                    \n",
      "                                                                   activation_7[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_9 (Convolution2D)  (None, 160, 32, 32)   230560      merge_3[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_8 (BatchNormal(None, 160, 32, 32)   320         convolution2d_9[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "activation_8 (Activation)        (None, 160, 32, 32)   0           batchnormalization_8[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_10 (Convolution2D) (None, 160, 32, 32)   230560      activation_8[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_9 (BatchNormal(None, 160, 32, 32)   320         convolution2d_10[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_9 (Activation)        (None, 160, 32, 32)   0           batchnormalization_9[0][0]       \n",
      "____________________________________________________________________________________________________\n",
      "merge_4 (Merge)                  (None, 160, 32, 32)   0           merge_3[0][0]                    \n",
      "                                                                   activation_9[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "maxpooling2d_1 (MaxPooling2D)    (None, 160, 16, 16)   0           merge_4[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_12 (Convolution2D) (None, 320, 16, 16)   461120      maxpooling2d_1[0][0]             \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_10 (BatchNorma(None, 320, 16, 16)   640         convolution2d_12[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_10 (Activation)       (None, 320, 16, 16)   0           batchnormalization_10[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_13 (Convolution2D) (None, 320, 16, 16)   921920      activation_10[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_11 (BatchNorma(None, 320, 16, 16)   640         convolution2d_13[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_11 (Convolution2D) (None, 320, 16, 16)   51520       maxpooling2d_1[0][0]             \n",
      "____________________________________________________________________________________________________\n",
      "activation_11 (Activation)       (None, 320, 16, 16)   0           batchnormalization_11[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_5 (Merge)                  (None, 320, 16, 16)   0           convolution2d_11[0][0]           \n",
      "                                                                   activation_11[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_14 (Convolution2D) (None, 320, 16, 16)   921920      merge_5[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_12 (BatchNorma(None, 320, 16, 16)   640         convolution2d_14[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_12 (Activation)       (None, 320, 16, 16)   0           batchnormalization_12[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_15 (Convolution2D) (None, 320, 16, 16)   921920      activation_12[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_13 (BatchNorma(None, 320, 16, 16)   640         convolution2d_15[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_13 (Activation)       (None, 320, 16, 16)   0           batchnormalization_13[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_6 (Merge)                  (None, 320, 16, 16)   0           merge_5[0][0]                    \n",
      "                                                                   activation_13[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_16 (Convolution2D) (None, 320, 16, 16)   921920      merge_6[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_14 (BatchNorma(None, 320, 16, 16)   640         convolution2d_16[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_14 (Activation)       (None, 320, 16, 16)   0           batchnormalization_14[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_17 (Convolution2D) (None, 320, 16, 16)   921920      activation_14[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_15 (BatchNorma(None, 320, 16, 16)   640         convolution2d_17[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_15 (Activation)       (None, 320, 16, 16)   0           batchnormalization_15[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_7 (Merge)                  (None, 320, 16, 16)   0           merge_6[0][0]                    \n",
      "                                                                   activation_15[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_18 (Convolution2D) (None, 320, 16, 16)   921920      merge_7[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_16 (BatchNorma(None, 320, 16, 16)   640         convolution2d_18[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_16 (Activation)       (None, 320, 16, 16)   0           batchnormalization_16[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_19 (Convolution2D) (None, 320, 16, 16)   921920      activation_16[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_17 (BatchNorma(None, 320, 16, 16)   640         convolution2d_19[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_17 (Activation)       (None, 320, 16, 16)   0           batchnormalization_17[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_8 (Merge)                  (None, 320, 16, 16)   0           merge_7[0][0]                    \n",
      "                                                                   activation_17[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "maxpooling2d_2 (MaxPooling2D)    (None, 320, 8, 8)     0           merge_8[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_21 (Convolution2D) (None, 640, 8, 8)     1843840     maxpooling2d_2[0][0]             \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_18 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_21[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_18 (Activation)       (None, 640, 8, 8)     0           batchnormalization_18[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_22 (Convolution2D) (None, 640, 8, 8)     3687040     activation_18[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_19 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_22[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_20 (Convolution2D) (None, 640, 8, 8)     205440      maxpooling2d_2[0][0]             \n",
      "____________________________________________________________________________________________________\n",
      "activation_19 (Activation)       (None, 640, 8, 8)     0           batchnormalization_19[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_9 (Merge)                  (None, 640, 8, 8)     0           convolution2d_20[0][0]           \n",
      "                                                                   activation_19[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_23 (Convolution2D) (None, 640, 8, 8)     3687040     merge_9[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_20 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_23[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_20 (Activation)       (None, 640, 8, 8)     0           batchnormalization_20[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_24 (Convolution2D) (None, 640, 8, 8)     3687040     activation_20[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_21 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_24[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_21 (Activation)       (None, 640, 8, 8)     0           batchnormalization_21[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_10 (Merge)                 (None, 640, 8, 8)     0           merge_9[0][0]                    \n",
      "                                                                   activation_21[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_25 (Convolution2D) (None, 640, 8, 8)     3687040     merge_10[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_22 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_25[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_22 (Activation)       (None, 640, 8, 8)     0           batchnormalization_22[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_26 (Convolution2D) (None, 640, 8, 8)     3687040     activation_22[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_23 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_26[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_23 (Activation)       (None, 640, 8, 8)     0           batchnormalization_23[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_11 (Merge)                 (None, 640, 8, 8)     0           merge_10[0][0]                   \n",
      "                                                                   activation_23[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_27 (Convolution2D) (None, 640, 8, 8)     3687040     merge_11[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_24 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_27[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_24 (Activation)       (None, 640, 8, 8)     0           batchnormalization_24[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "convolution2d_28 (Convolution2D) (None, 640, 8, 8)     3687040     activation_24[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "batchnormalization_25 (BatchNorma(None, 640, 8, 8)     1280        convolution2d_28[0][0]           \n",
      "____________________________________________________________________________________________________\n",
      "activation_25 (Activation)       (None, 640, 8, 8)     0           batchnormalization_25[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "merge_12 (Merge)                 (None, 640, 8, 8)     0           merge_11[0][0]                   \n",
      "                                                                   activation_25[0][0]              \n",
      "____________________________________________________________________________________________________\n",
      "averagepooling2d_1 (AveragePoolin(None, 640, 1, 1)     0           merge_12[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)              (None, 640)           0           averagepooling2d_1[0][0]         \n",
      "____________________________________________________________________________________________________\n",
      "dense_1 (Dense)                  (None, 10)            6410        flatten_1[0][0]                  \n",
      "====================================================================================================\n",
      "Total params: 36489290\n",
      "____________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import sklearn.metrics as metrics\n",
    "from keras.datasets import cifar10\n",
    "from keras.models import Model\n",
    "from keras.layers import Input\n",
    "from keras.optimizers import SGD\n",
    "import keras.callbacks as callbacks\n",
    "import keras.utils.np_utils as kutils\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "batch_size = 64\n",
    "nb_epoch = 5\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = cifar10.load_data()\n",
    "\n",
    "X_train = X_train.astype('float32')\n",
    "X_train /= 255.0\n",
    "X_test = X_test.astype('float32')\n",
    "X_test /= 255.0\n",
    "\n",
    "y_train = kutils.to_categorical(y_train)\n",
    "y_test = kutils.to_categorical(y_test)\n",
    "\n",
    "generator = ImageDataGenerator(featurewise_center=True,\n",
    "                               featurewise_std_normalization=True,\n",
    "                               rotation_range=10,\n",
    "                               width_shift_range=5./32,\n",
    "                               height_shift_range=5./32,\n",
    "                               horizontal_flip=True)\n",
    "\n",
    "generator.fit(X_train, seed=0, augment=True)\n",
    "\n",
    "test_generator = ImageDataGenerator(featurewise_center=True,\n",
    "                                    featurewise_std_normalization=True)\n",
    "\n",
    "test_generator.fit(X_test, seed=0, augment=True)\n",
    "\n",
    "model = WRN(nb_classes=10, N=4, k=10, dropout=0.0)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start Training:\n",
      "Epoch 1/5\n",
      "50000/50000 [==============================] - 5208s - loss: 3.1280e-04 - acc: 1.0000 - val_loss: 0.3443 - val_acc: 0.9409\n",
      "Epoch 2/5\n",
      "50000/50000 [==============================] - 5009s - loss: 0.0023 - acc: 0.9992 - val_loss: 0.3471 - val_acc: 0.9406\n",
      "Epoch 3/5\n",
      "50000/50000 [==============================] - 5013s - loss: 0.0080 - acc: 0.9972 - val_loss: 0.3442 - val_acc: 0.9408\n",
      "Epoch 4/5\n",
      "50000/50000 [==============================] - 5013s - loss: 0.0086 - acc: 0.9970 - val_loss: 0.3287 - val_acc: 0.9444\n",
      "Epoch 5/5\n",
      "50000/50000 [==============================] - 5002s - loss: 0.0080 - acc: 0.9972 - val_loss: 0.3495 - val_acc: 0.9393\n"
     ]
    }
   ],
   "source": [
    "print (\"Start Training:\")\n",
    "\n",
    "sgd = SGD(lr = 0.001, decay = 0.1, momentum = 0.9, nesterov = True)\n",
    "\n",
    "model.compile(loss=\"categorical_crossentropy\", optimizer=sgd, metrics=[\"acc\"])\n",
    "\n",
    "# model.load_weights(\"WRN-28-10.h5\")\n",
    "\n",
    "model.fit_generator(generator.flow(X_train, y_train, batch_size=batch_size),\n",
    "                    samples_per_epoch=len(X_train),\n",
    "                    nb_epoch=nb_epoch,\n",
    "#                     callbacks=[callbacks.ModelCheckpoint(\"WRN-28-10-Best.h5\", monitor=\"val_acc\", save_best_only=True)],\n",
    "                    validation_data=test_generator.flow(X_test, y_test, batch_size=batch_size),\n",
    "                    nb_val_samples=X_test.shape[0],\n",
    "                    verbose = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start Testing:\n",
      "Results:\n",
      "Test loss: 0.3487206139005721\n",
      "Test accuracy: 0.9414\n"
     ]
    }
   ],
   "source": [
    "print (\"Start Testing:\")\n",
    "\n",
    "# model.load_weights(\"WRN-28-10.h5\")\n",
    "results = model.evaluate_generator(test_generator.flow(X_test, y_test, batch_size=batch_size), X_test.shape[0])\n",
    "\n",
    "print (\"Results:\")\n",
    "print (\"Test loss: {0}\".format(results[0]))\n",
    "print (\"Test accuracy: {0}\".format(results[1]))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "name": "_merged"
 },
 "nbformat": 4,
 "nbformat_minor": 0
}