example-cls-layer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OptNet/qpth Example Notebook\n",
    "\n",
    "*By [Brandon Amos](https://bamos.github.io) and [J. Zico Kolter](http://zicokolter.com/).*\n",
    "\n",
    "---\n",
    "\n",
    "This notebook is released along with our paper\n",
    "[OptNet: Differentiable Optimization as a Layer in Neural Networks](https://arxiv.org/abs/1703.00443).\n",
    "\n",
    "This notebook shows a minimal example of constructing an\n",
    "OptNet layer in PyTorch with our [qpth library](https://github.com/locuslab/qpth).\n",
    "See [our qpth documentation page](https://locuslab.github.io/qpth/)\n",
    "for more details.\n",
    "The experiments for our paper that use this library are in\n",
    "[this repo](https://github.com/locuslab/optnet).\n",
    "\n",
    "\n",
    "## Setup and Dependencies\n",
    "\n",
    "+ Python/numpy/[PyTorch](https://pytorch.org)\n",
    "+ [qpth](https://github.com/locuslab/qpth):\n",
    "  *Our fast QP solver for PyTorch released in conjunction with this paper.*\n",
    "+ [bamos/block](https://github.com/bamos/block):\n",
    "  *Our intelligent block matrix library for numpy, PyTorch, and beyond.*\n",
    "+ Optional: [bamos/setGPU](https://github.com/bamos/setGPU):\n",
    "  A small library to set `CUDA_VISIBLE_DEVICES` on multi-GPU systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Function, Variable\n",
    "from torch.nn.parameter import Parameter\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from qpth.qp import QPFunction\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('bmh')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define the model\n",
    "\n",
    "+ We'll be using a network architecture that looks like:\n",
    "\n",
    "```\n",
    "FC-ReLU-(BN)-FC-ReLU-(BN)-QP-softmax\n",
    "```\n",
    "\n",
    "where the QP OptNet layer learns the coefficients `Q`, `q`, `G`, and `h` for\n",
    "a QP with inequality constraints:\n",
    "\n",
    "```\n",
    "z_{i+1} = argmin_z 0.5 z^T Q z + q^t z\n",
    "          s.t. Gz <= h\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class OptNet(nn.Module):\n",
    "    def __init__(self, nFeatures, nHidden, nCls, bn, nineq=200, neq=0, eps=1e-4):\n",
    "        super().__init__()\n",
    "        self.nFeatures = nFeatures\n",
    "        self.nHidden = nHidden\n",
    "        self.bn = bn\n",
    "        self.nCls = nCls\n",
    "        self.nineq = nineq\n",
    "        self.neq = neq\n",
    "        self.eps = eps\n",
    "\n",
    "        # Normal BN/FC layers.\n",
    "        if bn:\n",
    "            self.bn1 = nn.BatchNorm1d(nHidden)\n",
    "            self.bn2 = nn.BatchNorm1d(nCls)\n",
    "\n",
    "        self.fc1 = nn.Linear(nFeatures, nHidden)\n",
    "        self.fc2 = nn.Linear(nHidden, nCls)\n",
    "\n",
    "        # QP params.\n",
    "        self.M = Variable(torch.tril(torch.ones(nCls, nCls)))\n",
    "        self.L = Parameter(torch.tril(torch.rand(nCls, nCls)))\n",
    "        self.G = Parameter(torch.Tensor(nineq,nCls).uniform_(-1,1))\n",
    "        self.z0 = Parameter(torch.zeros(nCls))\n",
    "        self.s0 = Parameter(torch.ones(nineq))\n",
    "\n",
    "    def forward(self, x):\n",
    "        nBatch = x.size(0)\n",
    "\n",
    "        # Normal FC network.\n",
    "        x = x.view(nBatch, -1)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        if self.bn:\n",
    "            x = self.bn1(x)\n",
    "        x = F.relu(self.fc2(x))\n",
    "        if self.bn:\n",
    "            x = self.bn2(x)\n",
    "\n",
    "        # Set up the qp parameters Q=LL^T and h = Gz_0+s_0.\n",
    "        L = self.M*self.L\n",
    "        Q = L.mm(L.t()) + self.eps*Variable(torch.eye(self.nCls))\n",
    "        h = self.G.mv(self.z0)+self.s0\n",
    "        e = Variable(torch.Tensor())\n",
    "        x = QPFunction(verbose=-1)(Q, x, self.G, h, e, e)\n",
    "\n",
    "        return F.log_softmax(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the network\n",
    "\n",
    "+ Create random data for a regression task and then optimize the parameters with Adam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create random data\n",
    "nBatch, nFeatures, nHidden, nCls = 16, 20, 20, 2\n",
    "x = Variable(torch.randn(nBatch, nFeatures), requires_grad=False)\n",
    "y = Variable((torch.rand(nBatch) < 0.5).long(), requires_grad=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0, loss = 0.68\n",
      "Iteration 25, loss = 0.55\n",
      "Iteration 50, loss = 0.53\n",
      "Iteration 75, loss = 0.48\n",
      "Iteration 100, loss = 0.47\n",
      "Iteration 125, loss = 0.45\n",
      "Iteration 150, loss = 0.47\n",
      "Iteration 175, loss = 0.32\n",
      "Iteration 200, loss = 0.30\n",
      "Iteration 225, loss = 0.30\n",
      "Iteration 250, loss = 0.27\n",
      "Iteration 275, loss = 0.26\n",
      "Iteration 300, loss = 0.25\n",
      "Iteration 325, loss = 0.24\n",
      "Iteration 350, loss = 0.23\n",
      "Iteration 375, loss = 0.22\n",
      "Iteration 400, loss = 0.21\n",
      "Iteration 425, loss = 0.20\n",
      "Iteration 450, loss = 0.20\n",
      "Iteration 475, loss = 0.20\n"
     ]
    }
   ],
   "source": [
    "# Initialize the model.\n",
    "model = OptNet(nFeatures, nHidden, nCls, bn=False)\n",
    "loss_fn = torch.nn.CrossEntropyLoss()\n",
    "\n",
    "# Initialize the optimizer.\n",
    "learning_rate = 1e-3\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "\n",
    "losses = []\n",
    "for t in range(500):\n",
    "    # Forward pass: compute predicted y by passing x to the model.\n",
    "    y_pred = model(x)\n",
    "\n",
    "    # Compute and print loss.\n",
    "    loss = loss_fn(y_pred, y)\n",
    "    if t % 25 == 0:\n",
    "        print('Iteration {}, loss = {:.2f}'.format(t, loss.data[0]))\n",
    "    losses.append(loss.data[0])\n",
    "\n",
    "    # Before the backward pass, use the optimizer object to zero all of the\n",
    "    # gradients for the variables it will update (which are the learnable weights\n",
    "    # of the model)\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    # Backward pass: compute gradient of the loss with respect to model\n",
    "    # parameters\n",
    "    loss.backward()\n",
    "\n",
    "    # Calling the step function on an Optimizer makes an update to its\n",
    "    # parameters\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 0.70174154117703436)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEJCAYAAAByupuRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXl8ZGWV//8+qaSyp7J20ns30E3TIIsCDYILtgs6Cn57\ncAYccfTbjN+Zga+O+nVchxl19Deb2yiOK+IyiIAbg0ijiHuzNM3SGw29dzqppLJVpZKqVKVyfn/c\nSnUlnXQqyb1VlTzP+/XKK/fe56l7zye3cs99tnNEVbFYLBaLmZQU2gCLxWKxFA7rBCwWi8VgrBOw\nWCwWg7FOwGKxWAzGOgGLxWIxGOsELBaLxWDy5gRE5GoR2S8iB0TkQ1OUf05Enk7/PC8iA/myzWKx\nWExF8rFOQER8wPPAa4B24AngBlXdO039/wtcpKr/23PjLBaLxWDy1RK4FDigqodUNQHcBVx7mvo3\nAN/Pi2UWi8ViMKV5us5y4HjWfjuwaaqKIrIaWAv8aqryBx98UDs7OxERVJWGhgZaWlpIJpP4fD4A\nUqkUZWVljI6OAlBaWjqn8mQyiYjg8/kYHR3F5/OhqoyNjWXKS0pKKCkpYXR0lNLSUsbGxmZdLiKk\nUilKS0tJpVKoaqbcarKarCarab6aEolEz+bNm1umeqbmywnMhuuBe1U1NVVhIBBg06Yp/ceMHD16\nlNWrV8/HtgWH1WwGVrMZzFXzzp07j05Xlq/uoBPAyqz9FeljU3E9HnUFlZWVeXHaosZqNgOr2Qy8\n0JwvJ/AEsE5E1oqIH+dBf9/kSiKyAWgAtnthRCAQ8OK0RY3VbAZWsxl4oTkvTkBVR4FbgG3APuBu\nVd0jIp8QkWuyql4P3KUeTVnq6enx4rRFjdVsBlazGXihOW9jAqr6APDApGO3Ttr/Jy9tsG8OZmA1\nm4HV7A5GrBjujia486kgP3++v9Cm5J1EIlFoE/KO1WwGVrM7GOEEeoeT3PFkJ786PFhoU/JOLBYr\ntAl5x2o2A6vZHYxwAo2Vzoh6NCUFtiT/tLW1FdqEvGM1m4HV7A5GOIGGKmfoYyA2yphh6TSDwWCh\nTcg7VrMZWM3uYIQT8PtKqC33kVIYHJlyDdqixe/3F9qEvGM1m4HV7A5GOAGAhnSXUN9wssCW5Jfa\n2tpCm5B3rGYzsJrdwSAn4HQJ9cfMcgK9vb2FNiHvWM1mYDW7gzFOoLFqvCUwWmBL8ktDQ0OhTcg7\nVrMZWM3uYIwTaKl2nEAwatbcYjuNzgysZjOwU0TnwfJABQAd4XiBLckv8bhZesFqNgWr2R3McQJ1\n5QC0h0cKbEl+sXOpzcBqNgO7TmAerAg4TuBEZIR8pNQsFuxcajOwms3ArhOYBw2VpQTKSxgcSXGg\n15y+xIqKikKbkHesZjOwmt3BGCcgIrx0ZQ0A33my05jWQGVlZaFNyDtWsxlYze5gjBMAuHyJUOYT\nHjseYXfXUKHNyQv9/eZFTrWazcBqdgejnMB5q5aw+cxGAA70DBfYmvzQ1NRUaBPyjtVsBlazOxjl\nBAYHBzmjyWlOHek3Y3rZ4KB54bOtZjOwmt3BKCeQSCRY2+AMrBzuM2Nw2CbeMAOr2Qy80Jy39JLF\nQFtbGw3qA5yWwJgqJbK4cwzYudRmYDWbgV0nME+CwSB1FaU0VpUSHx2ja3Dxv0nYudRmYDWbgV0n\nME/Gp1etbXB+H+5f/F1CdhqdGVjNZmCniM6T8YQMaxvTTqBv8Q8O28QbZmA1m8GCTiojIleLyH4R\nOSAiH5qmzp+JyF4R2SMid7ptQzgcBmBNenD4iAGDw+OaTcJqNgOr2R3yMjAsIj7gNuA1QDvwhIjc\np6p7s+qsAz4MXKGq/SKyxG07mpubgayWgAHTRMc1m4TVbAZWszvkqyVwKXBAVQ+pagK4C7h2Up2/\nAm5T1X4AVe1224hxL7qqvoISgfZwnMTomNuXKSrs25IZWM1msGBbAsBy4HjWfjuwaVKd9QAi8gfA\nB/yTqj44+UTd3d1s3bqV0tJSUqkUW7Zs4eabbyYYDFJdXY3P5yMSidDS0kJfXx+qSktLC11dXcRi\nMcrLy4lGo6yoK+dYeITtzx3lkjNaCYVC1NXVkUqlGBoaoq2tjWAwSFlZGYFAgJ6eHgKBAIlEglgs\nlin3+/3U1tbS29tLQ0MDsViMeDyeKa+oqKCyspL+/n6ampoYHBwkkUhkyisrK/H7/YTDYZqbmwmH\nwySTyUz5TJpqapx4SNFolNZWR4eI0NjYSCgUIhaL4ff7F5Wmme5TPB6nu7t7UWma6T6NjIzQ1dW1\nqDTNdJ8SiQSdnZ2LStNM9ymRSNDR0TEnTdMh+QikJiLXAVer6k3p/RuBTap6S1ad+4Ek8GfACuC3\nwItUdSD7XNu3b9cNGzbMyY6RkRHKy52Q0p/57VG2Pd/H31y2nP91nus9T0VDtmZTsJrNwGrOnZ07\ndz65efPmi6cqy1d30AlgZdb+ivSxbNqB+1Q1qaqHgeeBdW4akT3HdsOSagCeCy3uGEJ2LrUZWM1m\nsJDXCTwBrBORtSLiB64H7ptU5yfAKwFEpBmne+iQm0ZUV1dntje0VAGwr3txRxPN1mwKVrMZWM3u\nkBcnoKqjwC3ANmAfcLeq7hGRT4jINelq24BeEdkLPAJ8QFV73bTD5/Nlttc0VFJeWkJwMMFALOnm\nZYqKbM2mYDWbgdXsDnlbJ6CqD6jqelU9U1U/lT52q6rel95WVX2fqm5U1Rep6l1u2xCJRDLbvhJh\nfbPTGljMXULZmk3BajYDq9kdjFox3NLSMmF/vEvouUXcJTRZswlYzWZgNbuDUU6gr69vwv456cHh\nfd2ntgQSo2M88FwPfcMLu6tosmYTsJrNwGp2B6OcwOTpsONOYH9oiNTYxLLvPRXk878/zse2Hcyb\nfV5gSi7lbKxmM7Ca3cEoJzC5KdVUXcbyunKGk2M8Pynd5OPHnZV5B3qnjy+0JxjloeddHbt2Hdtk\nNgOr2Qxsd9A86erqOuXYhcuc1XQ7T0xM25aLw33v/S/wH789VtRZyqbSvNixms3AanYHo5zAVMun\nL10ZAOAPRyYsTGY2EYV6hop33GCmJeOLEavZDKxmdzDKCUzFS5bXUlVWwoHeGM90ZLUGZmgJZAee\nGx0zr2/SYrEsDoxyAtFo9JRj/tISrtno9LN94IED/L/7X+Bnz/XQMTiSqROfItLoQHw0sz04MnpK\nebEwlebFjtVsBlazOxiVaL61tXXK439xURuDI6P88kA/zwajPBuc+IfujiZYVV8x4Vg4ywlkbxcb\n02lezFjNZmA1u4NRLYFQKDTl8fLSEt5z5Srueut5vOfKlVy+OoBklf/1j57jIw8e4IHneugfTqKq\nEx78kSJ2AtNpXsxYzWZgNbuDUS0BETltebXfx59saOZPNjQzODLK3q4h7t3Vza5glB3tg+xoH+Tz\nHGdprZ/16dXGAJGRlNemz5mZNC9GrGYzsJrdwSgn0NjYmHPd2vJSNq0KsGlVgIFYku3HIvzhyABP\ntkfoHEzQOZjI1D3cF6MzMkJbrb/ovpiz0bxYsJrNwGp2B9sdlAP1lWW8/uwm/vl1Z3L/Oy/kw1et\n5rJVdZny50LD/OXde3nTHc/w9cdOsKM9csoK5EJhm8xmYDWbge0Omid1dXUzV5oBX4lw1ZmNXHVm\nI6rK9mNhHjnYz28ODZBIKffs6uaeXd3UV5Ty6nWNvGxtfSY8RSFwQ/NCw2o2A6vZHYxyAqmUu333\nIsJLV9fz0tX19A+/kJlVVO33MRAf5d5d3dy7q5tNK+v4m8tXsKwu/6nw3Na8ELCazcBqdgejuoOG\nhrwLGV1RdvJP+f6XreIL16znjRuaqSwr4bHjEd71w33c+VSQZGo2a5Hnj5eaixWr2QysZncwygm0\ntbV5du6K0pN/yoqyEs5ZUs27r1zJHW/ZyKvPaiCRUu54spO/+fF+Hj8ezlsERC81FytWsxlYze6Q\nkxMQkfeJyIXp7ctE5JiIHBaRy123yEO8TEyd7QQqs7Ybqsr4+1eu4V/fcBYrAuUcG4jzsW2H+ODP\nD7CjPeK5M7DJuM3AajYDLzTnOibwXuCb6e3/D/gsMAh8HtjkulUeUVZW5tm5s7uDsrfHuWhZLV/Z\nsoH79oS48+kunu6I8nRHlNUNFfzpeUt41VkN+H3uN8y81FysWM1mYDW7Q65PnYCqhkWkFrgA+KKq\nfhM423WLPCQQCHh27gndQaVTJ4P2+0q47vxW7vizjbzz4qU0VpVytD/OZ393jH//9VFP7PJSc7Fi\nNZuB1ewOuTqB4yLyUuB64LeqmhKROmBBDc/39PR4du7JYwKno66ilBsubOO7f34uH3jFKkoEfn9k\ngFjS/T+nl5qLFavZDKxmd8jVCXwAuBf4KPDJ9LE3Ao+7bpGH5KslkD0mcDrKfCW8Zl0T65qrSCns\nCrofIdC+LZmB1WwGBWsJqOoDqrpMVdeo6pPpw/cA1+R6IRG5WkT2i8gBEfnQFOXvEJGQiDyd/rkp\n13PnSiKRmLnSHCnPevCX5+gExnlRm5Mo4vke9zOUeam5WLGazcBqdodcZwdtFJHW9HaNiHwc+AiQ\n0yiFiPiA24DXAxuBG0Rk4xRVf6CqF6Z/vpGTglkQi3mXBjI7ZpCvZHbxg1YEnEVkHREnh8FwIsUn\nHz7Mo8fC87bLS83FitVsBlazO+T6yvp9oD69/R/Ay4HLgK/m+PlLgQOqekhVE8BdwLWzMdQNinVe\n8fL0SuKOsOME7t3Vze8OD3DrQ4fmfe5i1ewlVrMZWM3ukOsU0TWqul+c190tOG/zMeBwjp9fDhzP\n2m9n6qmlfyoiLweeB96rqscnV+ju7mbr1q2UlpaSSqXYsmULN998M8FgkOrqanw+H5FIhJaWFvr6\n+lBVWlpa6OrqIhaL0dLSQjQapbW1lVAohIjQ2NhIKBSirq6OVCrF0NAQbW1tBINBysrKCAQC9PT0\nEAgESCQSxGKxTLnf76e2tpa+vt6MjUePHs2UV1RUUFlZSX9/P01NTQwODpJIJDLllZWV1KR9cXs4\nTkdHB0e7T+Y77unpOa2m8Zyj02mKxWI0NzfPSVNvby8NDQ3EYjHi8fisNPn9fsLhMM3NzYTDYZLJ\nZKZ8pvs0k6aZ7lM8Hs/YvVg0zXSfRkZGCAQCi0rTTPcpkUhQU1OzqDTNdJ8SiQTV1dVz0jQdksti\nJRHpAs7CefjfpqoXi0gp0KeqM0Y0EpHrgKtV9ab0/o3AJlW9JatOExBV1RER+T/An6vqqyafa/v2\n7bphw4YZbZ6Kzs5Oli5dOqfPzsT9+3r4zz84Puuhmy6a1WfHVLnmjmdIpJQfv/18vvbYCX6+v3dO\n55qMl5qLFavZDKzm3Nm5c+eTmzdvvniqsly7g+4EfgV8G7gjfezF5N4SOAGszNpfkT6WQVV7VXU8\nse83gJfkeO6cqa2tdfuUGa5Y7Yzav2xt/Qw1T6VEJBNcbnxcYJz5rij2UnOxYjWbgdXsDjl1B6nq\ne0XktUBSVR9JHx7DWUmcC08A60RkLc7D/3rgrdkVRGSpqnamd68B9uV47pzp7e2dsWk0Vxqqyrj/\nHRdQ5ptbUplldeUc6Y9zIjwyYb1ALDlGlX/qxWe5MJ3mE+ERwvFRNrYWLsy1V3h5n4sVq9kMvNCc\ncyhpVX1IRFal4wWdUNUds/jsqIjcAmwDfMDtqrpHRD4B7FDV+4B3i8g1wCjQB7xjNkJyoaGhwe1T\nTsA/y6mh2WS3BCYnsZ+PE5hO8zvv2QvA9284j6bqxbX83uv7XIxYzWbgheZcp4guFZHfAC8APwIO\niMhvRGRZrhdKrzVYr6pnquqn0sduTTsAVPXDqnquql6gqlep6nNz0HNainlK2fL0NNHvP9PF3u7h\nzPHwPJPYT6U5u4upe2jxzbUu5vvsFVazGRRyiuh/Ac8Ajaq6FGgAnga+4rpFHhKPxwttwrRctjLA\n2S1VjIyOMTJ6MufAu+97nnuf7eJYf3xOKSun0hzPOn/2tRYLxXyfvcJqNgMvNOfaHXQlsFRVkwCq\nOiQif8+kwd1ip5jnFTdVl/HFa8+mO5rg8eMRfro3xNF+54Z/7fEOvvZ4BzV+Hy9eXsvFK+p4yYpa\nWqr9M553Ks0DWa2LyDxbGsVIMd9nr7CazaBg+QSAfpzpodmcDQxMUbdoWQjxx5fU+HnjOc188rVn\nMHnhcTSR4reHB/js747xF9/fw599bxefevgw33myk13B6JRv9VNpjkwac3CLE+ERvry9nd7hJAOx\nZN4S50xmIdxnt7GazaCQ+QT+DfiliHwTOAqsBt4J/IPrFnlIRUVFoU3Imbbacj77xvUkUmPcsaOT\nl64OcOXaena0R9jRHuHZzigD8VF+c9jxw997KohPYG1jJetbqji7pZoNLVVUlJ+a1zj7wR8ZcS9y\n6Ue3HaAjkuAne0IAvHxtPZesrOO81prMmEc+WEj32S2sZjPwQnOuU0S/LiIHcaZ1ng90AG9V1Ydd\nt8hDKisrC23CrBifvvn5a07ODb5mYwvXbGxhT1eU9/7PCwC8el0jB3qGOTYQ50BvjAO9MR54zlls\nVu4T1rVE2NBSzfrmKs5eUjXRCbjYEuiITBxk/u3hAX6bdlLzXfQ2GxbafXYDq9kMvNA8mymiv8JZ\nMAY4QeFE5BOqeqvrVnlEf38/dXUzLnBeEJzbWsMnX3sGLdV+zmhyvhixZIoXemI8Hxpif2iY50LD\ndEUT7A4OsTt4MkF1aVY/U0dkhOFEal7TUHNhZHRs1tFV58pius+5YjWbgReac3YC03z2o8CCcQJN\nTU2FNsFVNq2aGFu8sszH+UtrOH/pycUkJ3oGOBET9oeGMz/ZLYHHjkfY8t1nOaOxknNbq9nYWsO5\nrdUsqZl50Hk27A8Ncf7S/KzwXGz3OResZjPwQvN8nADA3JbHFojBwUHjVhiWJGNcunIpl650HIaq\n0hVN0DWYYMeJQZ7uGORAz3CmG+mne53MRa01fl63vpFz22pY31xF9TxbCqGh5Ly15IqJ99lqNgMv\nNM/XCRRm+sccsUkonLwHbbXltNWWc8Ey5808PjrG/u4h9nQ5P3u7h+iKJvjOTmcmQonAuuYqLlxa\nwwXLarlwWe2ELqWc7MjjegR7n83AanaH0zoBETklimcW7vYX5AE7r3hqKkpLuGBZbcYpjKny8V8c\nZns6qY1ApivpB892U15awvrmKl68vJYXL69lfXMVMoNPGEnl733B3mczsJrdYaaWwDdnKD/mliH5\nIBgMsnr16kKbkVfmorlEhHdfsZIxVd60sZkXtdWwp2uIpzsG+ePRMO3hEXYFo+wKRvn2k51U+32c\nO0Mguny2BOx9NgOr2R1O6wRUda2rVyswdkpZ7jRVl/HJ152Z2b94RR0Xr6hj6yXLCMdH2d01xFMn\nBtl5YpATkREePx457flGUvlzAvY+m4HV7A7zHRNYUPj9C64Ha964rVlEqK8s48o19Vy5xsmd0DWY\n4KmOQfZ0Rdn2fN+Un8tnS8DeZzOwmt0hPxO3i4RweP6J2xca+dDcWuvn6rObeP/LV7Nt64V87/pz\n+cZ15/CitpOzGPI5JmDvsxlYze5glBNobm4utAl5J9+aRYQlNX5W1VfwmTeu491XOAnl8hmt1N5n\nM7Ca3cEoJ2DfHPKPP51pLZHHMYFCay4EVrMZFKwlICKfE5ELXb96nkkm87dgqVgotObxUBEjo/nr\nDiq05kJgNZuBF5pzbQn4gG0isltEPigiK1y3JA/YecX5x+9zvmLJPLYECq25EFjNZlCwfAKq+m5g\nGfAh4EJgn4j8UkTeLiILZt22jT+ef8pLne6gfE4RLbTmQmA1m4EXmnMeE1DVlKrer6o3AJcBLcAd\nQFBEviEiy123zmWqq0+/oGkxUmjN5emWQCKP3UGF1lwIrGYz8EJzzk5AROpEZKuIPAL8FngMeBlw\nDhAFfu66dS7j83kbLrkYKbRm//iYQB5bAoXWXAisZjPwQnOuA8P34uQT3oKTXH6Zqr5LVf+gqseB\n9wFFv7o4Ejn9qtbFSKE1j7cE8jlFtNCaC4HVbAZeaM61JfAosE5V/0RVf6CqI9mFqjoGtJ7uBCJy\ntYjsF5EDIvKh09T7UxFREbk4R9typqWlxe1TFj2F1uwvzf8U0UJrLgRWsxl4oTnXgeH/AEIicoWI\nvCX92zepzvB0n0/XvQ14PU7C+htEZHLiekSkFngPTleT6/T1TR3SYDFTaM0nWwL5GxMotOZCYDWb\ngReac+0OehHwAnAP8IH07xdE5IIcr3MpcEBVD6lqArgLuHaKep8E/hWI53jeWaG6oNIfuEKhNVeU\nOV+xWDKVN1sKrbkQWM1m4IXmXAPIfQvnTf6zqqoiIsB7gduBl+Tw+eXA8az9dmBTdgUReTGwUlV/\nJiIfmO5E3d3dbN26ldLSUlKpFFu2bOHmm28mGAxSXV2Nz+cjEonQ0tJCX18fqkpLSwtdXV2Ul5fT\n29tLNBqltbWVUCiEiNDY2EgoFKKuro5UKsXQ0BBtbW0Eg0HKysoIBAL09PQQCARIJBLEYrFMud/v\np7a2lt7eXhoaGojFYsTj8Ux5RUUFlZWV9Pf309TUxODgIIlEIlNeWVmJ3+8nHA7T3NxMOBwmmUxm\nymfSNJ5laDpNFRUV9PT0FFRTuU8YSSkD0RiRvtC8Nc10nyorK+nu7l5Q92m+373q6mq6uroWlaaZ\n7lNNTQ2dnZ2LStNM96m2tpaOjo45aZoOycWziEgEaFDVVNYxH9CvqjNmPRaR64CrVfWm9P6NwCZV\nvSW9X4KTxP4dqnpERH4N/D9V3TH5XNu3b9cNGzbMaPNUHD161Lj448Wg+R1376EjkuD2t5zDikCF\n59crBs35xmo2g7lq3rlz55ObN2+ecpw114HhB4BrJh17E/CzHD9/AliZtb8ifWycWuA84NcicgRn\nHcJ9bg8Om5aPFIpDc0NlGQB9w6Mz1HSHYtCcb6xmM/BCc67dQT7gLhF5EqdbZyVON9BPReQ745VU\n9e3TfP4JYJ2IrMV5+F8PvDXrc2EgEx7vdC0By8Jj3An0x8yL9WKxFDu5OoHd6Z9x9gLbcr2Iqo6K\nyC3pz/iA21V1j4h8Atihqvfleq75EI1GaWpqyselioZi0NxY5XzNjvZ7Mt5/CsWgOd9YzWbgheac\nnICqfny+F1LVB3C6lbKP3TpN3VfO93pT0dp62qUMi5Ji0DzeEvjeU0Gaq8t4wwZv48AXg+Z8YzWb\ngReaZxM24pUicruIbEv/vsp1azwmFAoV2oS8UwyaX7amnmV15QD891PeB/0qBs35xmo2Ay8057pO\n4CbgbiAI/AjoBL4vIn/lukUe4sxsNYti0LyqoYLb33IO1X4foaEk3dGEp9crBs35xmo2Ay805zom\n8PfAa1T1mSxjfgD8EPi661Z5RGNjY6FNyDvForlEhHNbq3n8eIS9XUMsqfEuSXixaM4nVrMZeKE5\n1+6gJpzB4Gz2AwvqLtjmY2FZXe+sEegcHJmh5vwoJs35wmo2g4J1BwG/Bz4rIlUAIlIN/DvwR9ct\n8pC6uhnXtS06iknz+Nt/KOrtVNFi0pwvrGYz8EJzrk7gr4HzgbCIdAEDwAXA/3HdIg9JpVIzV1pk\nFJPmcSfQPeTtmEAxac4XVrMZeKF5RieQjhNUCWzGyRnwJmCtqr5CVTtct8hDhoaGCm1C3ikmzS3V\nzlRRrweGi0lzvrCazcALzTMODKcDxu0CalW1HSf424LEJqYuLJmWgMdOoJg05wur2QwKlmgeeApY\n7/rV84xNTF1Yast9lJUIw8kxTzONFZPmfGE1m4EXmnOdIvpr4EERuQMndlAm9Kiq3u66VR5RVlZW\naBPyTjFpFhECFaX0DCcJx0c9myZaTJrzhdVsBl5oztUJXAEcBl4x6bji5BRYEAQCgUKbkHeKTXNd\nHpxAsWnOB1azGXihOdfYQQsuRMRU9PT0UF1dXWgz8kqxaQ5UOF+5cNy7sNLFpjkfWM1m4IXmXMNG\nPDXN8QUV6tm+ORSeQIWTmtpLJ1BsmvOB1WwGXmjOdWD4rMkH0lNHz3DXHG9JJLydlVKMFJvmQIXT\npxnx0AkUm+Z8YDWbgReaT9sdlJUwxp+dPCbNGmCP6xZ5SCwWK7QJeafYNI+3BAY8dAL51LynK8q3\nnujk3VesZFWD96kzp6PY7nM+sJrdYaYxgYPTbCvwB+Ae1y3yEDuvuPDUp3ML9Ax5Fzoin5rf+z8v\nAPDpR47wlS1zy33tBsV2n/OB1ewOp3UC48lkRORRVc05k1ixEgwGjUtMXWya17dUAfCLF/p41ZkN\nvHh5revhcQuh2csxjlwotvucD6xmd8h1dtA2ETkbJ15QzaSyBTNF1O/3LnxxsVJsms9qqqTG7yOa\nSPHhBw9y+eoAbzi7idryUs5sqqS8NOc8R9NyOs2/PzzAT/eG+Mir1tBQWcaYKiUuOCE9uXSmIBTb\nfc4HVrM75OQEROQjwK3AM8BwVtGCWidQW1tbaBPyTrFpLhHhHRcv5cH9vRzqi7H9aJjtR8MANFaW\ncstLV3Ll2vp5XeN0mj/x8GEA7nwqyKr6Cr75RAf/8SfriCZSpMbUk5ZJPii2+5wPrGZ3yHWx2N8B\nl6rqs65bkEd6e3upqamZueIiohg1X7OxhWs2tnCsP87DB/rYcSJCz1CSvtho5iH9srX1XHVGA5eu\nrMM/y9ZBLpoHR1J88Y9OGKyP//IwXel4Rh++ag1XndmQqRdLpijzlVBacnrHIBTWcRTjffYaq9kd\ncnUCMeA5V69cABoaGmautMgoZs2rGip45yXLeOclyxgdU77yaDv37e0B4HeHB/jd4QGqykoch3Bm\nAxcsrcU3w8MYZq85O6Dd8YF4Zjs+Osabv/0sitNKuerMBt7+kqVUlvlmdf58UMz32SusZnfI1Qn8\nA/BFEfknoCu7QFW9iwTmMrFYzLhEFAtFc2mJ8LeXr8AnwkhqjOV15TxysJ8DvTG2Pd/Htuf7aKgs\n5fylNWw5bwnrm6umdQi5aM7u8cnuze+PnZy1dCIcz5T1xUb54e4QvhLhpkuXz1GldyyU++wmVrM7\n5OoE7kjpqCmLAAAgAElEQVT/vinrmOD8/+T0WiQiVwNfSNf/hqr+y6TyvwZuBlJAFHiXqk5OaTkv\n4vH4zJUWGQtJc4kIf3P5isz+W85v5dhAnF8f7OdXB/vpiIzwm0MD/ObQAPUVpbzijHpeuqaeF7XV\nTOiuyUXzdO2JvtjJWT6pKV5vjg0U599zId1nt7Ca3SFXJ7B2PhcRER9wG/AanHwET4jIfZMe8neq\n6lfS9a8BPgtcPZ/rTsbOK154rKqv4O0vWcqNL27joRf6+MxvjwHOYrOf7u3hp3t7qPH7uGRlHZev\nCnDJyrppNSeywlcnU1PP5ukfPtkSGEqemsWpWAeNF/p9ngtWszvkOkX06DyvcylwQFUPAYjIXcC1\nZCWvV9VIVv1qcH/OnZ1XvHAREV63volV9RU0VJYSiaf43ZEBth8Nc2wgziMH+3nkYD8+gfUNpbzq\n7DYuWxWgtfbklLrBxMmHemRk6jR9/VktgaHEFE5gnjoGR0b52mMneP3ZzWxsdS8Q2GK5z7PBanaH\nmcJGPKWqF2Xtf0tV35m1362qS3K4znKcPATjtAObprjezcD7AD/wqqlO1N3dzdatWyktLSWVSrFl\nyxZuvvlmgsEg1dXV+Hw+IpEILS0t9PX1oaq0tLTQ1dXF6Ogovb29RKNRWltbCYVCiAiNjY2EQiHq\n6upIpVIMDQ3R1tZGMBikrKyMQCBAT08PgUCARCJBLBbLlPv9fmpra+nt7aWhoYFYLEY8Hs+UV1RU\nUFlZSX9/P01NTQwODpJIJDLllZWV+P1+wuEwzc3NhMNhkslkpnwmTeMzBabTlEql6OnpWTSaqmI9\nlJfV0SgpXr0kydvOP4NnDp5gd1+KXX0p9nbH2Nc3yr7t7dy2vZ21DeWcWy9cvKya+rqTD92uyNTL\n73uHEwwODhKNRjnWMXhKeSw2zNDQUEbTODo2Rn9//4yavraji0eOj7Dt+T5uv7rFte+eqtLV1VU0\n9ykf/08AnZ2di0rTTPdJROjo6JiTpukQ1elfuEVkUFVrs/b7VLVxuvLTnOc64GpVvSm9fyOwSVVv\nmab+W4HXqepfTi7bvn27btgwt+X5kUjEuIEk0zRH4qP85vkung4l2NEeIZY82QVU7fdl3u7LfDJt\nl9D3bziPpuoyfrirm68+dmJC2ZVrAtz66pNxE1/7DSfAbnNVGXe+9bwZ7fvYtoM8ftxp9D5000Uz\n1AZVJZYco8p/+qE30+4zWM2zYefOnU9u3rz54qnKZpqAPfm/ZHJrONcumxPAyqz9Felj03EX8OYc\nz50z428PJmGa5rqKUs4PjPIPm9dyz9texKevPpM3ntNMc1XZhO6d6RwAwJF+p5UwVXfQ6T6XC6d5\n55qST/3qCG/+zrN0RkZOW8+0+wxWs1vMdo3+XP8DngDWichaEfED1wP3ZVcQkXVZu38CvDDHa01L\nU1OT26csekzW7PeVcPGKOt59xUr++4Zzue3NZ/O3l69gzTTRPlenjx/uc5xAdAonkN2ySI2d/HcY\ny/HpPtvwEr89PADAIwdP/89v8n02CS80zzQwXC4in8jar5y0n1MgC1UdFZFbgG04U0RvV9U96XPt\nUNX7gFtE5NVAEugHTukKmi+Dg4PGrTC0mh1EhHXNVaxrruLN57YwEEvyXGiYR4+F+eORMCUlcNUZ\nDdzxZCdff7yDRw7180LPqeMG2TOGRrJmG43Ms4UwX/Z39PPlZ47xzkuWceUaJ+xGYnSMUp+4Ehup\nGLHfbXeYyQncycRunLsm7X8/1wup6gPAA5OO3Zq1/Z5czzVXbBIKM8hFc31lGZetCnDZqgDvvkIR\nnGmne7qGeLpzcEoHAHCkL0YsmaKyzEciayFB9vRTL5jpOX7HrgGOh0f5xC8P89BNFxFLprjuu7vY\nsKSaz7zRaWQPJ1JEEynPcjvnG/vddoeZQkm/83TlCw07r9gMZqt5/E25obKMT119JvHRMXYHo+zq\njPJEe4QDvScdQkphy3ee5eyWas5sqswcT45pThFJs3uNZqp/ukkbk0mVlAInp7fuDw2THFN2BaOZ\nY395917C8VG+f8N5HOgd5tnOKFsvXbZgWwr2u+0O84/bu4AIBoOFNiHvWM2zp6LUGUt45yXL+PL/\n2sBDN13Ef16zHl/6WZlS2Ns9xP/s65nwuTuf7uJIf4z4aVoF2WMK2dtTkT0IPdOAdDIxMUnPVLXH\ncx4c6B3mHx46xD27unn0WPi05y1m7HfbHXJdMbwoqKysnLnSIsNqdocNS6r57vXn4hOhzCfs7hri\n2c4o+7qH2NM1BMB3nuzkO092UloirG+u4kVLa7h8VYBzllRlVhpnzzgaSqSoPs3Uz+y6w1OsXs6m\npKQEJ+KKQ/agdTI1Rpnv5Pte9ot/ODa7ZDjDidSM01Xzhf1uu4NRTsAmoTADrzQ3V5887/h4AjjR\nRv9wZIAv/bGdoXRegr3dQ+ztHuIHzzjxFs9qquTc1hp6s8JSzPRgzy4fTpy+1SCTgullfzaWHJvQ\n5TMhJtIsuoJ2nojwoZ8f5G0XtfH2lywFnC6rzsEES2v9eQ+pYb/b7mCUEwiHw9TXzy9hyULDavae\nitISNp/VyGWrAvh9wsjoGHu7h/jd4QG2Pd8HwIHe2ISxBYBvPt7BpSvrOK+thtUNFaf0zQ9lPfhn\nchip0Ynl2Z8dSqYmPOsnnGsW4w7ffrITgO89Fcw4gbue6eJbOzrZesky/vyC1pzP5Qb2u+0OuWYW\nuwo4oqqHRWQp8C/AGPBhVV0wHXPNzc2FNiHvWM35Y7xrp8xXwqUrA1y6MsC1G1uoLS+lKzrC7qDT\ndfREu7Ni+LHjER5Lrx6uLfexcUk157XVcF5rNetaqiZMR51q4Vo2vtJSsruDhhMTt0uy1nlmx0dK\nzHNq67d2dKZ/d0zrBH59sJ+R1BivW+/uHHf73XaHXFsCXwZel97+TPp3DPgacI3bRnlFOBymutq9\noF0LAau5sJzVXAVAa62f85c6EVZUlcN9cfZ0RdndNcSuYJSeoeQEp1DmE1qyup9mcgKJrJbA6JhO\n7EpKjiFZDqJnKJFVdvrzZjPb1c7OZ5RPP3IEgJevrZ82Ic/I6Nis80sX033OF15oztUJLFfVYyJS\niuMMVgMJoMNVazwmO+CXKVjNxYeIcEZTJWc0VfKmjS2Ak91sVzDKnuAQu7uiHOmP05EVKuK50DA3\n3rWH5YFyzmys5JVnNnBGY2UmsU72TKPhRGrioHIiNSHeS89QckKZl2TPaoonx6Z0Ao8fD/OxbYf4\n28tX8Jp1jdz9TBeb1zWyqn7qld2Zcxf5ffYCLzTn6gQiItIKnAfsVdVoOvxDmesWeYidV2wGC1Hz\nkho/m89qZPNZTnzGSHyUvd1DHO2P89ixMAf7YnRFE3RFE+w8Mcg9u7qpKC3h7JYqNi6pZmj05MN2\nKJmaMJA8PGlMYIITmGGa6mz42mMnaKoq409fdDKw8MQWSYqGKR4Zt6VzPX95ezsnwiP8dG+IH+8J\ncd87LsjU+dlzPTx6NMxbzm/l/KXOitmFeJ/nS8HyCQBfxIn/48dJOg9wBQss77CNP24Gi0FzXUVp\nZgbSn1/QSmpMOdLvDC4/0xllb1eUjkiCZzqjPNMZnfDZL/+xneDgyS6foUkzi3qG59YddDrGFO7d\n1Q0wyQmMTbk9HQd7hwFOWWvxhd87kejD8VH+89qzgYn3eXRMefhAHy9ZXjthFtdiI+/5BMZR1X8V\nkR8DKVU9mD58gonpJose0/oPwWpeLPhKhDObqjizqSozwNo/nGRfaIi9XUPsOhFmX6/TfTQ+rjDO\nj3d301h18g08FM3uDsq9JZDrkEB8dIyKdP9+dndTzAWHE8tyDtn3+ce7u/n64x20VJfx3zecGtL7\nwf29lJcKV53ZeErZQsKL73bOU0RV9fnx7fRsoTFV/Y3rFnmIz1cci1zyidW8eGmoKuOlq+t56ep6\n+vurqK+v5+hAnH1dzhqFPx4NMziS4nh4hOPhk+ML2Q/zvlhyxkVr42QvQBsd0wl5nbOJxEepSMcn\nyn77n9wimcqeqRzNhGitWdvZ93m8NRQaOrXPPD46xmd/56QlzXYCu4NRvvbYCf7uylWc0bQwFp55\n8d3OdYrob4CPqOofROSDONm/RkXkNlX9tOtWeUQkEqGhoaHQZuQVq9kMxjWvaahkTUMlr9/QzPtx\nHsj7uofY1z3E8z3D7GifmC1tf2iYLd95lpX1FWxoqWLDkmrWNVeytrESv2/ibJ3Jg891FVM/PsLx\n0UyQuomL1mbfEnj8eJj79p4Mz5HdEsj1PseyWiOJ1FhG1/vvfwEF/vlXh7n9LRtnbVsh8OK7nWtL\n4Dzg0fT2XwFXAYPAH4AF4wRaWloKbULesZrNYDrNdRWlbFoVYFN6dfOYKh2RETojCZ7uGOTZYJRD\nvTGODcQ5NhDnoRecxW1lJc4MprNbqljfXMWGlupTwliczglk6iUmTlWdLR/bdmjCfrYjyvU+D0+K\n1zTuBMbbFJH47EJnjLO3a4gf7e7mslUBXr2ukTFVDvTGWNtQMSFMh5t48d3O1QmUACoiZ+KkpNwL\nICIL6nWrr6+PqqqqQpuRV6xmM8hVc4kIKwIVrAhUcMlKJ01hIjXGwd4Yv3yhj2MDcfpjoxwfiLM/\nNMz+0PCU5/nj0TBXrJ565eoEJ5DMfdVzLsSSKVQVEclZc2xSayQwjfOaLd97qpMd7YP89vAAr17X\nyP37evjSH9vZfFYDH3zlGleuMRkvvtu5/jV+D3wJWAr8GCDtEHpO96FiYzaheRcLVrMZzEez31fC\nOUuqOWfJyUHHoUSK53uGeT40zP7QEPtDwxP627/y6Am+8ujUGWIjE5zAxBhGM5EtYypNY+ok8Kko\nlQnlp5M/uSUwmbnGPIpnnUtVM1FlHz7QPycnMDgyyqPHwrxsbUNmYH0yXny3c3UC7wDeD4SAf08f\n2wB8wXWLPMR2E5iB1Tx/qv0+LlpWy0XLajPH+tODyI8c7OfZKaamjvNfj57gifYIZzZWcqQ/njk+\nXUtgNGuwN7vOdCG5Y4kUFaUlOWuOTVqr4BbZITcSKZ178t00//zwEZ7qGGR/aJhbXnoyd9fhvhif\n+90xbrp0OesL1R2kqr3ARyYd+5nr1nhMV1fXgp8/PlusZjPIh+aGyjIaKsu48cVO8LjuaIJkSomP\npvj1oQGe7hjk2ECcWHKMHe2DpwxC//FImBJIT3WtZFV9Bb4SmfB2nh1ltTs6dRat4eQYDUzUnJ27\neXKynplaAnNl8kK4+fJUh/P3+uOR8AQn8M8PH+Z4eIT/97MX+PprGguzTkBEyoCPATcCy3DCRXwX\n+JSqLpgcb6blIwWr2RQKoTk7TeWZTU4/taoSHExwsC/God4YB/ti7DwxyMjoGF3RBD/cHcp8xu8T\n1jZWThg8Hhw5ud05OPWjZfzNPltzfNKDPnvKa2yWXVK5kn3NuIvnnUwk62/ixX3OtTvo34BLgb8G\njuLEDvoHoA54r+tWWSyWBYmIsLSunKV15ZmE9+AsbNvfM8yB3hiHep3fwcHEtAPP4LwBT0Vsim6i\niW/7E9c9TC5zi9mOdxQruTqBtwAXpLuFAPaLyE7gGRaQE4hGozQ1uRvOttixms2g2DU3VJVNSMQD\nEB0Z5WBvjEN9jkN4LjTEkf545oE6XZjrB/f30jecpCw2wCX1DZT5Sib1+098IJ+uDGAuw8KqOmHM\nwk3ncjq8uM+5OoHp/k4LKkN1a2t+k14UA1azGSxEzTXlpVywrJYLsgafVZWRlLLjeCTtHEZ49FiE\n0hKhptxHe3iEX7zQxy/S6xlKHn2G5XXlE2YuPXo0TDI1xhmNlYjIpLzOzsM6ezA6NYcZNyMpJesU\nU7ZOvMCL+5yrE7gH+B8R+ThwDKc76GPA3bleSESuxplN5AO+oar/Mqn8fTixiEZxZiH9b1U9muv5\ncyEUCrFy5cqZKy4irGYzWCyaRYSKUuHKtfVcuXbiOoRYMsUfj4Y52h/naH+cgz2DhIbHJoTEAPjG\nEx184wmoKiuhrbacwZGTU1ZPhEfoGUpMWA09l/782KQQ3PnqDvLiPufqBP4e56F/G87A8AngLuCf\nc/mwiPjSn30N0A48ISL3jS86S/MUcLGqDovI3+CMQ/x5jvblRL5zoBYDVrMZmKC5ssyXCbUN0N7e\nTkvbMtrDjlPoj42yOxglNJSkK5ogHB/lUN/ElJ4PvdDHQy/0UeY7+fdKjim/PtjPyvpylgcqpp2j\nn83kN/9YMjWhRZEa00yuBzfx4j7P6ATSD/C3AZ9W1VvneJ1LgQOqeih9zruAa4GME1DVR7LqP5q+\npqs0Ni7sCIJzwWo2A1M1l5eWZKKrwskw1qrK4EiKzsEROiIJBmJJDvU54TE6IokJq5qBTPYzgCU1\nZSyvq2BlfXl6dXU5KwLlLKnxZ6aeTh4DiCXHJs4WGh07JSjfeEa5NY2n5pOejWa3mdEJqGpKRD6r\nqrfP4zrLgeNZ++3AptPU3wr8fKqC7u5utm7dSmlpKalUii1btnDzzTcTDAaprq7G5/MRiURoaWmh\nr68PVaWlpYWuri5isRgtLS1Eo1FaW1sJhUKICI2NjYRCIerq6kilUgwNDdHW1kYwGKSsrIxAIEBP\nTw+BQIBEIkEsFsuU+/1+amtr6e3tpaGhgVgsRjwez5RXVFRQWVlJf38/TU1NDA4OkkgkMuWVlZX4\n/X7C4TDNzc2Ew2GSyWSmfCZN41PGptMUi8Vobm5eVJpmuk/xeDxj92LRNNN9GhkZIRAILCpNM92n\nRCJBTU3NaTXVJsOcUZqkbV0bwWCM6o1N+Hw+gr0DaGWAEz0D7Aol6Bv1caxvmFBsjO5oku5oMjNv\nf5yyElhaU0ZbTSkyNtEJhIfiDCVOOpYXDh/jnDXLJmi6+6kT3LV/mOvPqWPzitIJmioqTmZRS42l\nOHr0aKZ8bOykczlx4gTV1dVzuk/TIbksQxaR7wJ3q+r/zFh56s9fB1ytqjel928ENqnqLVPUfRtw\nC/AKVR2ZXL59+3bdsGHDXMygv7/fuOiSVrMZWM3ukBpTgoNO6O328Ajt4TjtAyO0R+L0DU8faK5E\nmDBQ/PYXt3F2SzXL6spprfVTWiK89htPZcofuumiCZ9PpMZ447eeASBQUco9b3tRpuwt39uVabnc\ntWX1nFoDO3fufHLz5s0XT1WW65hABXCviGzHeaPPyFXVt+fw+RNA9mjGivSxCYjIq4GPMo0DmC+p\nVH6mcRUTVrMZWM3u4CsRlgcqWB44Nb/xUCLFiXHHEB4hNJTg7JZqfry7m/ZJg9Pf2RnMbJcItNZM\nzHb2m0P9rAicHIOYuPBsoq5k6mTZiAdTUXN1ArvTP3PlCWCdiKzFefhfD7w1u4KIXAR8FafF0D2P\na03L0NAQzc3NXpy6aLGazcBq9p5qv4/1LVWsb5kYxfON5zSTGB0jOJggpcoTxyN0DI7QEXF+QtHk\nKaufP/WrI5ntpqoyGipPPopHUsrjx8O01ZSzpNY/YT1CXyTK0lZ34wflGjvo4/O5iKqOisgtwDac\nKaK3q+oeEfkEsENV78MJTFcD3JMeAT+mqtfM57qTsYmpzcBqNoNi0uwvLWFVg9N6WNs4MUtZYnSM\nYDRBcHCERErZ1zWU6WrqHEzQO5ycEDMJTs2jME5do/tO77ROQESuAK5R1Q9OUfYvwE9U9dFTP3kq\nqvoA8MCkY7dmbb86J4vnwWJIQD5brGYzsJqLF39pCavqK1hV7ziJ7HAaqTGlZyhJV3SE0FCSWHKM\nF3qGCQ4m6Iom6I4mJixsOx7sZmVT7SnXmA8ztQQ+Anx5mrLf4PTfv8lVizykrKxs5kqLDKvZDKzm\nhYmvRGit9dNa65+yfEyV/uFRjoedkNx1o2HXbZjJCVwIPDhN2S+Ab7prjrcEAoGZKy0yrGYzsJoX\nJyUiNFWX0VTtOLyhIffTVs50xjpgahcFZYC77RKP6elZUInQXMFqNgOr2Qy80DyTE3gOeO00Za9N\nly8YTHhzmIzVbAZWsxl4oXmm7qDPAV9Nh474iaqOiUgJ8GacWEDvc90iD0kkFkz+G9ewms3AajYD\nLzSf1gmo6p0i0gZ8GygXkR6gGRgB/lFVv++6RR4Si8VmrrTIsJrNwGo2Ay805xI76LMi8g3gcqAJ\n6AW2q2rEdWs8ppjmFecLq9kMrGYz8EJzTkPNqhpR1W2qemf694JzAODMKzYNq9kMrGYz8EKz+/ON\nihi/f7qJTosXq9kMrGYz8EKzUU6gtnZBzWh1BavZDKxmM/BCs1FOoLe3t9Am5B2r2QysZjPwQrNR\nTsC0eOtgNZuC1WwGXmg2ygnYKWVmYDWbgdXsDkY5gXg8XmgT8o7VbAZWsxl4odkoJ2DnFZuB1WwG\nVrM7GOUE7LxiM7CazcBqdgejnEBFxal5Qxc7VrMZWM1m4IVmo5xAZWXlzJUWGVazGVjNZuCFZqOc\nQH9/f6FNyDtWsxlYzWbghWajnEBTU1OhTcg7VrMZWM1m4IVmo5zA4OBgoU3IO1azGVjNZuCFZqOc\ngE1CYQZWsxlYze6QNycgIleLyH4ROSAiH5qi/OUislNERkXkOi9ssPOKzcBqNgOr2R3y4gTS6Slv\nA14PbARuEJGNk6odA94B3OmVHXZesRlYzWZgNbvDjJnFXOJS4ICqHgIQkbuAa4G94xVU9Ui6bMwr\nI+yUMjOwms3AanaHfDmB5cDxrP12YNNcTtTd3c3WrVspLS0llUqxZcsWbr75ZoLBINXV1fh8PiKR\nCC0tLfT19aGqtLS00NXVhYjQ29tLNBqltbWVUCiEiNDY2EgoFKKuro5UKsXQ0BBtbW0Eg0HKysoI\nBAL09PQQCARIJBLEYrFMud/vp7a2lt7eXhoaGojFYsTj8Ux5RUUFlZWV9Pf309TUxODgIIlEIlNe\nWVmJ3+8nHA7T3NxMOBwmmUxmymfSVFNTAzCtppKSEnp6ehaVppnuk8/no7u7e1Fpmuk+lZWV0dXV\ntag0zXSf/H4/nZ2di0rTTPepvLycjo6OOWmaDlHVuTyLZ0W6j/9qVb0pvX8jsElVb5mi7h3A/ap6\n71Tn2r59u27YsGFOdhw9epTVq1fP6bMLFavZDKxmM5ir5p07dz65efPmi6cqy9fA8AlgZdb+ivSx\nvNLc3JzvSxYcq9kMrGYz8EJzvpzAE8A6EVkrIn7geuC+PF07QzgczvclC47VbAZWsxl4oTkvTkBV\nR4FbgG3APuBuVd0jIp8QkWsAROQSEWkH3gJ8VUT2uG1HMpl0+5RFj9VsBlazGXihOV8Dw6jqA8AD\nk47dmrX9BE43kWfYecVmYDWbgdXsDkatGLbzis3AajYDq9kdjHIC1dXVhTYh71jNZmA1m4EXmo1y\nAj6fr9Am5B2r2QysZjPwQrNRTiASiRTahLxjNZuB1WwGXmg2ygm0tLQU2oS8YzWbgdVsBl5oNsoJ\n9PX1FdqEvGM1m4HVbAZeaDbKCeQjREaxYTWbgdVsBl5oNsoJ2OajGVjNZmA1u4NRTqCrq6vQJuQd\nq9kMrGYz8EKzUU5gppCqixGr2QysZjPwQrNRTsBisVgsEzHKCUSj0UKbkHesZjOwms3AC81GOYHW\n1tZCm5B3rGYzsJrNwAvNRjmBUChUaBPyjtVsBlazGXih2SgnICKFNiHvWM1mYDWbgReajXICjY2N\nhTYh71jNZmA1m4EXmo1yArb5aAZWsxlYze5glBOoq6srtAl5x2o2A6vZDLzQbJQTSKVShTYh71jN\nZmA1m4EXmo1yAkNDQ4U2Ie9YzWZgNZuBF5qNcgI2MbUZWM1mYDW7g1FOwCamNgOr2QysZnfImxMQ\nkatFZL+IHBCRD01RXi4iP0iXPyYia9y24Sc/+Ynbpyx6rGYzsJrNwAvNeXECIuIDbgNeD2wEbhCR\njZOqbQX6VfUs4HPAv7ptx49+9CO3T1n0WM1mYDWbgRea89USuBQ4oKqHVDUB3AVcO6nOtcC309v3\nApvF5eVxo6Ojbp5uQWA1m4HVbAZeaJZ8pGgTkeuAq1X1pvT+jcAmVb0lq87udJ329P7BdJ2e7HM9\n8MADg52dnRnnVVdXF2psbJxQZzr6+vqac627WLCazcBqNoN5aF69efPmKdOSlc7Tprzzhje8obbQ\nNlgsFstiIV/dQSeAlVn7K9LHpqwjIqVAAOjNi3UWi8ViKPlyAk8A60RkrYj4geuB+ybVuQ/4y/T2\ndcCvNB99VRaLxWIweekOUtVREbkF2Ab4gNtVdY+IfALYoar3Ad8EvisiB4A+HEdhsVgsFi9R1UX/\nA1wN7AcOAB8qtD0u6rod6AZ2Zx1rBH4BvJD+3ZA+LsB/pv8GzwIvLrT9c9S8EngE2AvsAd6z2HUD\nFcDjwDNpzR9PH18LPJbW9gPAnz5ent4/kC5fU2gN89DuA54C7jdBM3AE2AU8jfOC7Pl3e9GvGM5x\njcJC5Q4cB5fNh4CHVXUd8HB6Hxz969I/7wL+K082us0o8H5V3QhcBtycvp+LWfcI8CpVvQC4ELha\nRC7DWUvzOXXW1vTjrLWBPKy5ySPvAfZl7Zug+SpVvVBVL07ve/vdLrTny4NnvRzYlrX/YeDDhbbL\nRX1rmNgS2A8sTW8vBfant78K3DBVvYX8A/wUeI0puoEqYCewCegBStPHM99znG7Xy9Pbpel6Umjb\n56B1Rfqh9yrgfpw338Wu+QjQPOmYp9/tRd8SAJYDx7P229PHFiutqtqZ3g4C45mpF93fIR1a5CKc\n5v+i1i0iPhF5Gqf77xfAQWBAVcdXD2XrymhOl4eBpvxa7AqfB/4eGEvvN7H4NSvwkIg8KSLvSh/z\n9Lu94NYJWHJHVVVEFuUMKxGpAX4I/J2qRrIXly9G3aqaAi4UkXrgx8CGApvkKSLyRqBbVZ8UkVcW\n2p48cqWqnhCRJcAvROS57EIvvtsmtARyWaOwmOgSkaUA6d/d6eOL5u8gImU4DuC/VXU8mMqi1w2g\nqjIjUyAAAAPiSURBVAM4A+OXA/XpNTUwUddiWHNzBXCNiBzBCTPzKuALLG7NqOqJ9O9uHGd/KR5/\nt01wArmsUVhMZK+3+EucPvPx428Xh8uAcFYTc8GQjif1TWCfqn42q2jR6haRlnQLABGpxBkD2Yfj\nDK5LV5useUGvuVHVD6vqClVdg/M/+ytV/QsWsWYRqRaR2vFt4LXAbrz+bhd6ICRPgy1vAJ7H6Uf9\naKHtcVHX94FOIInTH7gVpx/0YZzpZL8EGtN1BWeW1EGcKWgXF9r+OWq+Eqff9FmcaXRPp+/votUN\nnI8zTfLZ9EPh1vTxM3Cmjh4A7gHK08cr0vsH0uVnFFrDPPW/kpNTRBet5rS2Zzg5Ffij6eOefrfz\nEkDOYrFYLMWJCd1BFovFYpkG6wQsFovFYKwTsFgsFoOxTsBisVgMxjoBi8ViMRjrBCyWPCEiURE5\no9B2WCzZWCdgMQYROSIirxaRd4jI7z2+1q9F5KbsY6pao6qHvLyuxTJbrBOwWGZJVtgCi2XBY52A\nxTTOAb4CXJ7unhkAEJFyEfkPETkmIl0i8pV0iAZE5JUi0i4iHxSRIPAtEWkQkftFJCQi/entFen6\nnwJeBnwpfY0vpY+riJyV3g6IyHfSnz8qIh8TkZJ02TtE5Pdpe/pF5LCIvD7vfymLEVgnYDGNfcBf\nA9vT3TP16eP/AqzHSdpyFk5I3luzPteGk+FpNU4CjxLgW+n9VUAM+BKAqn4U+B1wS/oat0xhxxdx\ngpydAbwCeDvwzqzyTTjx4ZuBfwO+KdmhUi0Wl7BOwGI86Yfru4D3qmqfqg4Cn2Zinusx4B9VdURV\nY6raq6o/VNXhdP1P4TzMc7meL33uD6vqoKoeAT4D3JhV7aiqfl2dENLfxkkm0nrKySyWeWL7Ni0W\naMHJ2PVk1su24OS3HSekqvFMoUgVThrDq4GG9OFaEfGlH9ynoxkoA45mHTvKxIQgwfENVR1O21WT\nqyCLJVdsS8BiIpOjJvbgdOecq6r16Z+Aqtac5jPvB84GNqlqHfDy9HGZpv7k6yVxupLGWcUCznNg\nWbhYJ2AxkS5gRTq/BKo6Bnwd+Fw6oxMislxEXneac9TiOI4BEWkE/nGKa0y5JiDdUrgb+JSI1IrI\nauB9wPfmoclimRPWCVhM5Fc48dqDItKTPvZBnFj0j4pIBCdu+9mnOcfngUqct/pHgQcnlX8BuC49\nu+c/p/j8/wWGgEPA74E7gdvnJsdimTs2n4DFYrEYjG0JWCwWi8FYJ2CxWCwGY52AxWKxGIx1AhaL\nxWIw1glYLBaLwVgnYLFYLAZjnYDFYrEYjHUCFovFYjD/P7iUjFo/Cok1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7a886902e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses)\n",
    "plt.ylabel('Cross Entropy Loss')\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylim(ymin=0.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}