02_Quantization_Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quantized Neural Network \n",
    "## 02- Quantization Basics\n",
    "\n",
    "by [Soon Yau Cheong](http://www.linkedin.com/in/soonyau)\n",
    "\n",
    "\n",
    "Quantization isn't something new, it has been around for many years and have been extensively since the creation of digital electronics. In digital signal processing, quantization usually refers to converting continuous signal e.g. light, sound into discrete digital number. In deep learning, quantization genereally means converting signed single precision floating point (float32) to lower precision number format such as unsigned 8-bit integer (uint8). I wrote a [blog](https://medium.com/@soon_yau/speeding-up-deep-learning-with-quantization-3fe3538cbb9) introducing quantization which I'll summarize here. Although quantization from float32 to uint8 can reduce memory for 4x but it is not mainly to save storage but rather to speed up the inference. This can happen for 2 reasons:\n",
    "- Deep neural networks have a lot of parameters (weights) and loading them from memory can become the bottleneck rather than the actual computation, so is moving the activations back the the main memory\n",
    "- Fixed point computation is usually faster than the floating point counterpart. In some embedded platforms, floating point CPU is simply not available.\n",
    "\n",
    "\n",
    "\n",
    "In this tutorial, we'll learn to quantize a full precision graph into uint8 graph, and also study how the computation works in a quantized network.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow 1.10.0\n",
      "Python 3.5.2 (default, Nov 12 2018, 13:43:14) \n",
      "[GCC 5.4.0 20160609]\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import sys\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "print(\"Tensorflow\", tf.__version__)\n",
    "print(\"Python\", sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting from float32 into unsigned integer (uint)\n",
    "As mentioned in my blog, the quantization equation is as follow:\n",
    "\\begin{equation}\n",
    "x_{uint8} = \\frac{x_{float}}{x_{scale}} + x_{offset}\n",
    "\\end{equation}\n",
    "\n",
    "In quantization, we want to map the the minimum of float32 to minimum of unsigned integer (0), and the maximum of float32 to maximum of unsigned integer therefeore, the x_scale is the inverse of slope of linear equation.\n",
    "\n",
    "\n",
    "\\begin{equation}\n",
    "x_{scale} = \\frac{max(x_{float}) - min(x_{float})}{2^N-1}\n",
    "\\end{equation}\n",
    "\n",
    "$x_{offset}$ is an uint8 which in fact is quantized value that corresponds to 0 of real value (floating point numbers). N is bitwidth of quantized integer.\n",
    "\n",
    "Dequantization is simply the reverse of the above process to convert from integer back to floating point number.\n",
    "\n",
    "Below are the code to perform quantization and dequantization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "min = -2.398871 \n",
      "max = 2.760502 \n",
      "num_bits = 8 \n",
      "scale = 0.02023283 \n",
      "offset = 119\n"
     ]
    }
   ],
   "source": [
    "# set seed to ensure same random numbers are generated everytime\n",
    "np.random.seed(1515)\n",
    "\n",
    "# generate some random float numbers. By default, the generated numbers are in float64,\n",
    "# we cast it into float32. It is important to set the data type explicitly to make sure\n",
    "# correct arithmetic operation are used\n",
    "x = (1*np.random.randn(9,9)).astype(np.float32)\n",
    "\n",
    "def sat_uint(x, num_bits=8):\n",
    "    return np.uint8(np.clip(np.round(x), 0, 2**num_bits-1))\n",
    "    \n",
    "def quantize(x_float, num_bits=8, print_debug=False):\n",
    "    # calculate scale\n",
    "    x_min = np.min(x)\n",
    "    x_max = np.max(x)\n",
    "    x_range = x_max - x_min\n",
    "\n",
    "    quant_min = 0    \n",
    "    quant_max = 2**num_bits - 1\n",
    "\n",
    "    x_scale = (x_max - x_min)/(quant_max - quant_min)\n",
    "    \n",
    "    # calculate offset\n",
    "    x_offset = sat_uint(quant_max - x_max/x_scale, num_bits)\n",
    "    \n",
    "    # quantize\n",
    "    x_quant = x_float/x_scale + x_offset\n",
    "\n",
    "    if print_debug:\n",
    "        print(\"min = %f \\nmax = %f \\nnum_bits = %d \\nscale = %.8f \\noffset = %d\"\\\n",
    "              %(x_min, x_max, num_bits, x_scale, x_offset))\n",
    "    return sat_uint(x_quant, num_bits), x_scale, x_offset\n",
    "\n",
    "def dequantize(x_quant, x_scale, x_offset):\n",
    "    \n",
    "    x_quant = x_quant.astype(np.float32)\n",
    "    \n",
    "    x_dequant = (x_quant - x_offset) * x_scale\n",
    "    \n",
    "    return x_dequant\n",
    "\n",
    "\n",
    "x_quant, x_scale, x_offset = quantize(x, 8, True)\n",
    "\n",
    "x_dequant = dequantize(x_quant, x_scale, x_offset)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we plot the original floating point (x-axis) against quantized integer (y-axis) we can see that it is merely a linear mapping where the offset is quantized value that represent real-value of zero. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcFPW57/HPA4IIGAeFjASNY1xDFjHMdY8OwRhjNMq59xjN0aBZiFE8Gj1uxyTTuByjiZpEjRHjHvdrRIwrch13Y4CgQZGEKCoExYVRRgWc4bl/VLXTPdPTUzN0dXV3fd+vV7+669fVXU/ZyMNvN3dHRESkqwFJByAiIpVJCUJERApSghARkYKUIEREpCAlCBERKUgJQkREClKCEBGRgpQgRESkICUIEREpaIOkA1gfI0eO9IaGhqTDKOj9999n2LBhSYdRFmm510WLFtHR0cHYsWOTDqUs0vK7Qvru9cUXX3zL3Uf1dm5VJ4iGhgbmzJmTdBgFtbS00NTUlHQYZZGWe21qaqK1tbVi/8yVWlp+V0jfvU6YMOGVKOeqiUlERApSghARkYKUIEREpCAlCBERKUgJQkREClKCEBGpQplM/NdQghARqULTpsV/jaqeByEikjYffgjnn1+ea8VWgzCzLc3sYTN7wcyeN7MTwvKMmS0zs/nh44Ccz5xhZovNbJGZfS2u2EREqtFBB8HQoZ21B7PgEVdzU5w1iHbgZHefZ2YbA3PNbFb43sXu/svck81sLHAY8DngU8BDZra9u3fEGKOISMV75x3YbLPO4yOOgD/8AdzjvW5sNQh3X+7u88LXq4CFwJgiHzkYuMXd17j7y8BiYJe44hMRqQaZTH5yeOkluOGG8ly7LJ3UZtYA7Az8OSyaambPmdnVZjYiLBsDvJbzsaUUTygiIjXrlVeC5qNsc9KZZwY1hq23Do6bm+OPIfZOajMbDtwBnOju75nZ5cDZgIfPFwLf7cP3TQGmANTX19PS0lLymEuhra2tYmMrtbTca2trKx0dHam4V0jP7wqVda/ucMEFO3D//aM/Lpsx4wk22eQjckNsaoL+hNzW1hb53FgThJkNIkgON7r7HwHc/Y2c968E/hQeLgO2zPn4FmFZHnefDkwHaGxs9EpdgTFtq0Om4V7r6upobW1Nxb1Cen5XqJx7ffZZGDeu8/jyy+GYYwD2LNk1+pIIY0sQZmbAVcBCd78op3y0uy8PDycBC8LXM4GbzOwigk7q7YBn4opPRKRSrFsHEybAo48Gx0OHwptvBs9JirMGsSdwJPA3M5sflv03cLiZjSNoYloC/BDA3Z83s9uAFwhGQB2nEUwiUutmz4Z99+08vvNOOOSQ5OLJFVuCcPfHASvw1r1FPnMucG5cMYmIVIq1a2G77eDVV4Pjz30O5s+HDSpo+rKW2hARKbObb4YNN+xMDk88AQsWVFZyAC21ISJSNu+9B5ts0nl80EFw113BcNZKpBqEiEiMsstg/PKX+cnhhRdg5szKTQ6gBCEiEqtp04IkcMopwfHxxwejlj772WTjikJNTCIiMXCHvffOL1u6FMZU0foQqkGIiJTYkUfCgAHw+OP55VdemUw8/aUahIhIiXR0wKc/Df/6V3553KuuxkU1CBGRErjqqmCYajY5/OAH1ZsYslSDEBFZD6tXw6abBju9QdC09PbbUFcXHJdj1dW4qAYhItJPzc2w0UadyeGcc4JmpmxygPh2eysH1SBERProrbfgk5/sbEIaPjwo23DDZOMqNdUgRET6YPJkGDWqMzlcdx2sWlV7yQGUIEREiso2Ef3jH8GEt+uvD44//Wlob4fvfCex0GKnBCEiUsS0acHubdtv31n20EPBlqADByYWVlmoD0JEpAfnnBM8P/JI8LzrrvDUU5W9flIpKUGIiHRxyinB4npd7b9/epIDKEGIiOQ57DC49db8smqf8NZfShAiIsA//wnbbptftmxZdS2uV2rqpBaRVMtk4Pjjx+Ulh2OOCWoNn/pUdc+EXl+qQYhIKmUy8I1vBKOUIJj6PHAgrFwJG2+cf15aqQYhIqnzwQdBYthll86y884L5jXkJoe0U4IQkVQ59FAYNqx7+RlnpLu2UIiamEQkFd55BzbbrPB7kycv4dprG8oaTzVQDUJEat748fnJIVuDyA5fPeqoJWWPqRooQYhIzXrllWBi27x5nWUzZkBbW+dxmkcp9UZNTCJSk0aNCpbgzho9OpjXkJ0JnU0MmQy0tJQ7uuqgGoSI1JR584IkkJsc5swJtgLNXSZDHdK9U4IQkZrQ3BwkgPHjO8u++MWgnyG3TKJTE5OIVL2ZM+Gss/LLXn4ZGhoSCadmKEGISNVat677ngwHHAD33JNMPLUmtiYmM9vSzB42sxfM7HkzOyEs39TMZpnZP8LnEWG5mdlvzGyxmT1nZl+KKzYRqX6/+lXhDXvuvVf9C6USZw2iHTjZ3eeZ2cbAXDObBRwFzHb3n5vZ6cDpwGnA14HtwseuwOXhs4jIxz76CAYPzi/73vfgqqvSuyx3XGKrQbj7cnefF75eBSwExgAHA9eFp10HHBK+Phi43gNPA3VmNjqu+ESkemRrBFOn5icHM/jwQ/j97xMJq+aVZRSTmTUAOwN/BurdfXn41utAffh6DPBazseWhmUiknLTpgXJ4LLLOsvOPjvogxgyJDjWhLfSi72T2syGA3cAJ7r7e5YzENnd3cz6VCk0synAFID6+npaKnSGS1tbW8XGVmppudfW1lY6OjpSca9QOb/rSSd9AehcJ2ODDTq4//7HGDgwf4JbU1P/J7xVyr2WQ1vuNPJexJogzGwQQXK40d3/GBa/YWaj3X152IS0IixfBmyZ8/EtwrI87j4dmA7Q2NjoTU1NcYW/XlpaWqjU2EotLfdaV1dHa2trKu4Vkv9dTz4ZLrqoe/mZZw5k4sSmkl4r6Xstp74kwtgShAVVhauAhe6e+zPPBCYDPw+f78opn2pmtxB0Tr+b0xQlIimRycCNN8Lixfnl69blz4SW+MVZg9gTOBL4m5nND8v+myAx3GZm3wNeAQ4N37sXOABYDHwAHB1jbCJSgRYsyO7w1umee4Kd35Qcyi+2BOHujwM9/aQTC5zvwHFxxSMile0Tn4BVqzqPR46EN98MXqsDOhlai0lEEvXgg0HtIDc5QLDYXnZ4qya+JUNLbYhIIgotk7HjjvDii5rwVilUgxCRsrviiu7JYflyWLgwmXikMNUgRKRs1q6FDTfML9t3X5g1q/NY/Q2VQzUIESmLk07qnhxWrcpPDqD+hkqiBCEisWlqCpKAGVx8cWf5978f9DMMH55YaBKBmphEJDaPPBIMX80ygzVrYNCg5GKS6FSDEJGSe/XV7hPbzjsvGLmk5FA9iiYIMxtoZr8sVzAiUv2GDIGttupefv/95Y9F1k/RJiZ37zCzvcoVjIhUr3nzYPz47uWa01C9ovRB/NXMZgK3A+9nC3NWZxWRFHMPag1r13aWDRkSbOSj9ZOqW5Q+iCHA28BXgIPCx4FxBiUi1WHGDBgwID85PPFEkBwA9tknmbikNHqtQbi7VlUVkTzt7UEtoaOjs2zUKFixIv+8lOzBU7N6rUGY2fZmNtvMFoTHXzSzn8QfmohUogsvDEYi5SYHCFZeNdNEt1oSpQ/iSuAU4AoAd3/OzG4CzokzMBGpHJkMnHJK94ltAwYEtYkBA9QZXYui9EEMdfdnupS1xxGMiFSWbG1g2rTuyeGJJ4JahDqia1eUBPGWmW0DOICZ/R9AW4GKpMC0ad0TwMiRQW1hjz06y7TAXm2K0sR0HDAd2NHMlgEvA0fEGpWIJO6b39y9YPlxBfZ9VL9Dbeq1BuHuL7n7vsAoYEd338vdl8QemYgk4kc/yu7wlr/0anNzUHNQMkiPXmsQZnZSl2OAd4G57j4/prhEpMzcob6+cx/orJUrYcQIJYY0itIH0QgcA4wJHz8E9geuNLNTY4xNRMrkO98JRiLlJoe9934dgLo69TGkVZQ+iC2AL7l7G4CZNQP3AHsDc4EL4gtPROLU3Az/8z/BUNVca9bAk0++SEvL5oBqD2kVpQbxSWBNzvFHQL27f9ilXESqyE03wVln5SeHM88MmpoGDw6OlRjSLUoN4kbgz2Z2V3h8EHCTmQ0DXogtMhGJRXt7kAAKTWw791zYYAMlBglEWYvpbDO7D9gzLDrG3eeEr/8jtshEpGSamoLHkCFwxhk9n9fcrOQgnaJuOToEeM/drzGzUWa2tbu/HGdgIlI6jzwSPIrRUhnSVZTF+pqB04DsvzsGAX+IMygRKZ1Jk7qXnXdefkLQKCUpJEoNYhKwMzAPwN3/ZWYbxxqViKy3PfaAp54q/N7q1cFzNjGoWUkKiZIg1rq7m1l2LaZhMcckIushk4F77oE5c7q/17UZSYlBiomSIG4zsyuAOjP7AfBd4PfxhiUi/fH888ECe7kGDYKPPkomHqluUdZi+iXwf4E7gB2An7n7b3r7nJldbWYrshsNhWUZM1tmZvPDxwE5751hZovNbJGZfa1/tyOSXiNGwOc/n182Z06wHeg++6ifQfouylpM57v7acCsAmXFXAtcClzfpfziMOnkXmMscBjwOeBTwENmtr27d9mzSkS6mjUL9tuv8Ht33w3jx2vrT+mfKDOpv1qg7Ou9fcjdHwXeiRjHwcAt7r4mHD67GNgl4mdFUqm5OWg+KpQc3LXyqqy/HhOEmf3IzP4G7GBmz+U8XgaeW49rTg2/52ozGxGWjQFeyzlnaVgmIgVcdVX3ZTJ23FFzGaS0ijUx3QTcB5wHnJ5Tvsrdo9YMurocOJtgd7qzgQsJOr0jM7MpwBSA+vp6Wiq07tzW1laxsZVaWu61tbWVjo6ORO913TqYOHEfIH+bt5kzH2PjjTtoaYHJkxtoaVmy3tdKy+8K6bvXqHpMEO7+LsG+D4eb2UCgPjx/uJkNd/dX+xqYu7+RfW1mVwJ/Cg+XAVvmnLpFWFboO6YT7HBHY2OjNzU19TWMsmhpaaFSYyu1tNxrXV0dra2tidxrJgOzZ8Pjjxd+f+7cL3/cnBSE17De10zL7wrpu9eoonRSTwUywBvAurDYgS/2NTAzG+3u2f2sJwHZEU4zCRYAvIigk3o74Jm+fr9ILfrJT4JF9ApRk5LEKco8iBOBHdz97b58sZndDDQBI81sKdAMNJnZOIIEs4Rg8yHc/Xkzu41gddh24DiNYJK0y2Tg1lvhxRfzy489Fi67LNgWVCROURLEawRNTX3i7ocXKL6qyPnnAj38O0kkXU49FX7xi8Lv/fa3MGqU5jVI/KIkiJeAFjO7h5wNgtz9otiiEkmh7JLcM2bAs88WPkfLcUs5RUkQr4aPweFDREosk4m2JLeSg5RTlA2DpvV2joisn67rJ3W1zz7Z0Uki5dNjgjCzX7n7iWZ2N0Gnch53/2askYnUuEwG7rgDFiwofp6alSQpxWoQN4TPvyxyjoj0U2+1BlBykGQVmyg3N3zupVVURPrimmvgu0XWD8hdQ0nJQZIUdU9qEVkPTU1BP8JZZxU/Tzu8SSVRghApg55GKGVnQqvGIJVICUIkRh0dsEGB/8s+/BA22qjzWIlBKlGxUUwFRy9laRSTSM8yGTj77GD11UI22ihochKpZMVqENnRS/8GbA78ITw+nGDhPhEp4P33i49Q0gJ7Ui163DDI3R8JRzDt6e7fcve7w8e3gS+XL0SR6jFuHAwfnl928snJxCKyvqL0QQwzs8+4+0sAZrY1MCzesESqy0knwcUXF37vwgthk03gxBPLG5PI+oqSIH5MsFjfSwTbWG1FuEy3SJplMsFjww1h7drC56g5SapZj01MWe5+P8EGPicA/0mwN8QDcQcmUummTQv2ZMhNDmZKClI7ek0QZjYUOAWY6u7PAp82swNjj0ykghUaugqds6CbmzVKSapflCama4C5wO7h8TLgdjr3kxZJjVdeGVp0JzetnSS1JEqC2Mbdv2VmhwO4+wdm2uxQ0ieYCd3zlihqWpJa02sTE7DWzDYinDRnZtuQs7OcSK27+uru+z9vtVXwnE0K2v5TalGUGkQGuB/Y0sxuBPYEjooxJpGKsG4dDBzYvfyDD4KZ0NmkoWYlqVVRdpR70MzmArsRDHM9wd3fij0ykYRkMsFaSRdckF++ySbQ0ND68RpKWnlVal2vCcLMZgMXuvs9OWXT3X1KrJGJJKC9vfAyGe3tMHEitLZ2likxSK2L0gexNXCameW2sjbGFI9IYo45BgYN6l7mXripSaTWRUkQrcBEoN7M7jazTWKOSaRsMpmgT8EMrrii+/v19WUPSaRiROmkNndvB441s6OAx4ERsUYlUibTphVuUtKQVZFoNYjfZV+4+7UEI5gejCkekbJYsaL70FVQYhDJ1WOCMLNPhC9vN7NNsw/gZeC/yhKdSIllMvCJTxRuOsqOStKcBpFAsSamm4ADCZbZcIIhrlkOfCbGuERKKpOBww6L1pyk0UkigR4ThLsfGD5vXb5wREor+5d9ob6Gv/4Vdt657CGJVI1I8yDcfWJvZSKVqNjWnzNmqDlJpJgeE4SZDQGGAiPNbASdTUyfAMb09sVmdjVBE9UKd/98WLYpcCvQACwBDnX3leHif78GDgA+AI5y93n9vCcRoHAnNGhpDJGoio1i+iFB/8OO4XP2cRdwaYTvvhbYv0vZ6cBsd98OmB0eA3ydYFOi7YApwOXRwhfJl8nAfvv1nByy54hI74r1Qfwa+LWZHe/ul/T1i939UTNr6FJ8MNAUvr4OaAFOC8uvd3cHnjazOjMb7e7L+3pdSS/34k1Kak4S6Zsoi/VdYmZ7EDQLbZBTfn0/rlef85f+60B2sOEY4LWc85aGZd0ShJlNIahlUF9fT0tLSz/CiF9bW1vFxlZqSd/rgQfuxd57v859922RV77bbm/w9NPBH7HJk5fQ1LQEgP6G2traSkdHh37XGpS2e40qSif1DcA2wHygIyx2oD8J4mPu7mbW52lJ7j4dmA7Q2NjoTU1N6xNGbFpaWqjU2EotyXtduxbef59uyQHga1+r5+mns30ODQT/xum/uro6Wltb9bvWoLTda1RRltpoBMaGzT/r641s05GZjQZWhOXLgC1zztsiLBPp0Ve/Cg891L1c8xpESiPKUhsLgM1LdL2ZwOTw9WSCDu9s+XcssBvwrvofpCennx50QhdKDl0pOYj0X5QaxEjgBTN7hpytRt39m8U+ZGY3E3RIjzSzpUAz8HPgNjP7HvAKcGh4+r0EQ1wXEwxzPbpvtyFpse228M9/di/P1hqUEERKJ+qWo33m7of38Fa3CXZh89Vx/bmOpMNLL8E223Qvd88f0qoEIVI6UUYxPVKOQEQKyWTg3HODHd16en8T7VAiEote+yDMbDcz+4uZtZnZWjPrMLP3yhGcpFcmA7NnB/MacpPD0KHBs3vwyGTytwEVkdKJ0kl9KXA48A9gI+D7wGVxBiXplbu43r775r/3wgvBkFYRKY8oCQJ3XwwMdPcOd7+G7ktoiJTEtGk9L5MxdmyQQDQjWqQ8onRSf2Bmg4H5ZnYBwezmSIlFpC8GDer5Pe30JlJ+Uf6iPxIYCEwF3ieY0Pa/4wxK0mXs2KDW0LUjWjUFkWRFGcX0SvjyQ6DIUmgi0WUywWPgQFi3Lv+91athyBANWRVJWpRRTC+b2UtdH+UITmpXtq8hNzlMnRo8b7hhZ+0hm0hEpPyirsWUNQT4d2DTeMKRWtfe3nNfw6WXwj77BK+VFESSF6WJ6e0uRb8ys7nAz+IJSWrVTjvBc88Vfk+d0CKVJ8py31/KORxAUKOIUvMQIZMJFtfbaKPu72WTQrHd30QkOVH+or8w53U74V7SsUQjNWfatO67vM2ZA405DZcarSRSmaI0MU0oRyBSO5qa4L77OpfFyBo1ClaEO4DkJgX1N4hUpihNTCcVe9/dLypdOFILHnmke3IAOPbYztdKCiKVL+oopv9FsKkPwEHAMwRrM4l87O23YeTIpKMQkVKJkiC2AL7k7qsAzCwD3OPuR8QZmFSHTCZoUho6FD78sPA5GqEkUp2iLLVRD6zNOV4blknKZTJBB/SECU3dkkN2OW4RqV5RahDXA8+Y2Z3h8SHAtbFFJFWj6+gkgDVrgpnQWRqhJFK9eq1BuPu5BHtErwwfR7v7eXEHJpVrypTCcxeam2Hw4M7Z0KDOaJFqFmnCm7vPA+bFHItUgSFDglpCV83NncmgpaWcEYlIXLSvg0Ry//1BraFQcnj44RbVFERqkJbMkKLcgz6Fjz7KL+/oCJbqVh+DSO1SDUK6ydYGDjwQBgzITw633RYkjQED8puVRKT2qAYhebJDVwuNUOo6bFXJQaS2qQYheQolhgcf1JwGkTRSghAAfvKTnpfd3m8/1RZE0khNTCmXycADD8DTTxd+XzUHkfRSgkixlSsLNylBkBi0kY9IuqmJKYUyGRg7FjbtsrP4ggXBc3boqoawiqRbIjUIM1sCrAI6gHZ3bzSzTYFbgQbCXevcfWUS8dWy117rudZw++35Q1fV7yCSbkk2MU1w97dyjk8HZrv7z83s9PD4tGRCq0177glPPtm9XP0MIlJIJTUxHQxcF76+jmDVWCmB554L+hMKJQdQTUFECksqQTjwoJnNNbMpYVm9uy8PX7+O9pxYL9m/9LfbDnbaqbN82207X7trNrSI9Mw8gfYFMxvj7svM7JPALOB4YKa71+Wcs9LdRxT47BRgCkB9ff34W265pVxh90lbWxvDhw9P7PoTJjR1K/vtb+fw2c+2ffzeww+3lORaSd9ruZx44ol0dHRwySWXJB1KWaTld4X03etBBx00190bezs3kT4Id18WPq8INyLaBXjDzEa7+3IzGw2s6OGz04HpAI2Njd7U1FSmqPumpaWFpGL75Cfzj8ePh7/8BcyCPw/Z0Umlii/Jey2nuro6WltbU3GvkJ7fFdJ3r1GVvYnJzIaZ2cbZ18B+wAJgJjA5PG0ycFe5Y6t2xx4b9DW8+WZ++YEH5s9pyGTUrCQivUuiBlEP3GnB31gbADe5+/1m9hfgNjP7HvAKcGgCsVUld9hlF5gzp3u5iEh/lT1BuPtLwE4Fyt8GJpY7nmr32GOw9975ZUuWQENDEtGISC2ppGGu0gfNzfCZz+Qnh29/O6g1bLWVZkGLyPrTWkxVaMYMOOus/LLXX4f6nIHB6mMQkfWlGkQVaW8PksCkSZ1lU6cGtYZ6zRoRkRJTgqgSV14JgwbBii6Dfy+9VLUFEYmHmpgq3AcfwGabwerVnWWzZ8PEiRqlJCLxUg2igp19Ngwb1pkcxo+Hjz6Cr3wl2bhEJB1Ug6hAb78NI0fml82dC1/6UuexRimJSNxUg6gwxx6bnxz22w86OvKTA6jfQUTipxpEhViyBLbeOr9s0SLYfvtEwhERUQ2iEkyalJ8cjjgC1q1TchCRZKkGkaD582HnnfPLXn0VttwymXhERHKpBpGA5mbYfff85HDSScGwVSUHEakUqkGU2cMPd18mY8UKGDUqmXhERHqiGkQZZDJB7WDcuPw5DOefH5QrOYhIJVKCKINp02DAAHj22fzy007TcFURqVxqYorZT3+af3zNNXD00VomQ0QqnxJETK6+uoEJE7qXL1lS9lBERPpFCSIGjz4KN9zQkFemGoOIVBv1QZRQezt84Quwzz7B8Y47wpo13c9Tv4OIVAMliBK5995gv4YFC4Ljiy6az8KFMHiwFtYTkeqkBLGe1qyBzTeHb3wjON5rr6AmsfPOrR+foxqDiFQjJYj1cMstMGQIvPFGcPzMM/DYYzBwYLJxiYiUgjqp+6GtDerqgmW4AQ45BP74RzBLNi4RkVJSDaKPrrgCNt64Mzk8/zzceaeSg4jUHtUgIlq5EjbdtPP4+9+H6dOVGESkdqkGEcH55+cnh5dfhiuvVHIQkdqmGkQRr78Oo0d3Hp96apAsRETSQAmiB6edBhdc0Hm8fHkwnFVEJC3UxNTFkiVB01E2Ofz858EyGUoOIpI2qkHk+MEP4Pe/7zx+5x0YMSK5eEREklRxNQgz29/MFpnZYjM7vRzXXLgwqDVkk8PvfhfUGpQcRCTNKipBmNlA4DLg68BY4HAzGxvHtbK7vE2aBGPDK2ywQTAJ7oc/jOOKIiLVpaISBLALsNjdX3L3tcAtwMFxXCi7y9uMGcHxLbfARx/BsGFxXE1EpPpUWh/EGOC1nOOlwK49nbxo0SKampr6fJHVqztfDx4Mu+4Kl18ePEqltbWVurq60n1hBUvLvc6fP5/29vZ+/ZmrRmn5XSF99xpVpSWIXpnZFGAKwKBBg/p0s6+/PoQ33hiSV7Z2Lfz976vZfPPVPXyqfzo6OvoUWzVLy722t7fj7qm4V0jP7wrpu9eoKi1BLAO2zDneIiz7mLtPB6YDNDY2+pw5c/p1IbN4d3lraWlJzb8003KvTU1NtLa2Mn/+/KRDKYu0/K6QvnudUGg/5AIqrQ/iL8B2Zra1mQ0GDgNmJhyTiEgqVVQNwt3bzWwq8AAwELja3Z+P41ra5U1EpLiKShAA7n4vcG/c19EubyIixVVaE5OIiFQIJQgRESlICUJERApSghARkYKUIEREpCDzOGeLxczM3gReSTqOHowE3ko6iDLRvdYm3WttGgkMc/dRvZ1Y1QmikpnZHHdvTDqOctC91ibda23qy72qiUlERApSghARkYKUIOIzPekAykj3Wpt0r7Up8r2qD0JERApSDUJERApSgoiRmf3CzF40s+fM7E4zq9ktq8zs383seTNbZ2Y1NxrEzPY3s0VmttjMTk86njiZ2dVmtsLMFiQdS5zMbEsze9jMXgj/7J6QdExxMrMhZvaMmT0b3u+03j6jBBGvWcDn3f2LwN+BMxKOJ04LgH8DHk06kFIzs4HAZcDXgbHA4WY2NtmoYnUtsH/SQZRBO3Cyu48FdgOOq/HfdQ3wFXffCRgH7G9muxX7gBJEjNz9QXdvDw+fJtghrya5+0J3X5R0HDHZBVjs7i+5+1rgFuDghGOKjbs/CryTdBxxc/fl7j4vfL0KWAiMSTaq+HigLTwcFD6KdkIrQZTPd4H7kg5C+mUM8FrO8VJo8PF3AAADwElEQVRq+C+SNDKzBmBn4M/JRhIvMxtoZvOBFcAsdy96vxW3YVC1MbOHgM0LvHWmu98VnnMmQXX2xnLGVmpR7lWk2pjZcOAO4ER3fy/peOLk7h3AuLA/9E4z+7y799jXpASxntx932Lvm9lRwIHARK/yMcW93WsNWwZsmXO8RVgmVc7MBhEkhxvd/Y9Jx1Mu7t5qZg8T9DX1mCDUxBQjM9sfOBX4prt/kHQ80m9/AbYzs63NbDBwGDAz4ZhkPZmZAVcBC939oqTjiZuZjcqOpDSzjYCvAi8W+4wSRLwuBTYGZpnZfDP7XdIBxcXMJpnZUmB34B4zeyDpmEolHGgwFXiAoCPzNnd/Ptmo4mNmNwNPATuY2VIz+17SMcVkT+BI4Cvh/5/zzeyApIOK0WjgYTN7juAfPbPc/U/FPqCZ1CIiUpBqECIiUpAShIiIFKQEISIiBSlBiIhIQUoQIiJSkBKE1CQz+08zW2hmN5rZUWZ2aT+/p8nM9sg5PsbMvlO6SPsUS6/XNrNxNT5UU8pIM6mlVh0L7OvuS8PZ7P3VBLQBTwK4e2JzWSJeexzQCNwbcziSAqpBSM0JJyR+BrjPzH7c5b0GM/t/4R4ds83s02H5QWb2ZzP7q5k9ZGb14QJuxwA/DidRfdnMMmb2X+FnWszs/HCN/b+b2ZfD8qFmdlu4z8Cd4fd22yPDzJaY2QVm9rfwO7btJcai1w5neZ8FfCuM91vx/BeWtFCCkJrj7scA/wImuPvFXd6+BLgu3KPjRuA3YfnjwG7uvjPBct6nuvsS4HfAxe4+zt0fK3C5Ddx9F+BEoDksOxZYGe4z8FNgfJFw33X3LxDMuv9VLzEWvXa4FPnPgFvDeG8tcl2RXilBSNrsDtwUvr4B2Ct8vQXwgJn9DTgF+FzE78su8DYXaAhf70WQZAhXynyuyOdvznnevZcYo1xbpGSUIEQClwCXhv+a/yEwJOLn1oTPHfSvT897eF2Oa4sUpQQhafMkwWqsAP8BZJuNNqFzCe/JOeevIlhwsS+eAA4FCLew/EKRc7+V8/xULzFG0Z94RQpSgpC0OR44OlzR8kggu1F9BrjdzOYCb+WcfzcwKdtJHfEavwVGmdkLwDnA88C7PZw7IozlBCDbod5TjFE8DIxVJ7WUglZzFSkxMxsIDHL31Wa2DfAQsEPYiZx73hKg0d3fKvA1IolTu6VI6Q0lWHd/EGDAsV2Tg0g1UA1CREQKUh+EiIgUpAQhIiIFKUGIiEhBShAiIlKQEoSIiBSkBCEiIgX9f1ZqVlGcjNtsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.plot(x, x_quant, marker='+',color='b')\n",
    "ax.set_xlabel(\"floating point\")\n",
    "ax.set_ylabel(\"quantized integer\")\n",
    "ax.axhline(y=0, color='k')\n",
    "ax.axvline(x=0, color='k')\n",
    "ax.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  tf.fake_quant_with_min_max_args \n",
    "\n",
    "In fact, the quantize-dequantize cycle we have implemented is exactly tf.fake_quant_with_min_max_args is doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.placeholder(tf.float32)\n",
    "fake_quantized_x = tf.fake_quant_with_min_max_args(inputs, np.min(x), np.max(x), num_bits=8)\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    fake_x = sess.run(fake_quantized_x, feed_dict={inputs:x})\n",
    "\n",
    "np.testing.assert_array_almost_equal(fake_x, x_dequant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Offset\n",
    "\n",
    "It is worth mentioning why offset is rounded and cast to unsigned integer rather than leaving it as float. Let's use an example. If the floating numbers range is +-1.0, then \n",
    "\n",
    "\\begin{equation}\n",
    "x_{scale} = \\frac{2}{255},  x_{offset} = 127.5\n",
    "\\end{equation}\n",
    "and therefore\n",
    "\\begin{equation}\n",
    "x_{uint8} = round(127.5) = 128 \n",
    "\\end{equation}\n",
    "as the number will need to be cast into integer. Remember, decimal point is exclusive to floating point and don't exist in integer representation. \n",
    "\n",
    "Now if we de-quantize it\n",
    "\n",
    "\\begin{equation}\n",
    "x_{dequant} = (x_{quant} - x_{offset}) \\times x_{scale}\n",
    "\\end{equation}\n",
    "\\begin{equation}\n",
    "x_{dequant} = (128 - 127.5) \\times \\frac{2}{255}\n",
    "\\end{equation}\n",
    "\\begin{equation}\n",
    "x_{dequant} = 0.003921569\n",
    "\\end{equation}\n",
    "\n",
    "and the dequantized value is no longer 0. As 0 happens a lot in weights, activation and also used frequently as padding, we want to represent it precisely at the expense of moving the quantization error to other values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quantization Loss\n",
    "\n",
    "As shown earlier, rounding can create discrepancy in dequantized offset. Some information will be lost after quantization, we compare the error between the original and dequantized values and plot them on a graph and we can see that they are in fact non-zero despite not too big either. The error magnitude depends on the real values' range, distribution and quantization bit numbers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "error -0.010020614 0.009811968\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'quantization error')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAEKCAYAAAArYJMgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3X+0nFV97/H3xxDg6FXCjxTxQEzuBXEFqSBHpI3tElSItxZSQMFqRUvldglqq+UaLv7gR7uIsor1Fn8sxFZRKyBCiIKmYKBWKj9ODMjvSxQsOaUSgUCRqEn43j/mOWRyMnPmOTOz5/kxn9daZ2XmmWdm9jOT2d9n7/199lZEYGZmlsrzii6AmZnVmwONmZkl5UBjZmZJOdCYmVlSDjRmZpaUA42ZmSXlQGNmZkk50JiZWVIONGZmltQORRegDPbYY4+YP39+0cUwM6uU1atX/yIi5nbaz4EGmD9/PuPj40UXw8ysUiT9LM9+7jozM7OkHGjMzCwpBxozM0vKgcbMzJJyoDEzs6QcaMzMLCkHGjMzS8qBxszMknKgMTOzpBxozMwsKQcaMzNLqtBAI2mxpPslrZW0tMXjO0m6LHv8Fknzs+27S7pB0tOSLpzynEMk3Zk95/9K0mCOxszMWiks0EiaBXwGeBOwEHibpIVTdjsZeCIi9gU+BXwi2/4r4KPAX7V46c8B7wH2y/4W97/0ZmaWV5GzNx8KrI2InwJIuhQ4BrinaZ9jgLOy21cAF0pSRPwS+IGkfZtfUNJewIsi4ubs/iXAEuA7KQ/ErKyWr5ngrBV3s2HjJgB2ff5sPv6HB7Dk4NGCS2bDpMhAMwo83HR/HfCadvtExGZJTwK7A7+Y5jXXTXlN/6ISWL5mgvNX3s9/bNjIS+aMcPpR+7vyKpnlayY4/Rt3sOnZeG7bE89s4vQr7gDw92UDM7TJAJJOkTQuaXz9+vVFF6dSlq+Z4Iwr72Riw0YCmNiwkTOuvJPlayaKLpo1OX/l/dsEmUmbtgTnr7y/gBLZsCoy0EwA+zTd3zvb1nIfSTsAuwCPdXjNvTu8JgARcVFEjEXE2Ny5HReIsybnr7yfjZu2bLNt46YtrrxK5j82bOzqMbN+KzLQ3AbsJ2mBpB2BE4EVU/ZZAZyU3T4eWBUR25+iZSLiEeApSYdl2WbvBK7uf9GHW7tKypVXubxkzkhXj5n1W2GBJiI2A6cBK4F7gcsj4m5J50g6Otvti8DuktYCHwSeS4GW9BBwAfAuSeuaMtbeC1wMrAV+ghMB+q5dJeXKq1xOP2p/Zj9v++z+2bPE6UftX0CJbFgVmQxARFwLXDtl28eabv8KeEub585vs30ceEX/SmlTnX7U/pxx5Z3bdJ+NzJ7lyqtkJgf7nXVmRSs00Fg1TVZSzjorvyUHj/p7scI50FhXXIHZJKe6WycOND3wD8yG3WSq+2Q36mSqO/g6nTIbdN3lQNMl/8DMpk91z/s78AnbYBVRdw3tBZu98rUkZr2nuvvi38Erou5yoOmSryUx6z3V3Sdsg1dE3eVA0yVfS2LWSHUfmT1rm20zSXX3CdvgFVF3OdB0qdcfmFkdLDl4lPOOPZDROSMIGJ0zwnnHHpi7r98nbINXRN3lZIAu+VoSs4ZeUt198e/gFVF3aZqpw4bG2NhYjI+PF10Ms6HkrLPqkrQ6IsY67ecWjZkVyhf/1p/HaMzMLCkHGjMzS8pdZ2bT8PiBWe8caMza8DRDZv3hQGPWRj/m8eo3t7CsihxorCd1rvjKdtW6W1hWVU4GsK6VZULE5WsmWLRsFQuWXsOiZav69v5lu2rd84JZVTnQWNfKUPGlDHZlm2aobC0ss7wcaKxrZaj4Uga7Xufx6reytbDM8vIYjXXtJXNGmGgRVAZZ8aUOdmW6an0m84LVeezMqsctGutaGbqWhuksP28LqyxjZ9a7VOOPg+YWjXWtDDNYD9vsv3laWGVMy7aZq1OWoQON9aTorqVOwW4Yu5DKMHZmvavTCYMDjVVeu2BX5jPClAGwDGNn1rs6nTB4jMZqqwzp162kHkMpw9iZ9a5O449u0QypYehSanfmN7FhI4uWrXru2A9/+VxuuG/9wD6LVF0izd/pnOfPZqcdnseTGzfV9vutuzqNPzrQDKEydyn1U7suJMFz2yc2bOSrN//7c48N4rNI0SUy9Tt94plNjMyexadOOGgg3+kwnLgMWhmSbfrFXWdDqKxdSv3WqgtJQKfFy1N/Fim6RIr8Tp1Onc6Sg0e5aekRPLjsD7hp6RGVDDLgQDOUqjTI2Mt1BK2uO+kUZCal/CxSjKEU+Z0Oy4mLdc9dZ0OoKllJ/ejim5qRtmjZqpbHPlXKzyJFl0ie7zRV91aVTlysGG7RDKGqZCWlOFNudexTDeKz6HeXSKfvNGX3Vp2yoywNB5ohVLbJIttJcabc6tjfcdi80n8WnXT6TlN2b1XlxMWK466zIVX0Ff15pOriq8Kxd2O640rZvVWn7ChLw4HGSqtO1xEULfW4XF2Dt/WHu86stKrSxVcF7t6yIrlFY6XmM+WZaZdZ5u4tK1KhgUbSYuDTwCzg4ohYNuXxnYBLgEOAx4ATIuKh7LEzgJOBLcD7I2Jltv0h4L+y7ZsjYmwgB2NWsE7p4A7aVpTCus4kzQI+A7wJWAi8TdLCKbudDDwREfsCnwI+kT13IXAicACwGPhs9nqTDo+IgxxkbJj4wkkrqyLHaA4F1kbETyPiN8ClwDFT9jkG+HJ2+wrg9ZKUbb80In4dEQ8Ca7PXMxta000i6ulgrEhFBppR4OGm++uybS33iYjNwJPA7h2eG8A/S1ot6ZR2by7pFEnjksbXr1/f04GYlcF0GWSee8yKVMess9dGxKtodMmdKun3W+0UERdFxFhEjM2dO3ewJTRLYLpZD9yFZkUqMtBMAPs03d8729ZyH0k7ALvQSApo+9yImPz3UeAq3KVmQ2IyHbwdzz1mRZk20EiaJem+RO99G7CfpAWSdqQxuL9iyj4rgJOy28cDqyIisu0nStpJ0gJgP+BWSS+Q9MKs7C8AjgTuSlR+s9JZcvAoo557zEpm2kATEVuA+yXN6/cbZ2MupwErgXuByyPibknnSDo62+2LwO6S1gIfBJZmz70buBy4B/gucGpW1j2BH0i6A7gVuCYivtvvspuVmS/OtLJRo4EwzQ7S94GDaVTcv5zcHhFHt31SxYyNjcX4+HjRxagdr7pYHH/2NgiSVue5jCTPBZsf7UN5bMgMy3LRZeWLM61MOiYDRMS/APcBL8z+7s22mbXliwfNbFLHQCPprTS6zd4CvBW4RdLxqQtm1eZVF81sUp6uszOBV2fpwkiaC1xP40p9s5aqsly0maWX5zqa500GmcxjOZ9nQ8yZT2Y2KU+L5ruSVgJfz+6fAFybrkhWB56Wvr9aZZGBP1+rho7pzQCSjgVem93914i4KmmpBszpzVZmUzP4AGbPEgRsenbr73dk9iwvDGcD1Zf05mzq/esj4nDgyn4Vzszya5XBt2nL9ieIk1l9DjRWNnlmBnhW0i4DKo+ZTTGTTD1n9VkZ5RmjeRq4U9J1bDszwPuTlcrMntMug6/dvmZlkyfQXIm7zcwKc/pR++ceo3FWn5VRnjGaIyPi7QMqj5lN0S6Dr9U2j88MhueSm5lpA01EbJH0Ukk7ZsstmyU36B9xWSuNPOUqQzmHjefxm7k8XWc/BW6StIJtx2guSFYqG1qD/hGXtdIoa7ls+nn8/N20lucK/58A3872fWHTn1nfDXoyzrJO/lnWcpnn8etGxxZNRJwNIOn5EfFM+iLZMBv0j7islUZZy1VV/ewe9Tx+M5dn9ubfkXQPjaUCkPRKSZ9NXjIbSu1+rKl+xHneb/maCRYtW8WCpdewaNkqlq+ZSFKWmZbL8pnshpzYsJFgazdkt9+j5/GbuTxdZ38HHEVjMk0i4g7g91MWyobXoH/End6v35VUv8pl+fW7G3LJwaOcd+yBjM4ZQcDonBFP/dNBnmQAIuJhSc2btrTb16wXg56Ms9P7FTXw60lJ+ydFN6RXMJ2ZPIHmYUm/C4Sk2cAHgHvTFsuG2aB/xNO9X5FjJa7M+sNjKsXL03X258CpwCgwARyU3TerPY+VVJ+7IYvXMdBExC8i4u0RsWdE/FZEvCMiHhtE4cyK5kqq+jymUrxcYzRmw8pjJfXgbshiOdCYdeBKyqw3ecZozMzMutaxRSNpJ+A4YH7z/hFxTrpimZlZXeTpOrsaeBJYDfw6bXHMzKxu8gSavSNicfKSmJlZLeUJNP8m6cCIuDN5acysb8q6zo4NnzyB5rXAuyQ9SKPrTEBExG8nLZmZdc3r2cyMg3JaeQLNm5KXwsy20WvF58W58nNQTi/PejQ/k/RK4PeyTf+azeBsQ8pnf2m1qvhOv+IOzlpxN09u3JTrM/d6Nvk5KKeXZz2aDwBfA34r+/uqpPelLpiVU1HT5g+TVhXfpi3Bho2bcn/mnqMtPwfl9PJcsHky8JqI+FhEfAw4DHhP2mJZWdV9ieEiFjmbKk8F1+kz9xxt+Tkop5cn0Iht15/Zkm2zIVTns7+ytNbyVnDTfeaeSDI/B+X08iQD/CNwi6SrsvtLgC+mK5KVWZ3X9ihLX/3pR+2/zRhNO50+c8/Rlo8nTk0vTzLABZJupJHmDPDuiFjTjzeXtBj4NDALuDgilk15fCfgEuAQGktJnxARD2WPnUGjW28L8P6IWJnnNa03rSrBupz9laW1NrXim/P82Tz9q81sejae26cun3lZOCin1TbQSHpRRDwlaTfgoexv8rHdIuLxXt5Y0izgM8AbgXXAbZJWRMQ9TbudDDwREftKOhH4BHCCpIXAicABwEuA6yW9LHtOp9e0HtT57K9MrbWpFZ8z/azKpmvR/BPwZhpznEXTdmX3/3uP730osDYifgog6VLgGKA5KBwDnJXdvgK4UJKy7ZdGxK+BByWtzV6PHK9pParr2V+ZW2v9/swduGyQ2gaaiHhz9u+CRO89CjzcdH8d8Jp2+0TEZklPArtn22+e8tzJX0mn17SEqlyB1bm11swXKNqg5Vkm4HsR8fpO26pG0inAKQDz5s0ruDT1UIcKrK6ttWZlSXqw4dE2vVnSztn4zB6SdpW0W/Y3n62th15MAPs03d8729ZyH0k7ALvQSApo99w8rwlARFwUEWMRMTZ37tweDsMm1f0am7ooS9KDDY/pWjT/C/gLGoPtq9l67cxTwIV9eO/bgP0kLaARDE4E/njKPiuAk4AfAscDqyIiJK0A/knSBVn59gNuzcrY6TUtEVdg1dDPpIcqd5Xa4Ew3RvNp4NOS3hcRf9/vN87GXE4DVtJIRf6HiLhb0jnAeESsoHG9zleywf7HaQQOsv0upzHIvxk4NSK2ALR6zX6X3VorU9aWbTU1GBz+8rl8c/VEz0kPdegqtcFQRHTeSXoFsBDYeXJbRFySsFwDNTY2FuPj40UXo/KmVjzQqMB8RXpx2n0nxx0yyg33re+pJbJo2aqWJxajc0a4aekRPZfdyk/S6ogY67RfnmSAjwOvoxForqWxbMAPaFxIafacYcnaqpJ242Y33Le+52DgrlLLK88UNMcDrwTWRMS7Je0JfDVtsayqhiFrq0pSBoNBdJV6DKge8kyquTEingU2S3oR8CjbZnaZ9VUZZlCui5QzE6eejLIsk5xa7/IEmnFJc4Av0Mg++xGNLDCzvhu2yiV1UE0ZDFLPEO10+frIM6nme7Obn5f0XeBFEfHjtMWyYTVMFxMOImsr9bhZyq5SjwHVR66ZAYC/jYhrm2ZOvigiTkldOKu3Vv3vw1S5DCqoVnXcrC7p8h5nytd1tgD4cJZ9NqljOpvZdNp1ke0yMrvl/lWrXPIYpqDajTosSDZsXcHt5Ak0G4DXA3tK+pakXRKXyYZAu7N5icpXLnl5CeHp1WGVUI8zNeRJb1ZEbAbeK+ldNK6h2TVpqaz22p21b3hmE5864aCh6Goo87IEZVHVbr9JbrU25Ak0n5+8ERFfknQncGq6ItkwmK7/veqVS16+wDWNMo2J1GWcqVcdV9gEvpHN4jzpQeCvkpfMas1n8w3DElS7NdOgUbb51/z/vGEmK2yq6bF+rLBpQ8xn89ZJN0GjbOnx/n/ekGtSzbrzpJpWtDJ195RFN5N2Llh6Da1qNAEPLvuD/hbQck+q2THrLLuOpuM2M+uOU2Bb62Yg3Zl85VTkCptmhlNg2+kmaNTh2ps6KnKFTbPK60eXl1NgW+tmIN1jIuVU2AqbZlXXrwwnp8C21m3QcCZf+eSZVPPvJf0uML95/zqtsGnWjX5lODkFtj0HjXrIM6nmV4D/AdwOTP4SAq+waUOuX11e7u6xusszM8AYsDCcB2011e04Sz+7vHzmbnWWZ1LNu4AXpy6IWRF6SS12hpNZPnlaNHsA90i6Ffj15MaIODpZqcwGpJdxFnd5meWTJ9CclboQZkXpdZzFXV5mneXJOvuXQRTErAhOLTZLL88UNIdJuk3S05J+I2mLpKcGUThrbfmaCRYtW8WCpdewaNmqoZ+qpBceZzFLL0/X2YXAicA3aGSgvRN4WcpCWXtlmwa96jzOYpZenkBDRKyVNCsitgD/KGkNcEbaolkrZZsGvQ48zmKWVp5A84ykHYHbJX0SeIR8adGWgOfFMrOqyRNo/gSYBZwG/CWwD3BcykJZex68tiJ4vRzrRceWSUT8LCI2RsRTEXF2RHwwItYOonC2PQ9e26B5vRzrVZ65zh6E7Retiwgv5VwAD14XY5jP6D0uaL3KO9fZpJ2BtwC7pSmO5eHB68Ea9kw/jwtar/J0nT3W9DcREX8HePHtAfJ1M8Ua9hUwvTyy9SpP19mrmu4+j0YLJ1datPVu2M+my2DYz+i9Xo71Kk/A+Num25uBh4C3JimNbaeK/eN1G88Y9kw/jwtar/LMdXb4IApirVXtbLqOLTCf0Xtc0HqTp+vsg9M9HhEX9K84NlXVzqar2ALrxGf0Zr3Jm3X2amBFdv8PgVuBB7p9U0m7AZcB88m64iLiiRb7nQR8JLv71xHx5Wz7IcCXgBHgWuADERGSzgLeA6zPnvN/IuLabstZBlU7m65aCywvn9GbdS/PVDJ7A6+KiA9FxIeAQ4B52cWbZ3f5vkuB70XEfsD3svvbyILRx4HXAIcCH5e0a/bw52gElP2yv8VNT/1URByU/VU6yECjgjvv2AMZnTOCgNE5I5x37IGlrfScoWRmU+Vp0ewJ/Kbp/m+ybb04BnhddvvLwI3Ah6fscxRwXUQ8DiDpOmCxpBuBF0XEzdn2S4AlwHd6LFNpVelsumotMLNe1C3xJZU8geYS4FZJV2X3l9DoturFnhHxSHb7P2kduEaBh5vur8u2jWa3p26fdJqkdwLjwIdadclZOh7PSMeVWrnUMfEllTxZZ38j6TvA72Wb3h0Razo9T9L1wItbPHTmlNcPSdtNcdOlzwHn0pgy51waqdl/2qZ8pwCnAMybN69Pb29QrRZYVbhSK586Jr6kknc9mh8BP5rJC0fEG9o9JunnkvaKiEck7QU82mK3CbZ2r0FjrOjGbPveU7ZPZO/586b3+ALw7WnKdxFwEcDY2Fi/Ap1ZEq7UyqeuiS8pFLWuzArgpOz2ScDVLfZZCRwpadcsCeBIYGXW5fZUtsS0aKz4eTVAFrQm/RFwV6oDMBukQVZqnvIoHye+5FdUoFkGvFHSA8AbsvtIGpN0MUCWBHAucFv2d85kYgDwXuBiYC3wE7YmAnxS0p2SfgwcTmP9HLPKG1Sl5iUB8vOSHfkpwr1GY2NjMT4+XnQxhpIHuPOZOkYDjUqt36nui5atanmB8OicEW5aekTf3qcuhv3/r6TVETHWaT9PjmmFqcIAd1kqkkFl83ncYWac+JKPA40VpuwD3GULhIOo1Ko25ZFVQ1FjNGalP3sexnVoPO5gKTjQWGHKnrVT9kCYQtWmPLJqcNeZFabs09UMazeSxx2s39yiscKU/ezZ3Uhm/eEWjRWqzGfPnretP8qSuWfFcaAxm8bUQDh51bwrzXzKlrlnxXDXmVlOvmp+5oYxc8+25xaNWU5lv+4npW67v4Yxc8+25xaNWU7DWmn20pIrewq7DYYDjVlOZa80U8263Ev3lzP3DBxorA1PFb+9MleaKcePemnJlT2F3QbDYzS2nWHLFMo7/lDmdOeU40e9Xrha5hR2GwwHGtvOMA16zzSolrXSTDl+VPYZHKz83HVm2xmmQe+6pN+mHD9y95f1yi0a284wzfFVl6CautVR1pacVYNbNLadMg9691vZM8nycqvDyswtGttOmQa9U8+TlaIlUNTcXm51WFk50FhLZai0BpH91u+gWueMPU+Oad1SRBRdhsKNjY3F+Ph40cWwKRYtW9VyrGh0zgg3LT2igBJ1VsUy5zE1gEKj5efuueEmaXVEjHXaz2M0VlpVHKivYpnzqEt2nhXDgcZKq4oD9VUscx51DaA2GA40VlpVzH6rYpnzqGsAtcFwMoCV1nQD9e0GposesC5Txl4/eXYA64WTAXAyQNW0G5g+7pBRvrl6YruxhDkjsznr6AMqX9kXreggbuWTNxnAgQYHmqppl9k1S2JLm//PzpAy6z9nnVlttRuAbhdkwBlSZkVyoLHKaTcAPUua9nnOkDIrhgONVU67zK63vWaf7bY3c4aUWTGcdWaVM11m19hLd+Psb93NE89s2uY5zpAyK46TAXAyQB1VIUOqCmU0m07eZAC3aKyWyjAp6HTqPPmm2VQONGYF6LRctls7VicONFZbZa6sp5s7rNvWTpmP14abs86sliYr64kNGwm2VtbL10wUXTRg+rnDupkp+SPL7+QvL7u9tMdrw82Bxmqp7NPaTzf55kxnSl6+ZoKv3fzvTE3rKdPx2nArJNBI2k3SdZIeyP7dtc1+J2X7PCDppKbtfyPpYUlPT9l/J0mXSVor6RZJ89MeiZVV2ae1X3LwKOcdeyCjc0YQjYXRJqfImelMyeevvH+7IDOp6ONdvmaCRctWsWDpNSxatsotrCFV1BjNUuB7EbFM0tLs/oebd5C0G/BxYAwIYLWkFRHxBPAt4ELggSmvezLwRETsK+lE4BPACWkPxcroJXNGWs6HVqaLNttlxs10puTpgkmRx+vMOptUVNfZMcCXs9tfBpa02Oco4LqIeDwLLtcBiwEi4uaIeKTD614BvF7qMC+J1VKV14WZrrXTSrtgIij0eMvefWmDU1SLZs+mQPGfwJ4t9hkFHm66vy7bNp3nnhMRmyU9CewO/GLqjpJOAU4BmDdv3owKb+VX9XVhZnIdUKsWkIC3Hzav0OMte/elDU6yQCPpeuDFLR46s/lORISkgU9PEBEXARdBY2aAQb+/pVf2izb7paxBtQrdlzYYyQJNRLyh3WOSfi5pr4h4RNJewKMtdpsAXtd0f2/gxg5vOwHsA6yTtAOwC/DYTMptVkVlDKpeldMmFTVGswKYzCI7Cbi6xT4rgSMl7ZplpR2Zbcv7uscDq8KTuZkVYqZjTVZfRY3RLAMul3Qy8DPgrQCSxoA/j4g/i4jHJZ0L3JY955yIeDzb75PAHwPPl7QOuDgizgK+CHxF0lrgceDEQR6UmW2rjC0tGzzP3oxnbzYz64aXcjYzs1JwoDEzs6QcaMzMLCkHGjMzS8qBxszMknKgMTOzpBxozMwsKQcaMzNLyoHGzMyScqAxM7OkHGjMzCwpBxozM0vKgcbMzJLy7M2ApPU0lisooz1osRR1TflY62lYjnVYjhO2HutLI2Jup50daEpO0nieabjrwMdaT8NyrMNynDDzY3XXmZmZJeVAY2ZmSTnQlN9FRRdggHys9TQsxzosxwkzPFaP0ZiZWVJu0ZiZWVIONBUg6XxJ90n6saSrJM0pukypSHqLpLslPSupdhk8khZLul/SWklLiy5PKpL+QdKjku4quiypSdpH0g2S7sn+736g6DKlImlnSbdKuiM71rPzPM+BphquA14REb8N/D/gjILLk9JdwLHA94suSL9JmgV8BngTsBB4m6SFxZYqmS8Bi4suxIBsBj4UEQuBw4BTa/y9/ho4IiJeCRwELJZ0WKcnOdBUQET8c0Rszu7eDOxdZHlSioh7I+L+osuRyKHA2oj4aUT8BrgUOKbgMiUREd8HHi+6HIMQEY9ExI+y2/8F3AuMFluqNKLh6ezu7Oyv40C/A031/CnwnaILYV0ZBR5uur+OmlZIw0rSfOBg4JZiS5KOpFmSbgceBa6LiI7HukP6Ylkekq4HXtzioTMj4upsnzNpNNO/Nsiy9VueYzWrGkn/Dfgm8BcR8VTR5UklIrYAB2VjxVdJekVETDsW50BTEhHxhukel/Qu4M3A66PiOemdjrXGJoB9mu7vnW2zipM0m0aQ+VpEXFl0eQYhIjZIuoHGWNy0gcZdZxUgaTHwv4GjI+KZostjXbsN2E/SAkk7AicCKwouk/VIkoAvAvdGxAVFlyclSXMns14ljQBvBO7r9DwHmmq4EHghcJ2k2yV9vugCpSLpjyStA34HuEbSyqLL1C9ZQsdpwEoaA8aXR8TdxZYqDUlfB34I7C9pnaSTiy5TQouAPwGOyH6ft0v6n0UXKpG9gBsk/ZjGidN1EfHtTk/yzABmZpaUWzRmZpaUA42ZmSXlQGNmZkk50JiZWVIONGZmlpQDjVmBJH1J0vFleR2zFBxozPpADf49mbXgH4ZZlyTNz9aWuYTGFBz7SDpS0g8l/UjSN7L5r5D0MUm3SbpL0kXZ1eTtXvflkm6d8j535n0dSQ9J2iO7PSbpxuz2C7J1Ym6VtEZSLWeOtvJxoDHrzX7AZyPiAOCXwEeAN0TEq4Bx4IPZfhdGxKsj4hXACI1561qKiPuAHSUtyDadAFw209dp4UxgVUQcChwOnC/pBTN4vllXHGjMevOziLg5u30YjQXNbsqmUT8JeGn22OGSbslaJkcAB3R43ctpBBjYNtDM9HWaHQkszcp2I7AzMG8GzzfrimdvNuvNL5tui8bcT29r3kHSzsBngbGIeFjSWTQq+elcBnxD0pU01pt6YAavs5mtJ5HNjws4rsYLy1lJuUVj1j83A4sk7QvPjYm8jK1BNtMZAAAApklEQVSV/S+yMZuO2WER8RNgC/BRtrZm8r7OQ8Ah2e3jmravBN43Oa4j6eA8B2XWKwcasz6JiPXAu4CvZ7Pb/hB4eURsAL5AI2FgJY1Zb/O4DHgHjW40ZvA6ZwOfljROI1hNOpfG0rs/lnR3dt8sOc/ebGZmSblFY2ZmSTnQmJlZUg40ZmaWlAONmZkl5UBjZmZJOdCYmVlSDjRmZpaUA42ZmSX1/wFrB3Shbxqs0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "error = np.reshape(x-x_dequant, (-1))\n",
    "print(\"error\", np.min(error), np.max(error))\n",
    "fig, ax = plt.subplots()\n",
    "ax.scatter(x, error)\n",
    "ax.set_ylim([1.2*min(error), 1.2*max(error)])\n",
    "ax.set_xlabel(\"real value\")\n",
    "ax.set_ylabel(\"quantization error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error magnitude depends on the real values' range, distribution and quantization bit numbers. Since we don't realy control over the neural network's input and weight distribution, we'll examine the impact of quantization bitwidth on the error. As shown in the graph below, the error increase almost exponentially as the bitwidth decreases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'bitwidth')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEPCAYAAABcA4N7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8XHW9//HXJ3u6JKVtuqXpSimUgpSGslkqcLFsAopQNgVkq6JXRVEQr5fr9V70ov7UK7IVBAUslU2uAhWVVaA0bYG2QKEbbdMVShfapNk+vz/OSZmmWU7SmZxM5v18POaRmXPOzPkMmn7y3T5fc3dERERakhV3ACIi0rUpUYiISKuUKEREpFVKFCIi0iolChERaZUShYiItEqJQkREWqVEISIirVKiEBGRVuXEHUAy9O/f30eMGBF3GCIiaWXevHnvu3tJW9d1i0QxYsQIKioq4g5DRCStmNl7Ua5T15OIiLRKiUJERFqlRCEiIq1SohARkVYpUYiISKu6xaynjnhsQSU3z17C2i1VDOlTyLVTx3LWhNK4wxIR6XIyMlE8tqCS6x9ZSFVtPQCVW6q4/pGFAEoWIiJNZGTX082zl+xOEo2qauu5efaSmCISEem6MjJRrN1S1a7jIiKZLCMTxZA+he06LiKSyTIyUVw7dSyFudl7HMvPyeLaqWNjikhEpOvKyMHsxgHrxllPAIeUFmkgW0SkGRmZKCBIFo2J4Ud/fpPfvrSS1Zt3Uta3R8yRiYh0LRnZ9dTUZZNHkmVw14sr4g5FRKTLUaIABhcXcuZhpcycu4rNO2riDkdEpEtRoghNnzKK6toG7nlpZdyhiIh0KUoUof0H9OakcQP53csr2VlTF3c4IiJdhhJFgulTRrNlZy0zX10ddygiIl2GEkWCicP3Y9LIvsx4YTm19Q1xhyMi0iUoUTTx5SmjWbu1msdfWxt3KCIiXYISRROfGlvCgYN6c/vzy2ho8LjDERGJnRJFE2bG9CmjeWfDR/zj7Y1xhyMiEjslimacfuhgSvsUcttzy+IORUQkdkoUzcjJzuKKySOpeO9D5q7cHHc4IiKxUqJowbQjhtG3Zx63PatWhYhkNiWKFhTmZXPx0SP4+9sbWbJ+e9zhiIjERomiFV88ejiFudnc/rxaFSKSuTo9UZjZyWa2xMyWmtl1rVx3tpm5mZV3ZnyJ9uuZx/mThvH4a2up1DapIpKhOjVRmFk2cAtwCjAOON/MxjVzXW/g68CczoyvOZdPHgnAjBeWxxyJiEg8OrtFMQlY6u7L3b0GmAmc2cx1/wn8BKjuzOCaM6RPIWccNoSZr67mQ5UgF5EM1NmJohRIrLi3Jjy2m5kdDpS5+186M7DWTJ8ymqraeu59eWXcoYiIdLo2E4WZ5ZvZhWY2JtXBmFkW8HPgWxGuvdLMKsysYtOmTSmN64CBvfmXgwZw70sqQS4imafNROHuu4AZwJAk3K8SKEt4PTQ81qg3MB541sxWAkcBjzc3oO3ud7h7ubuXl5SUJCG01k2fMpoPd9Yya65KkItIZona9bQQOCAJ95sLjDGzkWaWB5wHPN540t23unt/dx/h7iOAV4Az3L0iCffeJ+Uj+nLEiP2484UVKkEuIhklaqL4JvAdMzvdzHI6ejN3rwO+CswG3gJmuftiM/uhmZ3R0c/tLNOnjKZySxV/fkMlyEUkc5h726W0zWwT0AMoABz4MPy5m7sPSEWAUZSXl3tFReobHQ0Nzsm/fB7DeOobkzGzlN9TRCRVzGyeu7e5Vi1q6+AWmiSGTJSVFZQgv2bW6zyzZCMnHDgw7pBERFIuUqJw9xtTHEfa+MwnhvCzv77Dbc8uV6IQkYzQrnUUZpZnZhPN7KTwZ16qAuuqcrOzuOyTI3l15WbmvacS5CLS/UVOFGb2HWAD8CrBYPRcYIOZXZui2Lqs8yaV0adHLrc+q7IeItL9RUoUZvYN4CbgAeB44CDgU+Hrm8zsX1MVYFfUIy+Hi48ewd/e2sC7G1SCXES6t6gtiquBH7v71e7+vLsvCX9eTVCTKaMSBcDFx4ygIDeL259Xq0JEureoiaIMeKaFc88SrLDOKH175nHeEcN4bEEla1WCXES6saiJYhXw6RbOnRSezziXTx6JA3e9uCLuUEREUiZqovgV8G0zmxFuPDTBzKaa2QzgGuCXqQux6xq6Xw/O+MQQ/vDqKrbsVAlyEemeIiUKd/81cBVwMvAEUAE8Gb6e7u63pCzCLu6qKaPYWVPP715+L+5QRERSIvL0WHe/k2CsYjhwdPizzN1npCi2tHDgoCJOOHAA97y0kqqa+rjDERFJuij7URSY2TtmdrIHVrv7q+HPjC/rAUGxwM07avjjPJUgF5HuJ8p+FNVAH0C1tVtwxIj9mDh8P+54fjl1KkEuIt1M1K6n+4FLUxlIOjMLigWu+bCKvyxcF3c4IiJJFbV67CrgXDObSzCIvYE9q8m6u9+a7ODSyYkHDmDMgF7c+uwyzvjEEJUgF5FuI2qi+Fn4czAwsZnzDmR0osjKMq6aMppv//F1nn1nE8ePjW17DhGRpIo6PTarjUd2qgNNB2d8YgiDiwu47dllcYciIpI0UWc93WlmR3VGQOksLycoQT5nxWbmr/ow7nBERJIi6qyn8wi2QZU2nD9pGMWFuWpViEi3EXXW0z8IyotLG3rm53Dx0cN5+q0NLN34UdzhiIjss6iJ4hbgUjP7qZmdYGYHm9m4xEcqg0w3Fx8zgvycLO54Xq0KEUl/URPFUwSlxK8B/ga8ASwMH4vCnxLq1yufaeVlPLqgknVbVYJcRNJb1Omx6nZqp8snj+K+Oau4+8UV3HCaGlwikr4iJQp3fy7VgXQ3ZX17cPqhg3lgziq+evwYinvkxh2SiEiHRK4eC2Bmp5jZv5nZHWY2LDx2nJkNSU146W36lNHsqKnn96+sjDsUEZEOi5QozGygmc0B/g+4GLgM6B+evhT4t9SEl94OGlzEp8aW8Nt/rqS6ViXIRSQ9RW1R/C/QCzgwfCQWMvobcGKS4+o2pk8ZzQc7avjjvDVxhyIi0iFRE8XJwPfdfSl7FgMEWAOUJjWqbuTIkX2ZMKwPdzy/TCXIRSQttWeMoq6F4/0BzQFtQWMJ8tWbq3hi0fq4wxERabeoieIF4F/NLLH4X2PL4ksEK7elBScdNJDRJT257dllaFNAEUk3URPFd4EjCBbX/SdBkrjCzJ4j2D/7+6kJr3toLEH+5rptPP/u+3GHIyLSLlHLjC8i2IeiArgEqAc+RzA+caS7v5OqALuLsw4rZVCRSpCLSPqJPEbh7svc/QvuPsTd89x9kLtf6O7vpjLA7qKxBPnLyz/gtdVb4g5HRCSydi24k31z/pHDKCrIUatCRNKKEkUn6pWfwxePHsHsN9ezbJNKkItIeuj0RGFmJ5vZEjNbambXNXN+upktNLPXzOzF7lbC/JJjR5CXncWdzy+POxQRkUg6NVGE02tvAU4BxgHnN5MIHnD3Q9z9MOB/gJ93Zoyp1r9XPueWl/HI/Eo2bKuOOxwRkTZ1dotiErDU3Ze7ew0wEzgz8QJ335bwsid7rwRPe1dMHkVdQwN3v7gi7lBERNrU7kRhZtlm1qPpI+LbS4HVCa+bLf9hZleb2TKCFsW/tjfGrm5Yvx6cdugQ7p+ziq1VtXGHIyLSqqjVY4vM7NdmthbYBWxv5pE07n6Lu48mWOjX7GI+M7vSzCrMrGLTpk3JvH2nmD5lFB/tquO+V96LOxQRkVZF3eHuduB0YAbwJlDTwftVAmUJr4eGx1oyE7i1uRPufgdwB0B5eXnadU8dPKSY4w4ISpBf9smRFORmt/0mEZEYRE0UU4FvuvuMfbzfXGCMmY0kSBDnARckXmBmYxIW8Z0GdNsFfdOnjOKCO+fw8Pw1XHjk8LjDERFpVtQxih0E4wn7xN3rgK8Cs4G3gFnuvtjMfmhmZ4SXfdXMFpvZa8A1BBsldUtHj+rHJ8r6cMfzy6lvSLtGkYhkiKgtip8BXzGzv7r7Pm2q4O5PAE80OfaDhOdf35fPTydmxpenjGL6ffN5ctE6Tj9UO8qKSNcTNVGUAp8AlpjZM0DTYkXu7t9NamQZ4qRxgxjVvye3PbeM0w4ZjJm1/SYRkU4UNVF8HmgIrz+pmfNOMENJ2ik7y7hqyii++/BCXlz6PpPHlMQdkojIHqKWGR/ZxmNUqgPtzs6aUMrAonxue07FAkWk61FRwC4gPyebLx07kn8u/YA31qgEuYh0LVG7ngAws08CBwAFTc+5+2+SFVQmuuDIYfz6maXc9twyfnPhxLjDERHZLVKiMLOBwN8JCvk50DjimjinU4liH/QuyOULRw3n1ueWseL9HYzs3zPukEREgOhdTz8DthKsqjbgSGAE8G8EC+IOSEVwmebSY0eSm53FHSpBLiJdSNREMYUgWawLX5u7r3L3/wbuQ62JpCjpnc85E4fy8Lw1bFQJchHpIqImij7ApnCx3TZgQMK5l4Bjkh1YprryuLAE+T9Xxh2KiAgQPVGsAAaHzxcDFyac+wywOZlBZbLh/XpyyiGDuf+V99hWrRLkIhK/qIniL8Cnw+c/As42szVmtoJgv4j/TUVwmerLU0azfVcd97+yKu5QRESizXpy9+sTnj9pZscCZwGFwNPu/mSK4stI40uLmTymP3f/cwWXHjtCJchFJFYdWnDn7nPd/QZ3v0ZJIjWmTxnNpu27eHRBa9t1iIikXuREYWb5ZvZlM7vLzP5qZmPC49PM7KDUhZiZjhndj0OHFnP7c8tUglxEYhV1K9QDgHeAmwjWT5wI9A5PTwaub/6d0lFmxvQpo1n5wU5mL14fdzgiksGitih+BawiSBJT+XhlNsBzwCeTG5YATD14ECPDEuTualWISDyiJorJwE3uvoU9y3YAbODjqbOSRNlZxpXHjeKNNVt5adkHcYcjIhkqaqKoJpjh1JxS9t7ISJLksxNKKemtEuQiEp+oieJp4HtmVpxwzM0sH/gaTbY2leQpyA1KkL/w7vssqtwadzgikoGiJoprgRJgKfB7gu6nHwALgSHADSmJTgC48Khh9M7P4Va1KkQkBlF3uFtNsGf2bQQD2ssIxiX+CEx0d03LSaGiglwuPGo4Ty5cx3sf7Ig7HBHJMJHXUbj7h+7+b+5+jLsf4O5HhYvuNMraCb507AhyslSCXEQ6n7ZCTRMDigo4e+JQ/jhvDRu3qwS5iHSeqAvu8szsO2b2spmtMrONTR+pDlSCEuS19Q3coxLkItKJou6ZfStBafE/Af8AalIWkbRoZP+enDJ+EL9/5T2+/KnR9C7IjTskEckAURPF54BvuPttqQxG2jZ9ymieWLieB+as4qopo+MOR0QyQNQxis0EJTwkZocO7cOx+/fjrhdXsKuuPu5wRCQDRE0UPwS+ZWY9UxmMRDN9ymg2bt/FYypBLiKdIOrGRfea2ThglZnNY++SHe7u05IenTTrk/v3Z3xpEbc/t5zPTywjO8vafpOISAdFnfX0LYLV2buAngSrtBMfA1IVoOytsQT58vd38PSbWusoIqkVdTD7OoJS49901bvuEk4ZP5jh/ZZw63PLmXrwIMzUqhCR1Ig6RmHAn5Ukuo7GEuSvr97Cy8u1OF5EUidqorgHODuFcUgHnH34UPr3yue251TWQ0RSJ2rX0xrgGjP7G8GCu+YGs29NamTSpoLcbC49dgQ3z17C4rVbOXhIcdtvEhFpJ4vSm2RmDW1c4u6enZyQ2q+8vNwrKiriun2stlbVMulHT5OVlUV1bT1D+hRy7dSxnDWhNO7QRKSLM7N57l7e1nVRp8eqeGAX9czbG6l32FUbLL6r3FLF9Y8sBFCyEJGk6PQEYGYnm9kSM1tqZtc1c/4aM3vTzN4ws7+b2fDOjjGd3Dx7CXUNe7YKq2rruXn2kpgiEpHuplMThZllA7cApwDjgPPDhXyJFgDl7n4o8BDwP50ZY7pZu6WqXcdFRNqrs1sUk4Cl7r7c3WuAmcCZiRe4+zPuvjN8+QowtJNjTCtD+hS267iISHt1dqIoBVYnvF4THmvJZcCTzZ0wsyvNrMLMKjZt2pTEENPLtVPHUpi79zyCI0bsF0M0ItIdddlBajO7CCgHbm7uvLvf4e7l7l5eUlLSucF1IWdNKOWmzx1CaZ9CDBjSp4DDhhbz2GtrufellXGHJyLdQNR1FLtZUCtiMLDR3eva+fZKoCzh9dDwWNN7/AtwAzDF3Xe1N8ZMc9aE0j1mONXWN/CV++fz748vJjc7iwuOHBZjdCKS7iK3KMzsVDObA1QT7E1xaHj8jvCv/yjmAmPMbKSZ5QHnAY83uc8E4HbgDHfXFqsdkJudxa8vmMDxY0u44bGF/LFiddtvEhFpQdTqsV8k+Af9beDKJu97l2AsoU1hC+SrwGzgLWCWuy82sx+a2RnhZTcDvYA/mtlrZvZ4Cx8nrcjPyebWiybyyf37852H3+BPr2nvChHpmKgrs5cAj7j79eEU11qCKazzzexU4LfuPjDFsbYok1dmt6Wqpp4v3TOXV1du5lfnTeC0QwfHHZKIdBFRV2ZH7XoaDjzdwrlqoChqYNK5CvOymXFxOYcP68PXZy7gr4u1f4WItE/URLEamNDCuXJgaXLCkVTomZ/D3ZccwfjSYq5+YD7PvK2hHxGJLmqiuAv493DQunEll5nZicB3gDtTEZwkT++CXO790iQOHFTEVffN4/l3MnftiYi0T9RE8RPg98C9wObw2EsEg9IPuvuvUhCbJFlxYS6/v2wSo0t6ccXvKnhp2ftxhyQiaSBSovDA1cABBLOWvg98HRgXHpc00adHHvddNonh/Xpw2T0VzF25ue03iUhGizo99jgz6+Xuy8IV0f/t7re5+ztm1tPMjkt1oJI8/Xrlc9/lRzK4TwGX/nYu81d9GHdIItKFRe16eoag2mtzDgzPSxoZ0LuABy4/in698rj47ldZuGZr3CGJSBcVNVFYK+d6ATtbOS9d1KDiAh644iiKC3O56K45vLl2W9whiUgX1GKtp7A76VMJhy43s5ObXFYAnAYsTH5o0hlK+xTyhyuOYtrtL3PRXXP4wxVHMXZQ77jDEpEupLWigEcCXwufO3AO0LQIYA1BWY9rkx+adJayvj144IqjOPf2l7lwxhxmXnkU+w/oFXdYItJFtNj15O43u3uJu5cQFAE8vvF1wqPU3U909/mdF7Kkwoj+PXngiqMAuODOV1j5/o6YIxKRriLq9NiR7v5aqoOReO0/oBf3X34kdQ3OBXe+wurNGnoSkehFAb/S1jXu/pukRNQBKgqYXG+u3cb5d75C74IcHrzqaEq1rapItxS1KGDURNHQymkHcPe99+PsJEoUybdwzVYumPEKfXvm8eCVRzOouCDukEQkyZJaPdbds5o+gL7A+cDrtLzGQtLUIUOLufdLk3h/+y4umPEKG7dXxx2SiMSkw3tmu/sWd38QuI1gRzrpZg4fth/3fGkS67ZUc9GMOXzwkXalFclEHU4UCVYQlBqXbuiIEX2565Jy3vtgJxfd9SpbdtbEHZKIdLJ9ShRmNhj4FkGykG7qmNH9ufOL5Szb9BFfuOtVtlbVxh2SiHSiqEUBN5nZxiaPLcAaYDLw7ZRGKbE77oASbr9oIm+v38bFd7/K9molC5FM0drK7ES3EM5uSlBNkCiecvcPkhqVdEnHHziAX19wOFffP58v3TOXey6dRM/8qP8XEpF0FWl6bFen6bGd6y9vrONrf5jPpJF9+e0lkyjMi21mtIjsg6ROjxVJdNqhg/l/0w5jzorNXPn7Cqpr6+MOSURSqLXqsXPZu7upRe4+KSkRSVo487BSauoauPahN/jyffO47QsTyc9Ry0KkO2qtg3kx7UgUknnOKS+jtt753qML+eoDC/jNhYeTm61Gqkh302KicPdLOjEOSVMXHDmM2voG/v3xxXxj5mv88rzDyFGyEOlW2j1lxcz6EZTv2KzZTgJw8TEjqK1v4Ed/eYucbOPn5x5GdlZrmyKKSDqJ/KefmU0zs7eAjQSbFW00s7fM7JyURSdp4/LJo/jOyWP502tr+e7Db9DQoF5Lke4iUovCzM4H7geeBG4CNgADgWnATDPLdveZKYtS0sJXPrU/NXUN/OJv75KbncV/f3Y8ZmpZiKS7qF1PNwB3uPv0Jsd/Z2a3Ad8HlCiEr584htr6Bm55Zhl52caNZxysZCGS5qImiv2Bb7Zw7mHgkqREI2nPzPj2p8dSU9fAnS+sIDc7ixtOO0jJQiSNRU0UGwgqxD7dzLny8LwIECSL7516ELX1zowXV5Cbk8V3po5VshBJU1ETxW+BG80sG3iIIDEMAM4h6Ha6KTXhSboyM/79M+OoqW/g1meXkZedxTdPOiDusESkA6Imih8CucB1wH8kHK8CfhqeF9mDmfGjM8dTW9fAL//+Lnk5WVx9/P5xhyUi7RQpUbh7A3CDmf0UGA8MBtYBi9z9wxTGJ2kuK8v48dmHUtfg3Dx7CXnZWVxx3Ki4wxKRdmjXgrswKbyQolikm8rOMm7+/KHU1DXwX0+8RW62ccmxI+MOS0Qiirpx0dlmdlnC65Fm9pKZbTGzh82sT9QbmtnJZrbEzJaa2XXNnD/OzOabWZ2ZfT7q50rXlpOdxS/OO4xPjxvIjf/3JvfPeS/ukEQkoqgrs78PFCW8/l+gP/Bj4HDgv6J8SDgYfgtwCjAOON/MxjW5bBXBdNsHIsYmaSI3O4tfX3A4Jxw4gBseXcSsuavjDklEIoiaKEYBCwHMrBj4NPBNd/8xwWK8z0T8nEnAUndf7u41BIv0zky8wN1XuvsbQEPEz5Q0kpeTxW8uPJzJY/rz3Ufe4NEFa+IOSUTa0J4yn43Fe6YA9cDfwtdrgJKIn1EKJP4ZuSY8JhmkIDebO79YzlEj+/GtWa/z5zfWxh2SiLQiaqJ4HbjQzHoClwPPuPuu8NwwgkKBncrMrjSzCjOr2LRpU2ffXvZRQW42d11STvnwvnx95ms8tWh93CGJSAuiJorvAZ8FthG0KG5MOHcWMCfi51QCZQmvh4bH2s3d73D3cncvLymJ2qCRrqRHXg53X3oEhw4t5mt/mM/f39ICf5GuKOo6ihfNbBhwALDM3bcknL4bWBrxfnOBMWY2kiBBnAdc0I54pZvplZ/DPZdO4gt3zeHL983n0k+O4M+vr2PtliqG9Cnk2qljOWuCeidF4mTunbtvgJmdCvwCyAbudvf/MrMfAhXu/riZHQE8CuwHVAPr3f3g1j6zvLzcKyoqUh26pNCWnTWc+ssXWLu1eo/jhbnZ3PS5Q5QsRFLAzOa5e3lb10VecGdmhwDXE8xcalyZPQf4SThLKRJ3fwJ4osmxHyQ8n0vQJSUZpE+PvGY3aK+qrefm2UuUKERiFHXjorOAWcAygqKAGwmKAp4JVJjZue7+WMqilIywvklrotHaLVW4u6rPisQk6mD2T4A/AePc/Tp3/7m7X0ewaO7x8LzIPhnSp7DZ4w6c+PPnuP25ZWzavqvZa0QkdaImijJghjcZ0Ahf38meM5lEOuTaqWMpzM3e41hBbhbnTSqjb488bnrybY6+6e9c9fsK/vH2BurqtSZTpDNEHaOoAA4GZjdzbjwwP2kRScZqHIe4efaSZmc9Ld34EbMqVvPwvDXMXryBgUX5nDOxjHPLyxjWr0ecoYt0ay3OejKzxN+8cQTlNu4EHuPjMYrPEizAO8/dY5t2pFlPmaWmroF/vL2BB+eu5rl3NtHgcPSofpw3qYypBw+ioEmrRESaF3XWU2uJogH2mIjSOJLY7DF3j+23U4kic63bWsVDFWuYNW81qzdXUVSQw1kTSpl2RBkHDymOOzyRLi0ZieISaHbGYrPc/d7I0SWZEoU0NDgvL/+AB+eu5qnF66mpa2B8aRHTyss447BSigtz4w5RpMvZ50SRTpQoJNGWnTU8tqCSByvW8Na6beTnZHHqIYOZdkQZR47sq2m2IiElCsl47s6iym3MnLuKx19by/ZddYzo14Nzysv4/MShDCwqiDtEkVglPVGY2TTgCoJ6T3v9hrn7gPYGmSxKFNKWqpp6nli4jgcrVvPqis1kZxnHjy1h2hHDOH5sCTnZ7am4L9I9JLWEh5ldQFD87x7ghPB5FnAGsAX4XYcjFekEhXnZnD1xKGdPHMryTR8xq2IND89fw9/eqqCkdz5nHz6UaUeUMbJ/z7hDFelyIrUozGwBQemOHwO1QLm7zzez3sDTwEPu/tOURtoKtSikI2rrG3jm7Y3MqljNM0s2Ud/gTBrZl2nlZZx6yGAK8zTNVrq3pHY9mdlHwOnu/qyZ1QInufuz4bnPAv/P3UfsW8gdp0Qh+2rDtmoemreGWRWree+DnfTOz+GMw4Yw7YgyDikt1gC4dEvJrh67DcgPn1cCBwHPNt4L6NfeAEW6koFFBVx9/P585VOjmbNiMw/OXc1D89Zw/5xVHDS4iGnlQzlrQil9euTFHapIp4vaovgT8KK732xmvwLOAX4A1IQ/l7v7SSmNtBVqUUgqbK2q5fHX1/Lg3FUsqtxGXk4WJx88iGlHlHH0qH5kZamVIekt2V1PRwHD3f1BM+sD3AucRjCgPRc4392X72PMHaZEIam2qHIrsypW89iCSrZV11HWt5BzJ5bx+fKhDC5uvuqtSFeX8nUUZpYP5Lv7tg59QBIpUUhnqa6tZ/bi9cx8dTUvL/+ALIMpB5Qw7YgyTjhwIHk5mmYr6UML7kRS7L0PdjCrIhjL2LBtF/165nH2xKGcW17GosqtLVbBFekqlChEOkldfQPPv7uJma+u5h9vb6SuwTGDxF8t7f0tXVHS98wWkeblZGdxwoEDOeHAgWzavosTf/Ys26rr9rimqraeG/9vMQcM7M2Ygb3I1UpwSSNKFCJJVNI7n+1NkkSjLTtrOfVXL5CXk8XYgb0ZX1rEuCHFjB9SxEGDi7SPhnRZShQiSTakTyGVW6r2Oj6gdz43nHYQi9duY/HarTyxcD1/eHU1AFkG+w/oxfghxYwbUsT40uBnUYHKo0v8lChEkuzaqWO5/pGFVNXW7z5WmJvN9049iDMPK+XMw4JxCnencksViyq38ebarSxau41x64OpAAAL3ElEQVR/LnufRxZU7n7f8H49OHhIEQcPKd79s6R3/l73FEklJQqRJGtr7+9GZsbQ/XowdL8enDx+0O7jm7bvYvHarbtbHosqt/HEwvW7zw8symd8Y+IoDX6W9ilUmRFJGc16EkkDW6tqeTNMHI0JZOnGj2gIf3379Mjl4CFFe3RdjezXU6vHpVWa9STSjRQX5nL06H4cPfrjsmpVNfW8vX4bi9aGXVeV2/jtP1dSU98AQI+8bMYNLtqj5TFmQG8tCpR2U4tCpBuprW/g3Q0f7dHyWLx2GztrgvGSvOwsDhjUa4+uq4MGFamkeobSgjsRAaChwVn5wQ4Wrd3G4sogcSxau5UtO2uBYMbV6JJejC/9eMB83JAiiguDGVePLajUKvNuSolCRFrk7qzdWs2iMHE0JpD126p3XzOsbw/69MjhzbXbqWv4+N8JrTLvPpQoRKTd3v9oV9DiqNzKm2u38dTi9dQ37P1vRG62ceKBAxlUXMDg4gIGFRcwqKiAwcWFDCjK1+LBNKHBbBFpt/698plyQAlTDigBYOR1f2n2utp6Z+mmj3hx6ft8tGvvleh9e+aFiePjJBIklUIGFeczqLiQXvn65ydd6H8pEWlRS6vMS/sU8rdrpgCwvbqWDduqWbe1mvXhY9228OfWahas3sLmHTV7fUbv/BwGNrZIwqTy8etCBhUXsF+PXK0P6QKUKESkRS2tMr926tjdr3sX5NK7IJf9B/Ru8XOqa+vZuG0X67ZWsT4hiTQmlXc3vM/G7dU07eXKy8nanUgGha2TwbufFzK4uID+vfLJjrheRAPzHaNEISItirrKvC0FudkM69eDYf16tHhNXX0Dmz7a9XGrZGv17qSyfms181d9yIatu3avE2mUnWUM6J2/e7xk4O4ur8LdLZUBRfk8uXD9HkmvcksV1z+ycI/vKc3TYLaIpA13Z/OOGtZtrd6ju+vj11Ws31rNjpr6vd6bZezVYgHolZ/DVceNoiA3m4K8bApysoLnudkU5IbPc7IpzMsiP2fP43GWi09G60iznkQkY22vrt2rVfLzp99J+n2ys4yCnCwK87LDJNIkyexOLInnEo8nXr9nYtrrsxIS02MLKpvtEmzvtOUuO+vJzE4GfglkAzPc/cdNzucDvwMmAh8A09x9ZWfHKSLpq3HcZMzAj8dNHpy7usWB+Wev/RTVtfVU1zaEP8PndfXNHE94XZd4Lry+5uPjW3bW7j63KzxWVVvf7JTjKBoTU1Vt/V6to6raem6evSQl3WidmijMLBu4BTgJWAPMNbPH3f3NhMsuAz509/3N7DzgJ8C0zoxTRLqf1gbmc7OzyM3OondB58RSW9+wR8LZlZBwqpomproGdiUkqKraeu56cUWzn7u2mUSYDJ3dopgELHX35QBmNhM4E0hMFGcCN4bPHwJ+bWbm3aGPTERik6yB+WTY18T01KL1zbaOhvQp3MfImtfZiaIUWJ3weg1wZEvXuHudmW0F+gHvJ15kZlcCVwIMGzYsVfGKSDdy1oTSbjHDKcq05WRK23rD7n6Hu5e7e3lJSUnc4YiIdJqzJpRy0+cOCTasIhhnSWX9rc5uUVQCZQmvh4bHmrtmjZnlAMUEg9oiIhLqzNZRZ7co5gJjzGykmeUB5wGPN7nmceDi8PnngX9ofEJEJD6d2qIIxxy+CswmmB57t7svNrMfAhXu/jhwF/B7M1sKbCZIJiIiEpNOX0fh7k8ATzQ59oOE59XAOZ0dl4iINC9tB7NFRKRzKFGIiEirukWtJzPbBLzXwbf3p8kajTSm79L1dJfvAfouXdW+fJfh7t7m+oJukSj2hZlVRCmKlQ70Xbqe7vI9QN+lq+qM76KuJxERaZUShYiItEqJAu6IO4Ak0nfperrL9wB9l64q5d8l48coRESkdWpRiIhIqzI2UZhZmZk9Y2ZvmtliM/t63DF1lJkVmNmrZvZ6+F3+I+6Y9oWZZZvZAjP7c9yx7AszW2lmC83sNTNL6716zayPmT1kZm+b2VtmdnTcMXWEmY0N//dofGwzs2/EHVdHmNk3w9/3RWb2BzNL2bZLGdv1ZGaDgcHuPt/MegPzgLOa7LaXFszMgJ7u/pGZ5QIvAl9391diDq1DzOwaoBwocvfT446no8xsJVDu7mk/X9/M7gVecPcZYUHPHu6+Je649kW442YlcKS7d3QdVizMrJTg93ycu1eZ2SzgCXe/JxX3y9gWhbuvc/f54fPtwFsEmyalHQ98FL7MDR9p+ReAmQ0FTgNmxB2LBMysGDiOoGAn7l6T7kkidCKwLN2SRIIcoDDcjqEHsDZVN8rYRJHIzEYAE4A58UbScWF3zWvARuBpd0/X7/IL4DtAQ9yBJIEDfzWzeeGOjOlqJLAJ+G3YJTjDzHrGHVQSnAf8Ie4gOsLdK4GfAquAdcBWd/9rqu6X8YnCzHoBDwPfcPdtccfTUe5e7+6HEWwGNcnMxscdU3uZ2enARnefF3csSfJJdz8cOAW42syOizugDsoBDgdudfcJwA7gunhD2jdh99kZwB/jjqUjzGw/4EyCJD4E6GlmF6XqfhmdKML+/IeB+939kbjjSYawS+AZ4OS4Y+mAY4Ezwr79mcAJZnZfvCF1XPhXH+6+EXgUmBRvRB22BliT0Ep9iCBxpLNTgPnuviHuQDroX4AV7r7J3WuBR4BjUnWzjE0U4QDwXcBb7v7zuOPZF2ZWYmZ9wueFwEnA2/FG1X7ufr27D3X3EQTdAv9w95T9lZRKZtYznCRB2E3zaWBRvFF1jLuvB1ab2djw0IlA2k36aOJ80rTbKbQKOMrMeoT/lp1IMM6aEp2+cVEXcizwBWBh2LcP8L1wY6V0Mxi4N5zFkQXMcve0nlraDQwEHg1+h8kBHnD3p+INaZ98Dbg/7LJZDlwaczwdFibuk4Cr4o6lo9x9jpk9BMwH6oAFpHCFdsZOjxURkWgytutJRESiUaIQEZFWKVGIiEirlChERKRVShQiItIqJQrJGGZ2T1tVXM3MzeyrCa+vNLOzOuPe4XXvm9mNbd0/rEz7032NSySKTF5HIdKco4EVCa+vJFgo99g+fu5/AoUdeF+y7i/SYUoUIglSVZrd3Zel4nNFOoO6niTjmNlZ4QY81Wb2opmNSzi3u+vJzJ4FJgIXh8fdzC4xs/8ws3cS3tPTzGrNbH7Csf5m1mBmJ4Wv9+p6MrPjws2mqsMKs8c0Od/s/Ztc800zW2NmH5rZzMZSLiLJpEQhmWY48HOCrqALgGJgdgu7g32FoGbWEwRdUkcDfwFeAMaY2cDwumMIyih8wsyKwmOTCUqlv9xcEGY2BHgS2Ax8HrgduJ9gX4G27t/oXIIaP1cC3wVOB/47wn8DkXZR15Nkmv7Ame7+EoCZzQOWAZcAtyVe6O5vmtkOYFNil5SZvUyQGCYTVFKdzMf/mB8DPBUeW5CwoVRT3wCqgdPcfWf4uTuA3dVyW7p/glqCXRnrwvePIyim+JXI/zVEIlCLQjLNxsYkARDubjaPdpQAd/cdBMXYJoeHjgOeJ2hpJB57oZWPmUSwwdTOhGOPRo0h9Exjkgi9CQwIy+eLJI0ShWSajS0cG9zOz3kBmBxWUz0yfN14rDdwGK0nikFNYwmTRkstkOY03Y60BjAgvx2fIdImJQrJNANaOLaunZ/zAvAJgjGCGuC18Ngk4HggG3ixlfevbxqLmfUAerUzDpGUU6KQTDMgcXaRmQ0j2K3t1RaurwGaG+h+geCv9+uAf7p7A7AQqAK+Bbzt7ptaiWMucFKYHBp9th33F+k0ShSSad4H7jOzC8zss8CfCbqA7mnh+rcJupOmmlm5mfUDcPfNBGMCjeMThMnin7Q9PgHwC4IFeH82s9PN7ErgRwSJps37i3QmJQrJNO8B3wZuJNiXezsw1d2rW7j+RwRbTM4iaAV8JuFcYzJ4vpljrXU7Ne6nfSrBLKyHCWYqXQTsbHJpa/cX6RTa4U5ERFqlFoWIiLRKiUJERFqlRCEiIq1SohARkVYpUYiISKuUKEREpFVKFCIi0iolChERaZUShYiItOr/AxRft8337FTTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bitwidths = [i for i in range(2,9)]\n",
    "bit_errors = []\n",
    "\n",
    "for num_bits in bitwidths:\n",
    "\n",
    "    x_quant, x_scale, x_offset = quantize(x, num_bits)\n",
    "\n",
    "    x_dequant = dequantize(x_quant, x_scale, x_offset)\n",
    "\n",
    "    error = np.reshape(x-x_dequant, (-1))\n",
    "\n",
    "    abs_error = np.mean(np.abs(error))\n",
    "\n",
    "    bit_errors.append(abs_error)\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.plot(bitwidths, bit_errors, marker=\"o\")\n",
    "ax.set_ylabel(\"absolute mean error\", fontsize=15)\n",
    "ax.set_xlabel(\"bitwidth\", fontsize=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What's Next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutoral, we have learn the basics of quantization, de-quantization and quantization loss (error). Next we will apply this new learned knowledge to perform actual computation in neural network and explore optimization methods to reduce computational efforts."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}