03 - Probability and Information Theory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Deep Learning Book (Simplified)\n",
    "## Part I - Applied Math and Machine Learning basics\n",
    "*This is a series of blog posts on the [Deep Learning book](http://deeplearningbook.org) where we are attempting to provide a summary of each chapter highlighting the concepts that we found to be most important so that other people can use it as a starting point for reading the chapters, while adding further explanations on few areas that we found difficult to grasp. Please refer [this](http://www.deeplearningbook.org/contents/notation.html) for more clarity on notation.*\n",
    "\n",
    "## Chapter 3: Probability and Information Theory\n",
    "Everything that we observe around us has some degree of [uncertainty](https://en.wikipedia.org/wiki/Uncertainty) to it. [Probability Theory](https://en.wikipedia.org/wiki/Probability_theory) provides a mathematical framework for representing this uncertainty. In AI applications, probability theory is used in two ways:\n",
    "\n",
    "- The law of probability specifies how an AI system should reason. For example, for those who are familiar with probability, a classification problem (e.g. given an image, classify whether the image is that of a \"cat\" or a \"dog\") can be viewed as finding $P(Y \\hspace{0.1cm} | \\hspace{0.1cm}  X)$, where $X$ is the input data and $Y$ is the label that we are predicting. We design our algorithms to compute or approximate (in case computing exact value is not feasible) various expressions derived using probability theory.\n",
    "\n",
    "- Use probability and statistics to analyze the behaviour of proposed AI systems. For example, we can analyze the accuracy of a classification model by observing how many of the predictions are correct.\n",
    "\n",
    "The sections present in this chapter are listed below. Feel free to navigate as you like: <br>\n",
    "\n",
    "**1. Why Probability?** <br>\n",
    "**2. Random Variables** <br>\n",
    "**3. Probability Distributions** <br>\n",
    "**4. Marginal Probability** <br>\n",
    "**5. Conditional Probability** <br>\n",
    "**6. The Chain Rule of Conditional Probabilities** <br>\n",
    "**7. Independence and Conditional Independence** <br>\n",
    "**8. Expectation, Variance and Covariance** <br>\n",
    "**9. Common Probability Distributions** <br>\n",
    "**10. Useful Properties of Common Functions** <br>\n",
    "**11. Bayes' Rule** <br>\n",
    "**12. Technical Details of Continuous Variables** <br>\n",
    "**13. Information Theory** <br>\n",
    "**14. Structured Probabilistic Models** <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Why Probability?\n",
    "Although Computer Scientists and Software Engineers deal mostly with entities that are entirely deterministic, Machine Learning makes heavy use of probability theory. The main reason for this being that Machine Learning must always deal with quantities which are either uncertain or [stochastic](https://en.wikipedia.org/wiki/Stochastic)  \n",
    "(non-deterministic). The possible sources of uncertainty are:\n",
    "- Inherent stochasticity in the system being modelled: For example, creating theoretical scenarios such as a hypothetical card game where we assume that the cards are truly shuffled into a random order.\n",
    "\n",
    "- Incomplete observability: When we cannot observe all the variables that drive the behaviour of a system, even deterministic systems can become stochastic, e.g. the [Monty Hall Problem](https://en.wikipedia.org/wiki/Monty_Hall_problem) where the outcome given the contestant's choice is deterministic, but from the point of view of the contestant, the outcome is uncertain.\n",
    "\n",
    "- Incomplete modelling: When we use a model that must discard some of the information that we have observed, the discarded information leads to an uncertainty in the model predictions.\n",
    "\n",
    "There are two types of kinds of probability:\n",
    "- **Frequentist Probability**: Probability theory was originally developed to analyze the frequencies of events (which are often repeatable, e.g. drawing a certain hand of cards in a poker game). When we say that an outcome has a probability p of occuring, it means that if we repeated the experiment infinte times, then proportion p of those repititions would produce that outcome. This kind of probability, rerelated directly to the rates at which events occur, is called frequentist probability.\n",
    "\n",
    "\n",
    "- **Bayesian Probability**: The above reasoning doesn't seem applicable to experiments which are not repeatable, e.g. when a doctor says that a patient has 40% chance of having the flu, the probability represents a **degree of belief**, with 1 indicating absolute certainty that the patient has the flu and 0 indicating absoluting certainty that the patient doesn't have the flu. This kind of probability, related to qualitative levels of reasoning, is called **Bayesian Probability**.\n",
    "\n",
    "However, in order to satisfy the properties that we expect common sense reasoning about uncertainty to have, we treat both Bayesian and frequentist probabilities as exactly the same. For example, the probability that a player will win a poker game given he/she has a certain set of cards is calculated in the exact same manner as the probability that a patient has a certain disease given that he/she has certain symptoms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Random Variables\n",
    "A **random variable** is a variable, e.g. x, that can take on different values (states) randomly. Since it takes on values randomly, there must be a probability associated with each of those values. Thus, a random variable must be coupled with a **probability distribution** that specifies how likely each of the states are.\n",
    "\n",
    "There are two types of random variables:\n",
    "- Discrete: The number of states are finite or [countably infinite](https://en.wikipedia.org/wiki/Countable_set).\n",
    "- Continuous: It is associated with a real value.\n",
    "\n",
    "More details on these in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Probability Distributions\n",
    "\n",
    "A **probability distribution** is a description of how likely a random variable or a set of random variables is, to take on each of its possible states. For example, if the random variable x can take values $\\{1,2\\}$, then $Pr(\\text{x}= 1) = 0.4$ specifies the probability distribution of x, where the probability of x taking the value 1 is 0.4 and the probability of x taking the value 2 is 0.6. Probability distributions are described based on whether the random variable is discrete or continuous."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1 Discrete Variables and Probability Mass functions\n",
    "The probability distribution over discrete random variables is described using a **probability mass function** (PMF). Along with its random variable, it is denoted as: x ~ $P(\\text{x})$. \n",
    "Note that $P(x)$ denotes $P(\\text{x} = x)$, which is the probability that x = $x$. A probability mass function acting over multiple variables is called a **joint probability distribution**. $P(\\text{x} = x, \\text{y} = y)$ denotes the probability that x = $x$ and y = $y$ simultaneously. \n",
    "\n",
    "For a function $P$ to be a PMF, it must satisfy these conditions:\n",
    "\n",
    "- The [domain](https://en.wikipedia.org/wiki/Domain_of_a_function) of P must be all possible states of x.\n",
    "- $\\forall x \\in $ x, $ 0 \\leq P(x) \\leq 1$\n",
    "- $ \\sum_{x \\in \\text{x}} P(x) = 1$ (normalization)\n",
    "\n",
    "For example, the uniform distribution on a random variable x with $k$ different states is given by: \n",
    "<br> <br>\n",
    "$$ P(\\text{x} - x_i) = 1/k $$\n",
    "\n",
    "To arrive at this, let we use the normalization condition:\n",
    "\n",
    "$$ \\sum_{i = 1}^{k} P(x_i) = 1 $$\n",
    "\n",
    "In a uniform distribution, all the states have the same probability (say p). Thus, we get:\n",
    "$$ kp = 1 $$\n",
    "$$ \\Rightarrow p = 1/k $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhIAAAFkCAYAAAB1rtL+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAH4FJREFUeJzt3XuQnXWd5/H3VwjXkbBOJgRkZgQCCOuCdINMarwgqBlx\nxXHUwQaLnaWGEsWRbdeFFS/M4iAjLgRRU4WXGRLRnkIcEUsgIzhmlOvQDVnBAAHDJVxiSKbC/ZZ8\n94/naTk56dPd50cn50l8v6qeKs5zfs/zfLpJ9fmc5xqZiSRJUolX9DqAJEnaclkkJElSMYuEJEkq\nZpGQJEnFLBKSJKmYRUKSJBWzSEiSpGIWCUmSVMwiIUmSilkkJElSsaIiERGnRMTyiHgmIm6MiMPG\nGXtgRFxWj18fER8fY8ybIuKKiHioHnPMGGP+sX6vdbqyJL8kSZoaXReJiDgWOA84EzgEWAIsiogZ\nHRbZCbgXOB14pMOYnYHbgFOA8R7+cRWwGzCrnga6zS9JkqbOtgXLDAIXZeZCgIg4GXgXcCJwbvvg\nzLwFuKUe+8WxVpiZVwNX12NinG0/l5mrCjJLkqRNoKs9EhExDegHrh2dl9XjQ68B5kxttDEdEREr\nI+LOiJgfEa/aDNuUJEkddLtHYgawDbCybf5KYP8pSdTZVcD3geXAPsA5wJURMSfHeBZ6RPw+MBe4\nD3h2E2eTJGlrsgPwGmBRZq4eb2DJoY2xBOOf2/CyZealLS/viIhfUp17cQTwr2MsMhf4zqbMJEnS\nVu544LvjDei2SDwGrKM64bHVTDbeS7FJZebyiHgMmM3YReI+gEsuuYQDDjhgc0ablMHBQebNm9fr\nGGNqaram5gKzlWpqtqbmArOVanK2Jlq6dCkf+tCHoP4sHU9XRSIzX4iIYeAo4Ar47cmRRwEXdp30\nZYiIPYHfp/OVIM8CHHDAAfT19W22XJM1ffr0RuaC5mZrai4wW6mmZmtqLjBbqSZna7gJTw0oObRx\nPrCgLhQ3U13FsRNwMUBELARWZOYZ9etpwIFUhz+2A14dEQcDT2bmvfWYnan2LIxesbF3PWZNZj5Y\nv38m1TkSj9ZjvwjcDSwq+BkkSdIU6LpIZOal9T0jzqI6xHEbMLflssw9gRdbFtkDuJWXzqH4ZD0t\nBo6s5x1KdXgi6+m8ev4CqstK1wEHAScAuwIPUxWIz2XmC93+DJIkaWoUnWyZmfOB+R3eO7Lt9f1M\ncJlpZi4eb0xmPgv8WfdJJUnSpuSzNnpkYKC5N+Vsaram5gKzlWpqtqbmArOVanK2LV2McQuGrUJE\n9AHDw8PDnmAjSVIXRkZG6O/vB+jPzJHxxrpHQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4Qk\nSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIk\nqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKk\nYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKK\nWSQkSVIxi4QkSSpmkZAkScWKikREnBIRyyPimYi4MSIOG2fsgRFxWT1+fUR8fIwxb4qIKyLioXrM\nMR3WdVZEPBwRT0fETyJidkl+SZI0NbouEhFxLHAecCZwCLAEWBQRMzosshNwL3A68EiHMTsDtwGn\nANlhu6cDHwM+DLwBeKre7nbd/gySJGlqlOyRGAQuysyFmXkncDLwNHDiWIMz85bMPD0zLwWe7zDm\n6sz8XGZeDkSH7Z4KfD4zf5SZtwMnAHsAf17wM0iSpCnQVZGIiGlAP3Dt6LzMTOAaYM7URttgu3sB\ns9q2+zhw06bcriRJGl+3eyRmANsAK9vmr6T6oN9UZlEd8tjc25UkSePYdorWE3Q4t2ETm3C7g4OD\nTJ8+fYN5AwMDDAwMbMpckiRtEYaGhhgaGtpg3tq1aye9fLdF4jFgHbBb2/yZbLy3YCo9SlUadmvb\nzkzg1vEWnDdvHn19fZswmiRJW66xvlyPjIzQ398/qeW7OrSRmS8Aw8BRo/MiIurX13ezri63u5yq\nTLRudxfg8E25XUmSNL6SQxvnAwsiYhi4meoqjp2AiwEiYiGwIjPPqF9PAw6k2qOwHfDqiDgYeDIz\n763H7AzM5qUrNvaux6zJzAfreRcAn4mIe4D7gM8DK4AfFvwMkiRpCnRdJDLz0vqeEWdRHWq4DZib\nmavqIXsCL7YssgfV4YfRcxk+WU+LgSPreYcC/1qPSar7VAAsoL6sNDPPjYidgIuAXYGfA+/MzDEv\nKZUkSZte0cmWmTkfmN/hvSPbXt/PBIdQMnPxRGPqcX8L/O1kc0qSpE3LZ21IkqRiFglJklTMIiFJ\nkopZJCRJUjGLhCRJKmaRkCRJxSwSkiSpmEVCkiQVs0hIkqRiFglJklTMIiFJkopZJCRJUjGLhCRJ\nKmaRkCRJxSwSkiSpmEVCkiQVs0hIkqRiFglJklTMIiFJkopZJCRJUjGLhCRJKmaRkCRJxSwSkiSp\nmEVCkiQVs0hIkqRiFglJklTMIiFJkopZJCRJUjGLhCRJKmaRkCRJxSwSkiSpmEVCkiQVs0hIkqRi\nFglJklTMIiFJkopZJCRJUjGLhCRJKmaRkCRJxSwSkiSpmEVCkiQVKyoSEXFKRCyPiGci4saIOGyc\nsQdGxGX1+PUR8fGSdUbEz+rlR6d1ETG/JL8kSZoaXReJiDgWOA84EzgEWAIsiogZHRbZCbgXOB14\n5GWsM4GvA7sBs4DdgdO6zS9JkqZOyR6JQeCizFyYmXcCJwNPAyeONTgzb8nM0zPzUuD5l7nOpzNz\nVWb+pp6eLMgvSZKmSFdFIiKmAf3AtaPzMjOBa4A5JQG6XOfxEbEqIn4ZEV+IiB1LtilJkqbGtl2O\nnwFsA6xsm78S2L8ww2TX+R3gfuBh4CDgXGA/4P2F25UkSS9Tt0Wik6A6h2EqbbDOzPxmy3t3RMSj\nwDURsVdmLp/ibUuSpEnotkg8BqyjOuGx1Uw23qOwqdd5E1XZmA10LBKDg4NMnz59g3kDAwMMDAwU\nhZUkaWsyNDTE0NDQBvPWrl076eW7KhKZ+UJEDANHAVcARETUry/sZl1TsM5DqPZYjHklyKh58+bR\n19dXEk2SpK3eWF+uR0ZG6O/vn9TyJYc2zgcW1B/+N1NdcbETcDFARCwEVmTmGfXracCBVHsPtgNe\nHREHA09m5r2TXOfewHHAlcBq4OB6mcWZeXvBzyBJkqZA10UiMy+t7+9wFtXhiNuAuZm5qh6yJ/Bi\nyyJ7ALfy0vkOn6ynxcCRk1zn88DbgFOBnYEHge8BZ3ebX5IkTZ2iky0zcz4w5l0lM/PIttf3M4nL\nTCdY5wrgiK6DSpKkTcpnbUiSpGIWCUmSVMwiIUmSilkkJElSMYuEJEkqZpGQJEnFLBKSJKmYRUKS\nJBWzSEiSpGIWCUmSVMwiIUmSilkkJElSMYuEJEkqZpGQJEnFLBKSJKmYRUKSJBWzSEiSpGIWCUmS\nVMwiIUmSilkkJElSMYuEJEkqZpGQJEnFLBKSJKmYRUKSJBWzSEiSpGIWCUmSVMwiIUmSilkkJElS\nMYuEJEkqZpGQJEnFLBKSJKmYRUKSJBWzSEiSpGIWCUmSVMwiIUmSilkkJElSMYuEJEkqZpGQJEnF\nLBKSJKmYRUKSJBWzSEiSpGJFRSIiTomI5RHxTETcGBGHjTP2wIi4rB6/PiI+XrLOiNg+Ir4WEY9F\nxBP1OmeW5JckSVOj6yIREccC5wFnAocAS4BFETGjwyI7AfcCpwOPvIx1XgC8C3gf8GZgD+D73eZv\ngrvvvpurrrqKZcuW9TrKRpqaram5wGylmpqtqbnAbKWanG2rkJldTcCNwJdbXgewAjhtEssuBz7e\n7TqBXYDngPe2jNkfWA+8ocO2+oAcHh7Opli9enXOnXt0Ar+d5s49OtesWdPraI3N1tRcZtv6sjU1\nl9m2zmxNNzw8PPo768uJPtsnGpAbfjhPA14AjmmbfzHwg0ksv1GRmMw6gSOBdcAubWPuA07tsK3G\nFYm5c4/ObbZ5VcIlCQ8kXJLbbPOqnDv36F5Ha2y2puYy29aXram5zLZ1Zmu6TVkkdq/3AhzeNv+L\nwA2TWH6sIjHhOoEB4Jkx1ncTcE6HbTWqSNx11131/5RLErJl+nYCeffdd5ttC8lltq0vW1NzmW3r\nzLYl6KZIbMvUiHqDU2ky65xwzODgINOnT99g3sDAAAMDAy8vXZfuvffe+r/e3PbOWwC455572Hff\nfTdrplFNzdbUXGC2Uk3N1tRcYLZSTc7WNENDQwwNDW0wb+3atZNevtsi8RjVIYbd2ubPBFZ2ua5u\n1vkosF1E7JKZj3ez3Xnz5tHX11cYberss88+9X/9G3B8yzuLAZg9e/bmjvRbTc3W1FxgtlJNzdbU\nXGC2Uk3O1jRjfbkeGRmhv79/ciuYaJdF+8TYJ0Y+CPyvSSy70aGNyayTsU+23I8t7GTLl47Xfbs+\nXvftxhyva2q2puYy29aXram5zLZ1Zmu6TXaORFYf0H8JPAOcALwWuAhYDfxB/f5C4Ast46cBBwOv\nBx6iOvfhYGCfya6zHjO/LiJHAP3AdcDPx8nZuCKxZs2axp5B3NRsTc1ltq0vW1NzmW3rzNZ03RSJ\nyOz+1IaI+ChwGtXhiNuAv8nMW+r3fgrcl5kn1q//uC4A7RtanJlHTmad9fvbA/+X6sTL7YGrgVMy\n8zcdMvYBw8PDw404tNFq2bJl3HPPPcyePbtxx+iamq2pucBspZqaram5wGylmpytqVoObfRn5sh4\nY4uKxJagyUVCkqQm66ZI+KwNSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNI\nSJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIh\nSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4Qk\nSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIk\nqZhFQpIkFbNISJKkYhYJSZJUrKhIRMQpEbE8Ip6JiBsj4rAJxn8gIpbW45dExDvb3p8ZERdHxEMR\n8VREXBkRs9vG/Cwi1rdM6yJifkl+SZI0NbouEhFxLHAecCZwCLAEWBQRMzqMnwN8F/gG8HrgcuDy\niDiwZdgPgdcA767HPABcExE7toxJ4OvAbsAsYHfgtG7zS5KkqVOyR2IQuCgzF2bmncDJwNPAiR3G\nnwpclZnnZ+ZdmXkmMAJ8DCAi9gUOB07OzJHMXAZ8BNgRGGhb19OZuSozf1NPTxbklyRJU6SrIhER\n04B+4NrReZmZwDXAnA6Lzanfb7WoZfz2VHsbnmtb53PAG9uWOz4iVkXELyPiC217LCRJ0ma2bZfj\nZwDbACvb5q8E9u+wzKwO42fV/30n1aGMcyJidO/GILAn1eGLUd8B7gceBg4CzgX2A97f5c8gSZKm\nSLdFopOg2qvQ9fjMfDEi/gL4FrAGeJFqD8aVrQtk5jdbXt4REY9SnUexV2Yu77ShwcFBpk+fvsG8\ngYEBBgbaj5pIkvS7Z2hoiKGhoQ3mrV27dtLLd1skHgPWUZ3w2GomG+91GPXoROMz81agLyJeCWyX\nmasj4kbg38fJchNVIZkNdCwS8+bNo6+vb5zVSJL0u2usL9cjIyP09/dPavmuzpHIzBeAYeCo0XkR\nEfXr6zssdkPr+Nrb6/nt63+iLhH7AodSXeHRySFUezUemfQPIEmSplTJoY3zgQURMQzcTHU+w07A\nxQARsRBYkZln1OO/DCyOiE8AP6a6EqMfOGl0hRHxfmAV1bkSBwEXAP+cmdfW7+8NHEd1uGM1cHCd\nY3Fm3l7wM0iSpCnQdZHIzEvre0acRXXI4jZgbmauqofsSXWew+j4GyJiADi7npYB78nMX7Wsdneq\nYjCTag/DAuDvWt5/Hngb1aWkOwMPAt+r1ydJknqk6GTLzJwPjHlXycw8cox53we+P876vgJ8ZZz3\nVwBHdB1UkiRtUj5rQ5IkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhF\nQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJ\nSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQk\nSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAk\nScUsEpIkqVhRkYiIUyJieUQ8ExE3RsRhE4z/QEQsrccviYh3tr0/MyIujoiHIuKpiLgyIma3jdk+\nIr4WEY9FxBMRcVlEzCzJL0mSpkbXRSIijgXOA84EDgGWAIsiYkaH8XOA7wLfAF4PXA5cHhEHtgz7\nIfAa4N31mAeAayJix5YxFwDvAt4HvBnYA/h+t/klSdLUKdkjMQhclJkLM/NO4GTgaeDEDuNPBa7K\nzPMz867MPBMYAT4GEBH7AocDJ2fmSGYuAz4C7AgM1GN2qdc/mJmLM/NW4L8DfxoRbyj4GSRJ0hTo\nqkhExDSgH7h2dF5mJnANMKfDYnPq91stahm/PZDAc23rfA54Yz3rUGDbtu3eRbXnotN2JUnSJtbt\nHokZwDbAyrb5K4FZHZaZNcH4O6kKwTkRsWtEbBcRpwN7ArvXY3YDns/Mx7vYriRJ2sSm6qqNoNqr\n0PX4zHwR+AtgP2AN8CTwFuBKYN0Ub1eSJE2hbbsc/xjVh/tubfNnsvFeh1GPTjS+PuehLyJeCWyX\nmasj4kbg31vWsV1E7NK2V2K87QIwODjI9OnTN5g3MDDAwMDAeItJkvQ7YWhoiKGhoQ3mrV27dtLL\nR3U6wuTVH/A3Zeap9eugOjRxYWZ+aYzx/wTsmJnvaZl3HbAkMz/aYRv7AkuBuZl5bX2y5Srgg5n5\ng3rMflSHRf4kM28eYx19wPDw8DB9fX1d/YySJP0uGxkZob+/H6A/M0fGG9vtHgmA84EFETEM3Ex1\nFcdOwMUAEbEQWJGZZ9TjvwwsjohPAD+muhKjHzhpdIUR8X6qovAAcBDVpZ7/nJnXAmTm4xHxLeD8\niPgP4AngQuC6sUqEJEnaPLouEpl5aX3PiLOoDlncRrXnYFU9ZE/gxZbxN0TEAHB2PS0D3pOZv2pZ\n7e5UBWUm8AiwAPi7tk0PUh1WuYzqSo+rgVO6zS9JkqZO14c2thQe2pAkqUw3hzZ81oYkSSpmkZAk\nScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIk\nFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJU\nzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIx\ni4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYhYJSZJUzCIhSZKKWSQkSVIxi0SPDA0N9TpCR03N\n1tRcYLZSTc3W1FxgtlJNzralKyoSEXFKRCyPiGci4saIOGyC8R+IiKX1+CUR8c6293eOiK9GxIMR\n8XRE3BERH24b87OIWN8yrYuI+SX5m6DJ/6ibmq2pucBspZqaram5wGylmpxtS9d1kYiIY4HzgDOB\nQ4AlwKKImNFh/Bzgu8A3gNcDlwOXR8SBLcPmAe8AjgNeC1wAfDUi/mvLmAS+DuwGzAJ2B07rNr8k\nSZo6JXskBoGLMnNhZt4JnAw8DZzYYfypwFWZeX5m3pWZZwIjwMdaxswBFmTmzzPzgcz8BlVBeUPb\nup7OzFWZ+Zt6erIgvyRJmiJdFYmImAb0A9eOzsvMBK6hKgNjmVO/32pR2/jrgWMiYo96O28F9q3H\ntTo+IlZFxC8j4gsRsWM3+SVJ0tTatsvxM4BtgJVt81cC+3dYZlaH8bNaXv8N1WGLFRHxIrAOOCkz\nr2sZ8x3gfuBh4CDgXGA/4P0dtrsDwNKlS8f5cXpn7dq1jIyM9DrGmJqaram5wGylmpqtqbnAbKWa\nnK2JWj47d5hwcGZOeqI6L2E9cHjb/HOB6zss8xxwbNu8jwIPt7z+JLAUOBp4Xf3+48CR42R5K1Xh\n2KvD+8dRnVfh5OTk5OTkVDYdN1E36HaPxGNUH967tc2fycZ7HUY9Ot74iNgBOBt4T2ZeXb9/e0Qc\nQlUwftphvTcBAcwGlo/x/iLgeOA+4NkO65AkSRvbAXgNG59isJGuikRmvhARw8BRwBUAERH16ws7\nLHbDGO+/vZ4PMK2esm25dYx/Dsch9TKPdMi6mupqEUmS1L3rJzOo2z0SAOcDC+pCcTPVVRw7ARcD\nRMRCYEVmnlGP/zKwOCI+AfwYGKA6YfMkgMx8IiIWA1+KiGepzoM4AjgB+B/1OvemOlRxJbAaOLjO\nsTgzby/4GSRJ0hToukhk5qX1PSPOojpkcRswNzNX1UP2BF5sGX9DRAxQHb44G1hGdRjjVy2rPRY4\nB7gEeBVVmfhUZn69fv954G1Ul5LuDDwIfK9enyRJ6pGoT0yUJEnqms/akCRJxSwSkiSpmEViM4qI\nN0XEFRHxUP3gsWN6nQkgIj4VETdHxOMRsTIifhAR+/U6F0BEnFw/6G1tPV0fEX/W61zt6t/h+og4\nv9dZACLizLaH3K2PiF9NvOSmFxF7RMS3I+Kx+iF9SyKirwG5lo/xO1sfEV9pQLZXRMTnI+LX9e/s\nnoj4TK9zAUTE70XEBRFxX53tFxFxaA9yTPj3NSLOioiH65w/iYjZmzvn1sgisXntTHVy6ilsfLlr\nL70J+ApwONVJrdOAf2nILcgfBE6nutKnn+q+Ij+MiAN6mqpF/fTbk6ieD9Mkt/PSQ+5mAW/sbRyI\niF2B66huVDcXOAD4n8B/9DJX7VBe+l3NorpMPYFLexmq9r+BD1PdrO+1VA8sPC0iPjbuUpvHt6gu\n8T+e6oaCPwGuiYjdN3OOcf++RsTpVM94+jDVc5yeonrg5HabM+TWyJMteyQi1gN/nplX9DpLu/qq\nnN8Ab87MX/Q6T7uIWA18MjP/sQFZfg8YBj4CfBa4NTM/0dtU1R4Jqqujev5Nv1VE/D0wJzPf0uss\nE4mIC4CjM7Pne+ci4kfAo5l5Usu8y6geZHhCD3PtADwBvLvlhoJExC3AlZn5uR7l2ujva0Q8DHwp\nM+fVr3ehujHif8vMJpTFLZZ7JDSWXaka/ZpeB2lV7979INV9S26YaPxm8jXgR5nZ6Q6svbRvvZv3\n3oi4JCL+sNeBgHcDt0TEpfVhtJGI+Oteh2pXP6DweKpv201wPXBUROwLEBEHA39KdW+dXtqW6vlL\nz7XNf4YG7AEbFRF7Ue1lan3g5ONUd0ju9MBJTVLJDam0FavvVHoB8Iu2e330TES8jqo4jH77eW/9\nCPueqkvN66l2iTfNjcBfAXdRPSPnb4F/i4jXZeZTPcy1N9Xem/Oo7gNzOHBhRDybmZf0MFe79wLT\ngQW9DlL7e2AX4M6IGL3r76cz8596GSozn4yIG4DPRsSdVN/wj6P6cF7Wy2xtZlF9OZroAZIqYJFQ\nu/nAgVTfdpriTqq7me4KvA9YGBFv7mWZiIg9qQrX2zPzhV7l6CQzW++Pf3tE3Ex1o7e/BHp5SOgV\nwM2Z+dn69ZKI+M9U5aJJReJE4KrMfLTXQWrHUn1AfxD4FVWB/XJEPJyZ3+5pMvgQ8A/AQ1Q3Ixyh\nejxBow6rdRA063y1LZKHNvRbEfFVqiewHpGZYz7DpBcy88XM/HVmjmTmp6lOajy1x7H6gT8AhiPi\nhYh4AXgLcGpEPF/v2WmMzFwL3E31kLteeoTqSb+tlgJ/1IMsY4qIP6I66fgbvc7S4lzgnMz8Xmbe\nkZnfAeYBn+pxLjJzeWa+lepkxz/MzD8BtmPshyn2yqNUpaGbB05qkiwSAn5bIt4DvDUzH+h1ngm8\nAti+xxmuAf4L1TfDg+vpFqpv1Qdnw85irk8K3YcOD7nbjK4D9m+btz/V3pKmOJHqw6XX5x+02omN\nvzmvp0F/wzPzmcxcGRH/ieqKnMt7nWlUZi6nKhNHjc6rT7Y8nEk+mEqdeWhjM4qInam+EY5+W927\nPmlqTWY+2MNc86kepnYM8FREjLb2tZnZ00ewR8TZwFVUl4G+kuoEuLcA7+hlrvo8gw3OIYmIp4DV\nmdn+jXuzi4gvAT+i+oB+NfB/qHY7D/UyF9W36Osi4lNUl1UeDvw19UP8eq3ek/RXwMWZub7HcVr9\nCPh0RDwI3EF12GAQ+GZPUwER8Q6qv2l3AftS7T1ZSv0gx82YY6K/rxcAn4mIe4D7gM8DK4Afbs6c\nW6XMdNpME9UH4HqqR6S3Tv/Q41xjZVoHnNCA39k3gV9TnQX+KPAvwJG9ztUh60+B83udo84yRPVH\n8hngAapj1nv1Oled7Wjg/wFPU30ontjrTC3Z3l7/25/d6yxtuXameuLxcqr7HyyjKofbNiDbB4B7\n6n9rD1E98fmVPcgx4d9XqpOOH67/7S1q2v/nLXXyPhKSJKlYY46vSZKkLY9FQpIkFbNISJKkYhYJ\nSZJUzCIhSZKKWSQkSVIxi4QkSSpmkZAkScUsEpIkqZhFQpIkFbNISJKkYv8fkYFxqby8CQUAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2043feac50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "k = 10\n",
    "x = range(1, k+1)\n",
    "prob = [float(1)/k] * k\n",
    "plt.scatter(x, prob)\n",
    "plt.xticks(np.arange(min(x), max(x)+1, 1.0))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2 Continuous variables and Probability Density Functions\n",
    "We describe probability distributions using a **probability density function** (PDF) (denoted by $p$) when working with continuous random variables. $p$ must satisfy the following conditions to be a PDF:\n",
    "- The domain of $p$ should be all the possible states of x.\n",
    "- $\\forall x \\in$ x, $p(x) \\geq 0$ i.e. $p(x)$ can be greater than 1.\n",
    "- $\\int p(x)dx = 1$\n",
    "\n",
    "It's important to understand here that $p(x)$ doesn't give the probability of a specific state. The probability of landing inside an infinitesimal region with volume $\\delta x$ is given by $p(x)\\delta x$. Thus, the probability that $x$ lies in the interval $[a,b]$ is given by: $\\int_{[a,b]} p(x)dx$. The diagram below better illustrates this.\n",
    "![pdf](http://slideplayer.com/slide/8387312/26/images/12/The+Probability+Density+Function.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, consider a uniform distribution on an interval of real numbers. This is denoted by: $ x \\sim U(a, b) $ <br>\n",
    "The corresponding function $u(x; a,b)$ (\";\" denoting \"parameterized by\") is given by:\n",
    "\n",
    "$$ \n",
    "\\\\ \n",
    "u(x; a,b) = \n",
    "     \\begin{cases}\n",
    "       \\frac{1}{b-a} &\\quad \\text{for } x \\in [a, b]\\\\\n",
    "       0 &\\quad otherwise\n",
    "     \\end{cases}\n",
    "\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAFkCAYAAACjCwibAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAHItJREFUeJzt3XuUZWV95vHv04CXlliY9HBREhwvAYwJ0iU6jJeMECUX\noxIzgxVcGklUEkxIkURHEmyD48TAAhxMegbHjHRLLBdmjMJaJJ2AmTGKDbFKYBm7iY6NgJcGBMto\n46W7f/PHPk2Koqr6Pae7OFXF97NWreK8e7/v+e3VdJ/nvPvde6eqkCRJ2ptVwy5AkiQtD4YGSZLU\nxNAgSZKaGBokSVITQ4MkSWpiaJAkSU0MDZIkqYmhQZIkNTE0SJKkJoYGSZLUZKDQkOSsJNuS3J9k\nc5ITFtj3tUl2J9nV+707yY5Z+6xLsiXJt5Pcm+TvkjxnkNokSdLi6Ds0JDkNuAhYBxwP3AxsSrJm\ngW7TwOEzfo6atf1W4CzgmcDzgNuAv03yI/3WJ0mSFkf6fWBVks3ADVV1du91gDuAS6vqgjn2fy1w\nSVX9cB/v8UN0QePkqvr7vgqUJEmLoq+ZhiQHAaPAdXvaqksd1wInLtD14CS3Jbk9yUeTPGMv7/FG\n4Jt0sxiSJGkJOLDP/dcABwDbZ7VvB46ep8+twBnALcAI8PvA9Ul+oqq+smenJL8AfAhYDXwVeHFV\n3TvXgL3TFqfQncb4bp/HIEnSI9ljgCcDm6rqG/107Dc0zCfAnOc5qmozsPmBHZNPA1uAN9Cti9jj\n48BxdMHk9cCHkzynqu6ZY9hTgL/YP6VLkvSIdDrwwX469Bsa7gF2AYfNaj+Uh84+zKmqdib5LPC0\nWe33A1/q/dyY5J+BXwP+ZI5hbgO44oorOPbYY/upf8kaHx/nkksuGXYZ+81KOp6VdCzg8SxlK+lY\nwONZqrZs2cKrX/1q6H2W9qOv0FBVP0gyCZwMXAUPLIQ8Gbi0ZYwkq+iukrhmL7uuAh49z7bvAhx7\n7LGsXbu25W2XvJGRkRVzLLCyjmclHQt4PEvZSjoW8HiWgb5P7w9yeuJiYEMvPNwIjNOtQ7gcIMlG\n4M6qOrf3+jy60xNfBA4B3kx3yeX7ettXA39AF0K+Rnd64k3AE4EPD1CfJElaBH2Hhqq6sndPhvPp\nTlPcBJxSVXf3djkS2DmjyxOA99Ldn+E+YBI4saq29rbvAo4BXkMXGL4B/CPw/Kra0vcRSZKkRTHQ\nQsiqWg+sn2fbSbNenwOcs8BY3wNeOUgdkiTp4eOzJ5aIsbGxYZewX62k41lJxwIez1K2ko4FPJ6V\nqO87Qi4FSdYCk5OTkyttUYokSYtqamqK0dFRgNGqmuqnrzMNkiSpiaFBkiQ1MTRIkqQmhgZJktTE\n0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhND\ngyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwN\nkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRI\nkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJajJQaEhyVpJtSe5PsjnJCQvs+9oku5Ps6v3enWTHjO0H\nJvmTJLck+XaSryTZkOSIQWqTJEmLo+/QkOQ04CJgHXA8cDOwKcmaBbpNA4fP+DlqxrbVwLOAP+qN\ndypwNPCxfmuTJEmL58AB+owDl1XVRoAkZwK/AJwBXDBPn6qqu+fZ8C3glJltSd4E3JDkyKq6c4Aa\nJUnSftbXTEOSg4BR4Lo9bVVVwLXAiQt0PTjJbUluT/LRJM/Yy1sdAhTwzX7qkyRJi6ff0xNrgAOA\n7bPat9OddpjLrXSzEC8DTu+95/VJnjTXzkkeDbwL+GBVfbvP+iRJ0iIZ5PTEXEI3M/AQVbUZ2PzA\njsmngS3AG+jWRTBj24HAh3tj/ebe3nR8fJyRkZEHtY2NjTE2NtZn+ZIkrTwTExNMTEw8qG16enrg\n8dKdXWjcuTs9sQN4ZVVdNaP9cmCkqk5tHOdK4AdVdfqMtj2B4cnASVV13wL91wKTk5OTrF27trl+\nSZIe6aamphgdHQUYraqpfvr2dXqiqn4ATAIn72lLkt7r61vGSLIKeCbwtRltewLDU4CTFwoMkiRp\nOAY5PXExsCHJJHAj3dUUq4HLAZJsBO6sqnN7r8+jOz3xRboFjm+mu+Tyfb3tBwD/m+6yy5cCByU5\nrPde9/aCiiRJGrK+Q0NVXdm7J8P5wGHATcApMy6pPBLYOaPLE4D30i2UvI9upuLEqto6Y/+X9v77\npt7vPWskXgR8ot8aJUnS/jfQQsiqWg+sn2fbSbNenwOcs8BYX6a7IkOSJC1hPntCkiQ1MTRIkqQm\nhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoY\nGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJo\nkCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFB\nkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpyUChIclZSbYluT/J5iQn\nLLDva5PsTrKr93t3kh2z9jk1yd8kubu3/acGqUuSJC2evkNDktOAi4B1wPHAzcCmJGsW6DYNHD7j\n56hZ2x8HfBJ4C1D91iRJkhbfgQP0GQcuq6qNAEnOBH4BOAO4YJ4+VVV3zzdgVV3RG+soIAPUJEmS\nFllfMw1JDgJGgev2tFVVAdcCJy7Q9eAktyW5PclHkzxjoGolSdLQ9Ht6Yg1wALB9Vvt2utMOc7mV\nbhbiZcDpvfe8PsmT+nxvSZI0RIOcnphLmGctQlVtBjY/sGPyaWAL8Aa6dRGSJGkZ6Dc03APsAg6b\n1X4oD519mFNV7UzyWeBpfb73Q4yPjzMyMvKgtrGxMcbGxvZ1aEmSlr2JiQkmJiYe1DY9PT3weOmW\nJPTRIdkM3FBVZ/deB7gduLSqLmzovwr4HHBNVf3erG1HAV8Cjq+qWxYYYy0wOTk5ydq1a/uqX5Kk\nR7KpqSlGR0cBRqtqqp++g5yeuBjYkGQSuJHuaorVwOUASTYCd1bVub3X59GdnvgicAjwZrpLLt+3\nZ8AkTwB+DHgS3amOY3ph5OtV1TSDIUmSFlffoaGqruzdk+F8utMUNwGnzLik8khg54wuTwDeS7dQ\n8j5gEjixqrbO2OdlwPvp1kUUsGcu5Y967yNJkoZsoIWQVbUeWD/PtpNmvT4HOGcv420ANgxSiyRJ\nenj47AlJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJ\nkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJ\namJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSp\niaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQm\nhgZJktRkoNCQ5Kwk25Lcn2RzkhMW2Pe1SXYn2dX7vTvJjjn2Oz/JV5PsSPJ3SZ42SG2SJGlx9B0a\nkpwGXASsA44HbgY2JVmzQLdp4PAZP0fNGvMtwJuANwLPAb7TG/NR/dYnSZIWxyAzDePAZVW1saq2\nAmcCO4AzFuhTVXV3Vd3V+7l71vazgXdU1dVV9TngNcATgVcMUJ8kSVoEfYWGJAcBo8B1e9qqqoBr\ngRMX6HpwktuS3J7ko0meMWPMf0s3+zBzzG8BN+xlTEmS9DA6sM/91wAHANtntW8Hjp6nz610sxC3\nACPA7wPXJ/mJqvoKXWCoecY8vM/6tATs2AFbtw67Ci3kmGNg9ephVyFpuek3NMwndB/8D1FVm4HN\nD+yYfBrYAryBbl1E32PuMT4+zsjIyIPaxsbGGBsba6tai2LrVhgdHXYVWsjkJKxdO+wqJC22iYkJ\nJiYmHtQ2PT098Hj9hoZ7gF3AYbPaD+WhMwVzqqqdST4L7Lk64ut0AeGwWWMcCnx2obEuueQS1vov\n35JzzDHdh5KWrmOOGXYFkh4Oc32RnpqaYnTAb3Z9hYaq+kGSSeBk4CqAJOm9vrRljCSrgGcC1/TG\n3Jbk670xbunt83jgucCf9VOflobVq/0WK0kr0SCnJy4GNvTCw410V1OsBi4HSLIRuLOqzu29Po/u\n9MQXgUOAN9Ndcvm+GWO+G/jDJF8EbgPeAdwJfGyA+iRJ0iLoOzRU1ZW9ezKcT3dK4SbglBmXUR4J\n7JzR5QnAe+kWNd4HTAIn9i7X3DPmBUlWA5fRBYt/AH6uqr7f/yFJkqTFMNBCyKpaD6yfZ9tJs16f\nA5zTMObbgbcPUo8kSVp8PntCkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwN\nkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRI\nkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJ\nkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJ\namJokCRJTQwNkiSpiaFBkiQ1GSg0JDkrybYk9yfZnOSExn6vSrI7yUdmtR+a5PIkX0nynSTXJHna\nILVJkqTF0XdoSHIacBGwDjgeuBnYlGTNXvodBVwIfGKOzR8Dngz8IvAs4Hbg2iSP7bc+SZK0OAaZ\naRgHLquqjVW1FTgT2AGcMV+HJKuAK4C3AdtmbXs68FzgzKqaqqovAL8BPBYYG6A+SZK0CPoKDUkO\nAkaB6/a0VVUB1wInLtB1HXBXVb1/jm2PBgr43qwxvwc8v5/6JEnS4ul3pmENcACwfVb7duDwuTok\neR7wOuDX5xlzK93piD9OckiSRyV5C3AkcESf9UmSpEVy4H4aJ3SzBQ9uTA4GPgC8vqrum6tjVe1M\n8kvAnwP3AjvpZi6u2dubjo+PMzIy8qC2sbExxsY8qyFJ0sTEBBMTEw9qm56eHni8dGcCGnfuTk/s\nAF5ZVVfNaL8cGKmqU2ftfxwwBeyiCxbwr7Mbu4Cjq2rbjP1/CHhUVX0jyWbgH6vqt+aoYy0wOTk5\nydq1a5vrlyTpkW5qaorR0VGA0aqa6qdvX6cnquoHwCRw8p62JOm9vn6OLluAn6S7IuK43s9VwMd7\n/33HrPH/pRcYng48G/hoP/VJkqTFM8jpiYuBDUkmgRvprqZYDVwOkGQjcGdVnVtV3wc+P7Nzkm/S\nrXXcMqPtl4G76dY2/BTwbuAjVXUdkiRpSeg7NFTVlb17MpwPHAbcBJxSVXf3djmSbl1CP46gCyOH\nAl8DNgD/pd/aJEnS4hloIWRVrQfWz7PtpL30fd0cbe8B3jNILZIk6eHhsyckSVITQ4MkSWpiaJAk\nSU0MDZIkqYmhQZIkNTE0SJKkJoYGSZLUxNAgSZKaGBokSVITQ4MkSWpiaJAkSU0MDZIkqYmhQZIk\nNTE0SJKkJoYGSZLUxNAgSZKaGBokSVITQ4MkSWpiaJAkSU0MDZIkqYmhQZIkNTE0SJKkJoYGSZLU\nxNAgSZKaGBokSVITQ4MkSWpiaJAkSU0MDZIkqYmhQZIkNTE0SJKkJoYGSZLUxNAgSZKaGBokSVIT\nQ4MkSWpiaJAkSU0MDZIkqYmhQZIkNTE0SJKkJoYGSZLUxNAgSZKaDBQakpyVZFuS+5NsTnJCY79X\nJdmd5COz2h+X5E+T3JFkR5J/SvLGQWqTJEmLo+/QkOQ04CJgHXA8cDOwKcmavfQ7CrgQ+MQcmy8B\nXgL8CnAM8G7gT5O8tN/6JEnS4hhkpmEcuKyqNlbVVuBMYAdwxnwdkqwCrgDeBmybY5cTgQ1V9Q9V\ndXtV/U+6MPKcAeqTJEmLoK/QkOQgYBS4bk9bVRVwLd0H/3zWAXdV1fvn2X498LIkT+y9z4uApwOb\n+qlPkiQtngP73H8NcACwfVb7duDouTokeR7wOuC4Bcb9LeC9wJ1JdgK7gNdX1af6rE+SJC2SfkPD\nfALUQxqTg4EP0AWA+xbo/9vAc4GXArcDLwTWJ/lqVX18P9UoSZL2Qb+h4R66WYDDZrUfykNnHwCe\nChwFXJ0kvbZVAEm+Tzc78TXgncDLq+pvevt8LsnxwO8B84aG8fFxRkZGHtQ2NjbG2NhYP8ckSdKK\nNDExwcTExIPapqenBx4v3ZKEPjokm4Ebqurs3uvQzQ5cWlUXztr3UcDTZg3xTuBgutmFLwCPBaaB\nn6uqTTP6/g/gyVX1s3PUsBaYnJycZO3atX3VL0nSI9nU1BSjo6MAo1U11U/fQU5PXAxsSDIJ3Eh3\nNcVq4HKAJBuBO6vq3Kr6PvD5mZ2TfJNu/eSWXtO/JPm/wIVJvgt8GfgPwGuA3xmgPkmStAj6Dg1V\ndWXvngzn052muAk4paru7u1yJLCzz2FPA/6Y7rLMH6YLDm+tqvf2W58kSVocAy2ErKr1wPp5tp20\nl76vm6PtLuDXBqlFkiQ9PHz2hCRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoY\nGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJo\nkCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFB\nkiQ1MTRIkqQmhgZJktTE0CBJkpoYGiRJUhNDgyRJamJokCRJTQwNkiSpiaFBkiQ1MTRIkqQmhgZJ\nktTE0CBJkpoYGiRJUhNDwxIxMTEx7BL2q5V0PCvpWMDjWcpW0rGAx7MSDRQakpyVZFuS+5NsTnJC\nY79XJdmd5COz2ncn2dX7PfPndwepbzlaaf8zrqTjWUnHAh7PUraSjgU8npWo79CQ5DTgImAdcDxw\nM7ApyZq99DsKuBD4xBybDweO6P0+HDgD2A38Zb/1SZKkxTHITMM4cFlVbayqrcCZwA66D/o5JVkF\nXAG8Ddg2e3tV3TXzB3gF8PdV9eUB6pMkSYugr9CQ5CBgFLhuT1tVFXAtcOICXdcBd1XV+xve41Dg\n54H39VObJElaXAf2uf8a4ABg+6z27cDRc3VI8jzgdcBxje/xq8C3gL9aYJ/HAGzZsqVxyKVvenqa\nqampYZex36yk41lJxwIez1K2ko4FPJ6lasZn52P67ZtuoqBx5+QI4CvAiVV1w4z2C4DnV9W/n7X/\nwcAtwG9U1aZe2/uBkar6pXneYwuwqap+Z4E6fgX4i+bCJUnSbKdX1Qf76dDvTMM9wC7gsFnth/LQ\n2QeApwJHAVcnSa9tFUCS7wNHV9UDaxySvAD4ceA/7qWOTcDpwG3Ad/s7BEmSHtEeAzyZ7rO0L33N\nNAAk2QzcUFVn914HuB24tKounLXvo4CnzRrincDBwG8DX6iqnTP2vxx4RlU9p8/jkCRJi6zfmQaA\ni4ENSSaBG+muplgNXA6QZCNwZ1WdW1XfBz4/s3OSb9Ktn9wyq/3xwC/3xpMkSUtM36Ghqq7s3ZPh\nfLrTFDcBp1TV3b1djgR2ztd/Aaf1fn9ogL6SJGmR9X16QpIkPTL57AlJktTE0CBJkposy9Aw6AOz\nlpokL0hyVZKv9B7Q9bJh1zSoJG9NcmOSbyXZnuSvkvz4sOsaVJIzk9ycZLr3c32Snx12XftD789q\nd5KLh13LIJKsm+Phdp/fe8+lK8kTk3wgyT1JdvT+31s77LoG0fu3efafz+4k7xl2bf1KsirJO5J8\nqffn8sUkfzjsuvZFkoOTvDvJbb1j+mSSZ7f2X3ahYdAHZi1Rj6NbSHoWsNwXl7wAeA/wXOBngIOA\nv03y2KFWNbg7gLfQ3TZ9FPg48LEkxw61qn3UC9ivp/t7s5x9jm4h9p6H3D1/uOUMLskhwKeA7wGn\nAMcCvwvcN8y69sGz+dc/l8OBF9P9+3blMIsa0H8G3gj8JnAM8GbgzUneNNSq9s2fAyfT3evomcDf\nAdf2bt64V8tuIeQ894m4g+4+ERcMtbh9kGQ38IqqumrYtewPvRB3F/DCqvrksOvZH5J8A/i9lmeo\nLEW9O7ROAr8BnAd8tqrOGW5V/UuyDnh5VS3Lb+KzJXkX3V12f3rYtSyGJO8Gfr6qlt3MY5Krga9X\n1etntP0lsKOqXjO8ygaT5DHAvwC/WFV/M6P9M8A1VfW2vY2xrGYa9uGBWXr4HUL37eLeYReyr3pT\nlK+iux/Jp4ddzz74M+Dqqvr4sAvZD57eO633/5JckeRHh13QPvhF4DNJruyd2ptK8uvDLmp/6P2b\nfTrdt9vl6Hrg5CRPB0hyHPA84JqhVjW4A+meH/W9We330zhbN8jNnYap7wdm6eHXm/15N/DJqlq2\n55qTPJMuJOxJ56f2Hge/7PRCz7Popo6Xu810D7a7FTgCeDvwiSTPrKrvDLGuQT2FbvbnIro75j4X\nuDTJd6vqiqFWtu9OBUaADcMuZEDvAh4PbE2yi+6L9h9U1bK8n1BVfTvJp4Hzkmyl++z8Fbov3V9o\nGWO5hYb5hOW/JmAlWQ88gy6RL2db6Z7OegjwSmBjkhcut+CQ5Ei6EPfiqvrBsOvZV3seftfzuSQ3\nAl8G/hOwHE8drQJurKrzeq9vTvITdEFiuYeGM4C/rqqvD7uQAZ1G96H6Krq7Gz8L+G9JvlpVHxhq\nZYN7NfC/6B4+uROYAj4INJ3uW26hod8HZulhluRPgZ8HXlBVXxt2Pfui91yUL/VeTiV5DnA23T/m\ny8ko8G+AyRkPjjsAeGFvQdeja7ktbpqhqqaT/DMPfc7NcvE1YMusti3AnE8CXi6S/BjdouhXDLuW\nfXAB8F+r6sO91/+U5MnAW4FlGRp6D4l8UW+R+uOranuSDwHb9tIVWGZrGnrfkibpVn4CD0yFn0x3\n7klD1AsMLwdeVFW3D7ueRbAKePSwixjAtcBP0n1LOq738xm6b7HHLefAAA8s8Hwq3YfvcvQpHnp6\n9Wi62ZPl7Ay6L3PL9fw/dOuYZv/92M0y++ycS1Xd3wsMT6C7auejLf2W20wD7OWBWctJksfRfTva\n8+3vKb2FNvdW1R3Dq6x/SdYDY8DLgO8k2TMbNF1Vy+7x5UneCfw13ZU5P0S3mOungZcMs65B9M7z\nz35w3HeAb8x+cNxykORC4Gq6D9UnAX9EN806Mcy69sElwKeSvJXussTnAr9Od2nsstT7MverwOVV\ntXvI5eyLq4E/SHIH8E90U/jjwPuGWtU+SPISus+cW4Gn082mbKHxM3TZhYaGB2YtJ88G/p4uyRbd\nQijoFg2dMayiBnQm3TH8n1ntrwM2PuzV7LvD6Oo+ApgGbgFeskKuPIDlvQboSLpzsD8C3A18Evh3\nVfWNoVY1oKr6TJJT6RbdnUc3TXz2cl1s1/MzwI+yPNeYzPQm4B10Vx4dCnwV+O+9tuVqBPhjusB9\nL/CXwB9W1a6WzsvuPg2SJGk4lv15GUmS9PAwNEiSpCaGBkmS1MTQIEmSmhgaJElSE0ODJElqYmiQ\nJElNDA2SJKmJoUGSJDUxNEiSpCaGBkmS1OT/A3OXRPqG77PmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f203bc03750>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "a = 2\n",
    "b = 4\n",
    "\n",
    "value = float(2) / 4\n",
    "x = [a, b]\n",
    "y = [value, value]\n",
    "plt.plot(x, y)\n",
    "plt.xticks(range(10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Marginal Probability\n",
    "\n",
    "Sometimes we know the probability distribution over a set of variables and need to find the probability over just a subset of them. The probability over the subset is called **marginal probability distribution**.\n",
    "\n",
    "For example, if we know $P(\\text{x}, \\text{y})$, we can find $P(\\text{x})$ as:\n",
    "$$ \\forall x \\in \\text{x}, P(\\text{x} = x) = \\sum_{y} P(\\text{x} = x, \\text{y} = y)$$\n",
    "\n",
    "This image better illustrates how we calculate marginal probabilities: <br> <br>\n",
    "![marginal pdf](http://www.statisticshowto.com/wp-content/uploads/2014/02/conditional-distribution-3.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.  Conditional Probability\n",
    "\n",
    "This is one of the most important concept in probability theory. Sometimes we are interested in calculating the probability of some event (X) , \"conditioned\" on the knowledge that another event (Y) has happened. It is denoted by $P(X \\hspace{.1cm}| \\hspace{.1cm} Y)$. For example, at the beginning of a class, the probability that a student will score well on a subject is the same for all the students. But if we are given that a particular student does well on most of his subjects, then the probability that the student will score well, given that the student scores well on most of his subjects, increases. It can be computed by the formula:\n",
    "\n",
    "$$ P(\\text{y} = y \\hspace{.1cm}| \\hspace{.1cm} \\text{x} = x) = \\frac{P(\\text{y} = y , \\text{x} = x)}{P(\\text{x} = x)} $$\n",
    "\n",
    "The conditional probability is defined only when $P(\\text{x} = x) > 0$ as we can't condition on an event that never occurs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. The Chain Rule of Conditional Probabilities\n",
    "\n",
    "Any joint probability distribution over many random variables may be decomposed into conditional distributions over only one variable. Let's take a simple example before we look at the general expression. From the definition of conditional probability, we have: $ P(\\text{b}, \\text{c}) = P(\\text{b} \\hspace{.1cm} | \\hspace{.1cm} \\text{c}) P(\\text{c}) $. <br>\n",
    "We want to decompose $P(\\text{a}, \\text{b}, \\text{c})$. Applying the formula for conditional probability to $P(\\text{a}, \\text{b}, \\text{c})$, we get:\n",
    "\n",
    "$$ P(\\text{a}, \\text{b}, \\text{c}) = P(\\text{a} \\hspace{.1cm} | \\hspace{.1cm} \\text{b}, \\text{c})P(\\text{b}, \\text{c}) $$\n",
    "$$\\Rightarrow P(\\text{a}, \\text{b}, \\text{c}) = P(\\text{a} \\hspace{.1cm} | \\hspace{.1cm} \\text{b}, \\text{c})P(\\text{b} \\hspace{.1cm} | \\hspace{.1cm} \\text{c}) P(\\text{c})\n",
    "$$\n",
    "\n",
    "The general expression is given by:\n",
    "$$P(\\text{x}^{(1)}, ..., \\text{x}^{(n)}) = P(\\text{x}^{(1)}) \\prod_{i=2}^{n} P(\\text{x}^{(i)} \\hspace{.1cm} | \\hspace{.1cm} \\text{x}^{(1)},..., \\text{x}^{(i-1)}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Independence and Conditional Independence\n",
    "Two random variables x & y are said to be **independent** (x $\\perp$ y) if they satisfy: <br> <br>\n",
    "$$ \\forall x \\in \\text{x}, y \\in \\text{y}, P(\\text{x} = x, \\text{y} = y) = P(\\text{x} = x)P(\\text{y} = y) $$\n",
    "\n",
    "Using the definition of conditional probability, this implies: <br> <br>\n",
    "$$ P(\\text{x} = x, \\text{y} = y) = P(\\text{x} = x \\hspace{.1cm} | \\hspace{.1cm}  \\text{y} = y)P(\\text{y} = y)=  P(\\text{x} = x)P(\\text{y} = y)$$\n",
    "$$ \\Rightarrow P(\\text{x} = x \\hspace{.1cm} | \\hspace{.1cm}  \\text{y} = y)=  P(\\text{x} = x)$$\n",
    "<br>\n",
    "Similarly, $P(\\text{y} = y \\hspace{.1cm} | \\hspace{.1cm}  \\text{x} = x)=  P(\\text{y} = y)$. However, this scenario rarely occurs. The more commonly observed phenomenun is that of **conditional independence**. Two random variables x & y are said to be conditional independent (x $\\perp$ y | z) given a random variable z if: <br> <br>\n",
    "$$ \\forall x \\in \\text{x}, y \\in \\text{y}, \\forall z \\in \\text{z}, P(\\text{x} = x, \\text{y} = y \\hspace{.1cm} | \\hspace{.1cm}  \\text{z} = z) = P(\\text{x} = x \\hspace{.1cm} | \\hspace{.1cm} \\text{z} = z)P(\\text{y} = y \\hspace{.1cm} | \\hspace{.1cm} \\text{z} = z) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Expectation, Variance and Covariance\n",
    "\n",
    "The **expectation**, or **expected value**, of some function $f(x)$ with respect to a probability distribution $P(x)$ is the average, or mean value, that the function $f$ takes on, when $x$ is drawn from $P$. For discrete variables this can be computed with a summation: <br> <br>\n",
    "$$ \\mathbb{E}_{x \\sim P}[f(x)] = \\sum_{x} P(x)f(x) $$ \n",
    "\n",
    "For continuous variables, it is computed with an integral:\n",
    "$$ \\mathbb{E}_{x \\sim p}[f(x)] = \\int p(x)f(x) $$\n",
    "\n",
    "The expectation of $f(x)$ is simply represented as $\\mathbb{E}[f(x)]$. Expectations are linear: <br> <br>\n",
    "$$ \\mathbb{E}_{x}[\\alpha f(x) + \\beta g(x)] = \\alpha \\mathbb{E}_{x}[f(x)] + \\beta \\mathbb{E}_{x}[g(x)]$$\n",
    "\n",
    "**Variance** gives a measure of how much the values of random variable x vary (denoted by $\\sigma$). It is given by: <br> <br>\n",
    "$$ Var(f(x)) = \\mathbb{E}[(f(x) - \\mathbb{E}[f(x)])^2] $$\n",
    "\n",
    "The image below shows how the probability distribution of a Gaussian (explained later) varies with the variance. Lower the variance, more peakier the distribution. The square root of variance is called **standard deviation**.\n",
    "![variance](https://qph.ec.quoracdn.net/main-qimg-08986ffaf7525f799a2bded927ccb86e)\n",
    "\n",
    "**Correlation** measures how two variables are linearly related. **Covariance** additionally measures the scale of the variables as well. High values of correlation indicate that the variables are far from their respective means. A positive values indicate that both the variables tend to take high values simultaneously, whereas a negative value indicates that when one of the variables takes a high value, the other one tends to take a lower value. Covariance is given by:  <br> <br>\n",
    "$$ Cov(f(x), g(y)) = \\mathbb{E}[(f(x) - \\mathbb{E}[f(x)])(g(y) - \\mathbb{E}[g(y)])] $$\n",
    "\n",
    "Covariance and independence and related but distinct concepts. Independence implies zero covariance, but zero covariance doesn't necessarily imply independence. Hence, independence is a stronger condition. \n",
    "\n",
    "**Covariance matrix** of a random vector $\\boldsymbol{x} \\in \\mathbb{R} ^{n}$ is an $n$ x $n$ matrix denoted by $\\Sigma$, such that:\n",
    "$$ Cov(x)_{i, j} = Cov(x_i, x_j), i \\neq j$$\n",
    "$$ Cov(x)_{i, i} = Var(x_i)$$\n",
    "![cov matrix](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS6BwYwUXKlhcgBdq8hwYybgFuBqMgLSxqCSwUDBaEXMoXm0ryS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Common Probability Distributions\n",
    "\n",
    "#### 9.1 Bernoulli Distribution\n",
    "The Bernoulli distribution is a distribution over a single binary random variable. It is controlled by a single parameter $\\phi \\in [0,1]$, which gives the probability of the random variable being equal to 1. For e.g., in the event of tossing a coin, $p$ represents the probability that a heads shows up (Assuming heads is represented by 1). <br>\n",
    "**Properties**:\n",
    "- $P (x = 1) = \\phi $ <br>\n",
    "- $P (x = 0) = 1 − φ$ <br>\n",
    "- $P (\\text{x} = x) = \\phi^x(1 − \\phi)^{1−x}$ <br>\n",
    "- $E_x[x] = \\phi$ <br>\n",
    "- $Var_x(x) = \\phi(1 − \\phi)$ \n",
    "\n",
    "#### 9.2 Multinoulli Distribution\n",
    "The Multinoulli distribution is similar to the Bernoulli distribution, with the difference being that the discrete random variable can have k different states. It is parameterized by a vector $\\boldsymbol{p} \\in [0,1] ^{k-1}$, where $p_i$ indicates the probability of the $i^{th}$ state. The final $k^{th}$ state is given by $(1 - \\boldsymbol{1}^T \\boldsymbol{p})$ where $\\boldsymbol{1}^T \\boldsymbol{p} \\leq 1$.  <br>\n",
    "\n",
    "#### 9.3 Gaussian Distribution \n",
    "\n",
    "The most widely used distribution over real numbers is the **normal** or **gaussian distribution**: <br> <br>\n",
    "$$ \\mathcal{N}(x; \\mu, \\sigma) = \\sqrt{\\frac{1}{2 \\pi \\sigma^2}} exp(-\\frac{1}{2\\sigma^2}(x-\\mu)^2)$$\n",
    "\n",
    "where $\\mu$ is the mean (or peak value) of the distribution and $\\sigma^2$ denotes the variance. Feel free to play around with the values of *variance* and *mu* in the snippet below to see how the parameters effect the distribution. \n",
    "![gaussian](images/gaussian.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhQAAAFkCAYAAAB4sKK5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xm81mP+x/HXp0WpVLYkRgxZsqSyhbHFNIaxb4dkLBVC\nTqWSUoqapBBStpIl+5I1wlgmWSrbjDAmopSypJS0XL8/Pnc/p+Ns933uc657eT8fj/uR872/9/e8\nL/dZPuf6XouFEBARERGpjBqxA4iIiEj2U0EhIiIilaaCQkRERCpNBYWIiIhUmgoKERERqTQVFCIi\nIlJpKihERESk0lRQiIiISKWpoBAREZFKU0EhIiIilZZSQWFm3cxsjpmtMLPpZrZ3BV93mpmtNbPH\nSnhusJnNN7PlZvaime2QSjYRERGpfkkXFGZ2KjASGAi0Bt4HppjZZuW8rjkwAnithOf6ABcBXYF9\ngJ8T19wg2XwiIiJS/SzZzcHMbDrwVgihe+JjA74CRocQri3lNTWAV4G7gIOARiGEE4o8Px8YEUK4\nPvFxQ2AhcFYI4aGkWyUiIiLVKqkeCjOrDbQFXlp3LHhFMhVoV8ZLBwLfhhDGl3DN7YCmxa75E/BW\nOdcUERGRDFEryfM3A2rivQdFLQR2KukFZnYAcDbQqpRrNgVCKddsWso1NwU6AF8Av1Qgt4iIiLi6\nwLbAlBDCd+m6aLIFRWkMLwrWP2jWALgH6BxC+CEd10zoANyX5PVERETkN2cA96frYskWFIuBNcAW\nxY434fc9DADbA82BpxJjLSBxm8XMfsV7NRbgxcMWxa7RBJhVSo4vAO6991522WWXJJuQXQoLC7n+\n+utjx6gW+dLWbG5nCPD11zBrFsycCXPnwjffwKJF/tz6CoHf2lmnDmy5pT922gnatIFWraBBg+ps\nQfpl8/uZDLUzd3z88cd07NgREr9L0yWpgiKEsMrMZgDtgcnw/4My2wOjS3jJx8DuxY5dAzQALgG+\nCiGsNrMFiWt8kLhmQ2Bf4JZSovwCsMsuu9CmTZtkmpB1GjVqlPNtXCdf2ppt7fz5Z3jiCXjmGXjt\nNZg3D8xgjz1gt93g6KOheXPYdlvYaiuolfipcv75jRg7tg3LlsGXX8IXX/i/c+bAc8/BhAlQowa0\nbg2HHAKnnAJ77+3XzibZ9n6mSu3MSWkdMpDKLY9RwN2JwuJt/M+QesAEADObCHwdQugXQvgV+E/R\nF5vZj/hYzo+LHL4B6G9m/8UrpiHA18CTKeQTkUpauxb++U+YOBEefRSWLYO99oKCAjj4YDjwQGjc\nuOxrNGgAO+7o/13853MI8NlnXqC8+ircfz+MHOnnd+oEHTt6kSIi2SPpgiKE8FBizYnB+G2K94AO\nIYRFiVO2BlYnec1rzaweMA5oDLwOHJkoSESkmixdCmPHwk03wVdfQYsW0Lu3/4Lfbrv0fR4zLx52\n3BHOOw/WrIFXXvECZtgw6N8fDjsMLr8c2rfPvl4LkXyU0kqZIYQxIYRtQwgbhhDahRDeLfLcYSGE\nc8p47dlF16AocnxQCKFZCKFeCKFDCOG/qWQTkeT9+CMMGeK3La64Av78Z3jzTfjkExgwIL3FRElq\n1oTDD/eCYsEC//enn+CII6BdO3j66ZLGaIhIJtFeHhmuoKAgdoRqky9tzaR2LlvmBUPz5jB0KJxx\nBnz+OdxxB+y3X+V6BlJtZ4MGcOaZ8Pbb8PzzPibjb3/zsRZPP516nqqSSe9nVVI7pTxJr5SZCcys\nDTBjxowZ+TR4RiRtQoCHH4YePeC77+Cii6BnT2ha4sovcYXgYy2uuspvixx9NNx4I/zxj7GTiWSn\nmTNn0rZtW4C2IYSZ6bqueihE8szs2X4r4dRTfVbFxx/DiBGZWUyA95IcfDC89JIPEH3/fWjZ0guM\nFStipxORdVRQiOSJVav89sYee/gUzmeegccf93ET2cAMTjjBC6AePeCaa2DXXX02iojEp4JCJA/8\n979wwAHwj39Av37w0Ufw17/GTpWa+vV9vMeHH8I22/hskH79vGASkXhUUIjksBB8AanWreH77+Ff\n/4JBg6Bu3djJKm+nnfw2yNChfsvmgAO8cBKROFRQiOSoH3+E006Ds8+Gk07y5bL32Sd2qvSqWRP6\n9oVp0+CHH2DPPb2AEpHqp4JCJAd9+qlP+5wyBR58EMaPh402ip2q6uy9txdMp57qBVS3broFIlLd\nVFCI5JgXX4R99/X/fucd3yMjHzRoAHfeCePGwW23wV/+4rd5RKR6qKAQyREhwOjRcOSR3jsxfbov\nnZ1vunSBqVN9euk++8B//lP+a0Sk8lRQiOSAVauga1fo3t0fTz9d/uZduezgg713ZsMNvbh67rnY\niURynwoKkSz3yy9w4ok+TuKuu3zXzpo1Y6eKb7vtfLDmwQf70t333Rc7kUhuS2X7chHJED/9BMce\nC2+9BU895eMG5DcbbeSLd3Xu7PuDLFkCF14YO5VIblJBIZKlFi/28RKffQYvvAAHHhg7UWaqVcsH\nazZu7LM/fvjBF8LSlugi6aWCQiQLff21bzH+3Xe+9PSee8ZOlNlq1IBRo2CTTaB/fy8qRoxQUSGS\nTiooRLLM11/DQQfBmjXw+uuw446xE2UHM9/LpHFjuOQSH3ty000qKkTSRQWFSBZZsADat4e1a31L\n7+bNYyfKPhdf7LM/OneGOnXguutUVIikgwoKkSyxaJEXE8uWqZiorPPO8x6KdcXF1VfHTiSS/VRQ\niGSBH374bczEq6/C9tvHTpT9LroIVq6EXr18s7T+/WMnEsluKihEMtxPP0GHDvDVVz4Ac6edYifK\nHT17wooVPraibl0vLkQkNSooRDLYypVw3HE+NfTll2G33WInyj39+/vtj8sug0039c3FRCR5KihE\nMtTatf7Lbdo035uidevYiXLXkCG+rkfnztCsmfcIiUhytPS2SIa6/HJ44AG4914tWlXVzODmm32h\nsJNOgpkzYycSyT4qKEQy0M03w7XX+mJMJ50UO01+qFXLC7hddoGjjoIvvoidSCS7qKAQyTCPP+4L\nLxUWwqWXxk6TX+rX951a69Xz3orvv4+dSCR7qKAQySBvvw2nnw4nn+wLLkn1a9IEnn/e1/047jj4\n9dfYiUSygwoKkQwxfz4cf7zvy3H33b7/hMTRogVMnuy7uF50EYQQO5FI5tOPLJEM8MsvXkyYwWOP\n+ZoIEtf++8Ott8Ltt8OYMbHTiGQ+TRsViSwE6NoV3n/fN/vacsvYiWSdc86BDz6A7t2hZUs49NDY\niUQyl3ooRCK7/nqYOBHuvBP23jt2Ginuuuu8kDj5ZJgzJ3YakcylgkIkoilTfIXG3r3hjDNip5GS\n1KoFDz7o254fcwwsXRo7kUhmUkEhEskXX0BBga/KOHRo7DRSlk028UGaX34J556rQZoiJVFBIRLB\nypXehd64Mdx/P9SsGTuRlKdlS5gwAR5+2BceE5H1pVRQmFk3M5tjZivMbLqZlXrn18yON7N3zOwH\nM1tmZrPMrGOxc8ab2dpij2dTySaSDXr08MF+Dz/sRYVkhxNO8MXGevb0KaUi8pukCwozOxUYCQwE\nWgPvA1PMbLNSXvIdcDWwH7A7MB4Yb2ZHFDvvOWALoGniUZBsNpFscP/9Pg1x9Gho2zZ2GknW8OGw\n117ew/Tdd7HTiGSOVHooCoFxIYSJIYTZwPnAcuCckk4OIbwWQngyhPBJCGFOCGE08AFQfLujlSGE\nRSGEbxOPJSlkE8loH38MXbr4AMwuXWKnkVRssIEP0ly+HM4803eFFZEkCwozqw20BV5adyyEEICp\nQLsKXqM9sCPwarGnDjGzhWY228zGmNkmyWQTyXQ//+wbfTVvDmPH+iJWkp3+8Ae47z5fonvYsNhp\nRDJDsgtbbQbUBBYWO74Q2Km0F5lZQ2AeUAdYDVwYQni5yCnPAY8Cc4DtgWHAs2bWLlGwiGS9bt18\nlsA770CDBrHTSGV16AADBsCVV/r28gcfHDuRSFzpWinTgLJ+8S8FWgENgPbA9Wb2vxDCawAhhIeK\nnPtvM/sQ+Bw4BHglTRlForn/ft+fY8IE3x5bcsOVV8Krr0LHjr7S6SbqV5U8lmxBsRhYgw+eLKoJ\nv++1+H+JXob/JT78wMxaApcDr5Vy/hwzWwzsQBkFRWFhIY0aNVrvWEFBAQUFGs8pmWPOHLjgAl9z\nolOn2GkknWrWhHvugVatoHNneOQR3cqSzDJp0iQmTZq03rElS6pmiKIle0fBzKYDb4UQuic+NmAu\nMDqEMKKC17gT2C6EcFgpz28NfAkcG0J4uoTn2wAzZsyYQZs2bZLKL1KdVq+Ggw6Cb76B996DYvWv\n5IhHH/XxMbffDuedFzuNSNlmzpxJW59i1jaEMDNd103llsco4G4zmwG8jc/6qAdMADCzicDXIYR+\niY/7Au/itzDqAEcBHfHZIZhZfXwK6qPAArxXYjjwKTAlxXaJZIQhQ+Dtt33TLxUTuevEE72Hont3\nH0+x886xE4lUv6QLihDCQ4k1Jwbjtz7eAzqEEBYlTtkaH3i5Tn3glsTxFcBs4IwQwiOJ59cAewCd\ngMbAfLyQuDKEsCrpFolkiNdfh6uvhkGDoF2F5kBJNrv+enjtNb+1NX061KkTO5FI9Ur6lkcm0C0P\nyXQ//uj31Zs3h1de0dLa+WLWLNh3X7j4Yhg5MnYakZJV1S0P7eUhUgUuvhiWLIF771UxkU9at/Z1\nKUaN8kJSJJ+ooBBJs8ce80Lipptgm21ip5HqVljoa1KcfTb89FPsNCLVRwWFSBotXAhdu8Jxx/na\nBJJ/atSA8eN9n48ePWKnEak+KihE0iQEOP98X4dg3DitR5DPttvOb3vceSc880zsNCLVQwWFSJrc\ncw888YQXE02axE4jsZ13Hhx5pP+rXUklH6igEEmDr77ygZhnngnHHx87jWQCM7jjDli50vdxEcl1\nKihEKikEOPdc2GgjGD06dhrJJM2awS23+HbnDz4YO41I1VJBIVJJd94JL77o/zZuHDuNZJrTTvNl\nuS+6CBYtKv98kWylgkKkEubNg549fYpghw6x00gmMoObb4a1a+GSS2KnEak6KihEUhSC7yJar55W\nRZSybbEF3HgjPPAATJ4cO41I1VBBIZKiBx6Ap56CW2+FjTeOnUYy3RlnwF//6lOLf/wxdhqR9FNB\nIZKCRYu8+/qUU3wRK5HymMHYsbBsGfTqFTuNSPqpoBBJwSWX+C2Pm26KnUSyyR/+ACNG+ADeqVNj\npxFJLxUUIkmaPNlvd4werQWsJHmdO8Ohh/q/y5bFTiOSPiooRJLw009w4YVw1FFQUBA7jWSjGjXg\n9tt935eBA2OnEUkfFRQiSejf3wfUjRmjvTokddtvD4MGwQ03wMyZsdOIpIcKCpEKevttX09gyBBt\nSy6VV1gIu+0GXbrA6tWx04hUngoKkQpYvdp/8Ldu7Xt2iFRW7dq+kdzMmb48t0i2U0EhUgE33AAf\nfgi33Qa1asVOI7liv/18cbT+/X2DOZFspoJCpBxffOGD5y6+GNq2jZ1Gcs3Qob6xnHq+JNupoBAp\nQwi+9fQmm/jYCZF0a9TIpyA/+SQ8/njsNCKpU0EhUoZHHoFnn/XBmBttFDuN5KoTT4Sjj/YdSZcu\njZ1GJDUqKERKsXQpXHopHHMMHHts7DSSy9btSPrDDz6dVCQbqaAQKcVVV/kP+NGjYyeRfNC8OQwY\n4LuSfvhh7DQiyVNBIVKCjz7ymR39+/sPepHq0LMn7LCDj9sJIXYakeSooBApJgRfXnuHHfwHvEh1\n2WADX5Pi9dfhnntipxFJjgoKkWLuucd/oN9yC9SpEzuN5Jv27eG00+Cyy3yZd5FsoYJCpIgff/Qf\n5Ked5j/YRWIYORJWrPBbbiLZQgWFSBH9+/sP8pEjYyeRfNasmQ8KvvVWmDEjdhqRilFBIZIwa5bv\nInrVVf4DXSSmiy+GXXf1AZpr18ZOI1I+FRQi+EDMiy+Gli19cSGR2GrV8rUp3noLJk6MnUakfCoo\nRID77oN//Qtuusl3gRTJBAcdBAUF0KcPLFkSO41I2VRQSN5buhR694aTT4ZDD42dRmR9I0bAzz/7\nrTiRTKaCQvLekCE+u+O662InEfm9rbbywcI33QT/+U/sNCKlS6mgMLNuZjbHzFaY2XQz27uMc483\ns3fM7AczW2Zms8ysYwnnDTaz+Wa23MxeNLMdUskmkoxPPvEVMS+/HLbZJnYakZIVFsK228Ill2gF\nTclcSRcUZnYqMBIYCLQG3gemmNlmpbzkO+BqYD9gd2A8MN7MjihyzT7ARUBXYB/g58Q1N0g2n0hF\nhQDdu8PWW/vaEyKZqk4dL3xfegkeeyx2GpGSpdJDUQiMCyFMDCHMBs4HlgPnlHRyCOG1EMKTIYRP\nQghzQgijgQ+AA4uc1h0YEkJ4KoTwEdAJaAYcl0I+kQqZPBmmTIHrr4e6dWOnESnbUUf5o0cPWL48\ndhqR30uqoDCz2kBb4KV1x0IIAZgKtKvgNdoDOwKvJj7eDmha7Jo/AW9V9JoiyVq50n8wd+jg25OL\nZIMbboAFC+Daa2MnEfm9ZHsoNgNqAguLHV+IFwUlMrOGZrbUzH4FngIuDiG8nHi6KRCSvaZIZdxw\nA8yd670TZrHTiFTMDjv4eIprr/WvX5FMUitN1zG8KCjNUqAV0ABoD1xvZv8LIbxWiWtSWFhIo0aN\n1jtWUFBAQUFBhUJLfvrmG7j6al+BcJddYqcRSc4VV8CECb42xaRJsdNIpps0aRKTin2hLKmiRU0s\nJDFkOHHLYzlwYghhcpHjE4BGIYTjK3id24GtQwhHJm55fA7sGUL4oMg5/wRmhRAKS3h9G2DGjBkz\naNOmTYXziwCcc46Pn/jsM9h449hpRJJ3111w7rnwxhtwwAGx00i2mTlzJm3btgVoG0KYma7rJnXL\nI4SwCpiB9zIAYGaW+Hhakp+3TuKac4AFxa7ZENg3yWuKlOvdd2H8eF97QsWEZKu//x3atvVZStrn\nQzJFKrM8RgFdzKyTme0MjAXqARMAzGyimQ1dd7KZ9TWzw81sOzPb2cx6Ah2Be4pc8wagv5n9zcx2\nByYCXwNPptQqkRKEAJdeCrvtBp07x04jkroaNXwc0IwZ2udDMkfSYyhCCA8l1pwYDGwBvAd0CCEs\nSpyyNbC6yEvqA7ckjq8AZgNnhBAeKXLNa82sHjAOaAy8DhwZQvg1+SaJlOzBB32/jqlTfeMlkWx2\n4IFw2mm+KNuJJ8JGG8VOJPkuqTEUmUJjKCRZy5fDTjvBXnvB44/HTiOSHnPnws47+62PYcNip5Fs\nkRFjKESy1YgR8O232q9Dcss22/gqr6NGwf/+FzuN5DsVFJLz5s3zefvdu8P228dOI5JevXvD5ptD\n376xk0i+U0EhOe+KK6BePf9XJNfUrw9Dh8LDD/s0UpFYVFBITpsxA+6+GwYPhmJroInkjI4dfRpp\njx6aRirxqKCQnBUC9OwJLVtqmqjktho1fBzFO+9o9UyJRwWF5KwnnoBXX4WRIzVNVHLfQQfBCSf4\nWArtRioxqKCQnPTrrz76vUMH+MtfYqcRqR7XXgsLF3pvhUh1U0EhOenmm2HOHE0Tlfyy/fZwySXw\nj3/A/Pmx00i+UUEhOee773wQZpcuvsy2SD7p3x/q1vV/RaqTCgrJOYMH+0j3q66KnUSk+jVu7F/7\nEybA++/HTiP5RAWF5JTPPoMxY3x/gyZNYqcRiaNLF2jRAnr18tlOItVBBYXklL59YcstfVdRkXxV\nu7YP0Jw6FaZMiZ1G8oUKCskZb7wBjz3mqwZuuGHsNCJxHXMMHHyw91KsXl3++SKVpYJCcsK6Raza\ntIHTT4+dRiQ+M5/l9O9/w/jxsdNIPlBBITnhwQfh7bd9Easa+qoWAWCvveCMM2DAAFi2LHYayXX6\n0StZ75dffOzEMcfAIYfETiOSWYYOhR9/9DEVIlVJBYVkvZtvhq+/huHDYycRyTzbbAOFhX77Y968\n2Gkkl6mgkKz23Xdw9dXQtSvsvHPsNCKZqW9f3+Z8wIDYSSSXqaCQrHb11b6I1cCBsZOIZK5Gjfx7\nZMIE+OCD2GkkV6mgkKz1v//BLbdAnz5axEqkPF27wg47+PeLSFVQQSFZq18/2Hxzvz8sImWrXds3\nDXv+eV/wSiTdVFBIVnr7bZ8qOmQI1KsXO41Idjj+eNh/f7jsMr9VKJJOKigk64TgPxB33x3OOit2\nGpHsYQYjRsB778F998VOI7lGBYVknaeegtde83n1NWvGTiOSXfbfH048Ea64AlasiJ1GcokKCskq\nq1f7oLLDD4cOHWKnEclOw4bBN9/A6NGxk0guUUEhWeXOO+GTT7x3wix2GpHs1KIFnH++r6K5eHHs\nNJIrVFBI1li2zOfSn3EGtG4dO41IdrvySh+PdM01sZNIrlBBIVlj5Ejfk+Dqq2MnEcl+m2/utw9v\nucXXdBGpLBUUkhUWLvTR6RdfDM2bx04jkhsKC72w6N8/dhLJBSooJCtcdRVssIEvZiUi6VGvHgwe\nDJMmwbvvxk4j2U4FhWS8Tz6B227zaW4bbxw7jUhuOessaNkSevf2MRUiqVJBIRmvXz/Yaivo1i12\nEpHcU6sWDB8Or7ziy3KLpEoFhWS0adPgscd8JHrdurHTiOSmo46Cgw7yQZpr1sROI9kqpYLCzLqZ\n2RwzW2Fm081s7zLOPc/MXjOz7xOPF4ufb2bjzWxtscezqWST3LFuie0994TTT4+dRiR3rVuS+8MP\n4Z57YqeRbJV0QWFmpwIjgYFAa+B9YIqZbVbKSw4G7gcOAfYDvgJeMLMti533HLAF0DTxKEg2m+SW\nJ5/0Hoprr4Ua6ksTqVL77AMnnwwDBmhJbklNKj+mC4FxIYSJIYTZwPnAcuCckk4OIZwZQhgbQvgg\nhPApcF7i87YvdurKEMKiEMK3iceSFLJJjli9Gvr2hSOO8IeIVL2hQ2HBAi3JLalJqqAws9pAW+Cl\ndcdCCAGYCrSr4GXqA7WB74sdP8TMFprZbDMbY2abJJNNcstdd/nsjuHDYycRyR877ABdu/peH999\nFzuNZJtkeyg2A2oCC4sdX4jfpqiI4cA8vAhZ5zmgE3AY0Bu/TfKsmXZryEc//6wltkViufJKH5g5\ndGjsJJJtaqXpOgaUO4PZzPoCpwAHhxB+XXc8hPBQkdP+bWYfAp/j4y5eKe16hYWFNGrUaL1jBQUF\nFBRo+EU2u/56+P57LbEtEkOTJj4Y+pprfGXabbeNnUgqY9KkSUyaNGm9Y0uWVM2IAgtJrGSSuOWx\nHDgxhDC5yPEJQKMQwvFlvLYX0A9oH0KYVYHP9S1wRQjh9hKeawPMmDFjBm3atKlwfsl8ixbB9tvD\neefBqFGx04jkp2XL/PbHEUdo1kcumjlzJm3btgVoG0KYma7rJnXLI4SwCphBkQGVidsS7YFppb3O\nzC4DrgA6VLCY2BrYFPgmmXyS/YYM8RkdV1wRO4lI/mrQAAYNgvvug/fei51GskUqszxGAV3MrJOZ\n7QyMBeoBEwDMbKKZ/f/dNzPrDQzBZ4HMNbMtEo/6iefrm9m1ZravmTU3s/bAE8CnwJTKNE6yy+ef\nw9ixPrtj001jpxHJb+eeCy1a+GJXIhWRdEGRGO/QExgMzAL2wHseFiVO2Zr1B2hegM/qeASYX+TR\nM/H8msQ1ngQ+AW4H3gEOSvSISJ644gq/f9u9e+wkIlK7ts/2eOEFmDq1/PNFUhqUGUIYA4wp5bnD\nin28XTnX+gX4Syo5JHe88w48+CDceSdsuGHsNCICcPzx0K6dbxz27rtaYE7Kpi8PiS4E71bddVff\n+VBEMoOZr1Q7axY88EDsNJLpVFBIdM8/7zsd/uMfULNm7DQiUtSBB8Ixx/gtyZUrY6eRTKaCQqJa\ns8Z7Jw46yHc8FJHMM2wYzJ3rg6ZFSqOCQqK6917f4XD4cO9eFZHM07IlnHOOT+uuojWRJAeooJBo\nfvnFdzY88UTYb7/YaUSkLFddBcuX+5gKkZKooJBobroJ5s/XngEi2aBZMygs9KXx582LnUYykQoK\nieL7772Q6NIFdtwxdhoRqYjevaFePV9FU6Q4FRQSxbBhsGqV72woItmhUSPo3x/uugv+85/YaSTT\nqKCQajd3rt/u6NkTmlZ003sRyQgXXADNm8Pll8dOIplGBYVUuwED/C+dXr1iJxGRZNWp41ubT54M\nb7wRO41kEhUUUq3ef9+3Qx44EDbaKHYaEUnFqadC27Zw2WW+0q0IqKCQata3L+ywA3TuHDuJiKSq\nRg1fO2b6dHjssdhpJFOooJBq89JLvsz2sGG+k6GIZK/27aFDBx9LsUr7QgsqKKSarF3rU8722w9O\nOCF2GhFJh+HD4b//hTvuiJ1EMoEKCqkWDz4IM2f6KntaYlskN7RqBWee6etSLF0aO43EpoJCqtzK\nlb5T4THHwJ/+FDuNiKTTuv09Ro6MnURiU0EhVW7sWPjySx87ISK5ZZtt4OKL4brrYMGC2GkkJhUU\nUqV+/NH/gjn3XN+xUERyT79+sMEGvoGY5C8VFFKlhg+HFSu09r9ILtt4Y7+tefvtMHt27DQSiwoK\nqTJffQU33OBLbDdrFjuNiFSlbt1g6621JHc+U0EhVebKK301zMsui51ERKpa3bq+JPcTT2hJ7nyl\ngkKqxAcfwN13+60OLbEtkh8KCqB1ay3Jna9UUEiV6NNHS2yL5JsaNWDECC3Jna9UUEjaTZ3qS2z/\n4x9aYlsk37RvD3/5i+/boyW584sKCkmrdUtst2sHxx8fO42IxDB8OHz+OYwbFzuJVCcVFJJW998P\ns2Z5t6eW2BbJT3vsAWedBYMHw08/xU4j1UUFhaTNihU+F/2EE+CAA2KnEZGYhgyBZcu8t0LygwoK\nSZvRo2H+fB87ISL5beutobAQRo3yNWkk96mgkLRYvBiGDoXzz4cWLWKnEZFM0KePTxsfMCB2EqkO\nKigkLYYM8X+vvDJuDhHJHA0b+lo0EyfC++/HTiNVTQWFVNp//wtjxvg0sc03j51GRDJJ587ea6kV\nc3OfCgppehOOAAAgAElEQVSptMsvh6ZN4dJLYycRkUxTu7YPzHzxRZgyJXYaqUoqKKRSpk2DRx7x\nNfw33DB2GhHJRMceC3/6E/TqBWvWxE4jVSWlgsLMupnZHDNbYWbTzWzvMs49z8xeM7PvE48XSzrf\nzAab2XwzW544Z4dUskn1CcF/QLRqBR07xk4jIpnKDK67Dj76CCZMiJ1GqkrSBYWZnQqMBAYCrYH3\ngSlmtlkpLzkYuB84BNgP+Ap4wcy2LHLNPsBFQFdgH+DnxDU3SDafVJ9HHoE334SRI30NfxGR0uyz\nD5x2ms/4WLYsdhqpCqn8GigExoUQJoYQZgPnA8uBc0o6OYRwZghhbAjhgxDCp8B5ic/bvshp3YEh\nIYSnQggfAZ2AZsBxKeSTarBypU8JO+ooX7tfRKQ8w4bBd995b4XknqQKCjOrDbQFXlp3LIQQgKlA\nuwpepj5QG/g+cc3tgKbFrvkT8FYS15RqdvPNMHcuXHtt7CQiki223Ra6d/el+efPj51G0i3ZHorN\ngJrAwmLHF+JFQUUMB+bhRQiJ14VKXlOq0XffwdVX+3Swli1jpxGRbNKvnw/g1mJXuSddd74NLwrK\nPsmsL3AKcFwI4dd0XFOq35AhPlL7qqtiJxGRbNO4sS92NX68FrvKNbWSPH8xsAbYotjxJvy+h2E9\nZtYL6A20DyH8u8hTC/DiYYti12gCzCrrmoWFhTRq1Gi9YwUFBRQUFJT1MqmEzz6DW27xXQSbNImd\nRkSyUdeuftu0Vy944QXtTFyVJk2axKRJk9Y7tmTJkir5XOZDIJJ4gdl04K0QQvfExwbMBUaHEEaU\n8prLgH7An0MI75Tw/HxgRAjh+sTHDfHiolMI4eESzm8DzJgxYwZt2rRJKr9UzgknwIwZMHu21p0Q\nkdRNnuzrUzzzDPz1r7HT5JeZM2fStm1bgLYhhJnpum4qtzxGAV3MrJOZ7QyMBeoBEwDMbKKZDV13\nspn1Bobgs0DmmtkWiUf9Ite8AehvZn8zs92BicDXwJOpNEqqxuuvw+OP+yZgKiZEpDL+9jc45BBf\nknv16thpJB2SLihCCA8BPYHB+C2JPYAOIYRFiVO2Zv3BlBfgszoeAeYXefQscs1rgZuAcfjsjg2B\nIyswzkKqydq1vhXxXnuB7iiJSGWZ+Ro2H38Mt98eO42kQ7JjKAAIIYwBxpTy3GHFPt6ugtccBAxK\nJY9UvXvv9Vsdr7+uRaxEJD3atIGzzvJdigsKfMCmZC/9apBy/fyzbwB20klw4IGx04hILrnmGli+\n3P+V7KaCQso1YgQsXuw7BoqIpFOzZr7q7o03wuefx04jlaGCQso0b56vhnnppfDHP8ZOIyK5qFcv\n2GILLywke6mgkDL16wcNGsAVV8ROIiK5ql493+fj0Ufhtddip5FUqaCQUr37Lkyc6CtjNmwYO42I\n5LLTT4e994YePXxWmWQfFRRSohB8muhuu8G558ZOIyK5rkYNuP56n012zz2x00gqVFBIiR55BN54\nw+eJ10ppcrGISHIOOABOOcVnlS1bFjuNJEsFhfzOihW+et3RR8Of/xw7jYjkk+HD4fvv4R//iJ1E\nkqWCQn5n5EiYP9//FRGpTttu63/QXHcdzJkTO40kQwWFrGfePB9tfcklsOOOsdOISD7q0wc23RR6\n946dRJKhgkLW07cv1K8PAwbETiIi+apBA7/18cgj8OqrsdNIRamgkP83fbrv2XHNNdCoUew0IpLP\nTj8d9t3XF9VbsyZ2GqkIFRQC+LzvSy+FVq3gnHNipxGRfFejhi/H/d57cNddsdNIRaigEADuuw/e\nesu/gWvWjJ1GRMR7KM4801fqXbIkdhopjwoKYdkyHztx0klw8MGx04iI/GbYMN+NdPDg2EmkPCoo\nhGuu8XnfI0bETiIisr6ttvI9hUaPhtmzY6eRsqigyHOffQajRnkPxbbbxk4jIvJ7PXpA8+bQvbtv\nCyCZSQVFnuvRA7bcUvO9RSRz1a3r+3y88AJMnhw7jZRGBUUee/ZZePppXxFzww1jpxERKd3RR8Nf\n/uKbFv7yS+w0UhIVFHlq5UqfJtq+PZxwQuw0IiJlM4MbboCvvtK2AJlKBUWeuvFG+N///F+z2GlE\nRMq3007+h9DQoV5YSGZRQZGH5s+HIUPgootg111jpxERqbgBA6BhQ99ATDKLCoo81Lu3j5kYNCh2\nEhGR5DRs6Pt8PPggvPJK7DRSlAqKPPPaa74q5vDh0Lhx7DQiIsnr2BH23997WVetip1G1lFBkUdW\nrYJu3aBdOzjrrNhpRERSU6MG3HKLL3Q1enTsNLKOCoo8cvPN8J//+DdiDb3zIpLF9tzT/0AaNMjH\nhUl8+rWSJ775BgYOhAsugNatY6cREam8wYOhXj3o1St2EgEVFHnjsst8tbkhQ2InERFJj8aN4dpr\nYdIkDdDMBCoo8sCrr/42EHPjjWOnERFJnzPPhAMO0ADNTKCCIsdpIKaI5LKiAzRvvDF2mvymgiLH\n3XgjfPyxBmKKSO5q1cr/cLrqKvj669hp8pd+xeSwuXN9BPRFF2kgpojktiFDoEEDX5pb4lBBkcO6\nd/dV5TQQU0RyXaNGvsX5o4/6TspS/VIqKMysm5nNMbMVZjbdzPYu49yWZvZI4vy1ZnZJCecMTDxX\n9PGfVLKJe/ppeOIJ352vYcPYaUREqt6pp8IRR3iv7PLlsdPkn6QLCjM7FRgJDARaA+8DU8xss1Je\nUg/4HOgDfFPGpT8CtgCaJh4HJptN3PLl/g3VoQOcfHLsNCIi1cPMx4vNnw/XXBM7Tf5JpYeiEBgX\nQpgYQpgNnA8sB84p6eQQwrshhD4hhIeAX8u47uoQwqIQwreJx/cpZBP8FseCBb4yprYmF5F80qIF\nXH45jBjhA9Kl+iRVUJhZbaAt8NK6YyGEAEwF2lUySwszm2dmn5vZvWb2h0peLy/9+99w3XVwxRWw\nww6x04iIVL8+faB5c18ZOITYafJHsj0UmwE1gYXFji/Eb1Okajrwd6AD3uOxHfCamdWvxDXzztq1\ncOGF8Mc/+hblIiL5qG5dGDPGF/WbODF2mvyRrlkeBqRcB4YQpoQQHg0hfBRCeBH4K7AxcEqa8uWF\n8eN9e/Jbb4U6dWKnERGJ54gjoKAAevaExYtjp8kPtZI8fzGwBh88WVQTft9rkbIQwhIz+xQos9O+\nsLCQRo0arXesoKCAgoKCdEXJGgsX+gY5f/87HHZY7DQiIvFdfz3ssosXFXffHTtNHJMmTWLSpEnr\nHVuyZEmVfC4LSd5gMrPpwFshhO6Jjw2YC4wOIYwo57VzgOtDCGXuYG9mDYAvgYEhhJtLeL4NMGPG\njBm0adMmqfy5qqAApk715Wc33TR2GhGRzHDXXXDuufDii3D44bHTZIaZM2fStm1bgLYhhJnpum4q\ntzxGAV3MrJOZ7QyMxaeGTgAws4lmNnTdyWZW28xamdmewAbAVomPty9yzggzO8jMmpvZ/sDjwGpg\n/bJKSvTcc/DAA16Nq5gQEfnN2WfDIYfA+efDihWx0+S2pAuKxPTPnsBgYBawB9AhhLAoccrWrD9A\ns1nivBmJ472AmcDtRc7ZGrgfmA08ACwC9gshfJdsvnzz888+kvmII+CMM2KnERHJLGYwbpzv8aFV\ng6tWsmMoAAghjAHGlPLcYcU+/pJyCpcQQv4NekiTgQPh22/h5Ze15oSISEl23BH69/fNw047DfbY\nI3ai3KS9PLLYzJl+m2PQIJ8qKiIiJevd2wuLLl1gzZrYaXKTCoostWqVDzTafXcoLIydRkQks22w\nAdx+O7z9tq8iLOmngiJLXXcdfPAB3Hkn1K4dO42ISObbf3/o1g369YM5c2KnyT0qKLLQ7Nl+L7BX\nL/CZPyIiUhFDh8Jmm/mtDy3LnV4qKLLMmjV+q2ObbXzshIiIVNxGG/mtj6lTfY0KSR8VFFnmlltg\n2jS/1bHhhrHTiIhknz//2VcV7tnTtzqX9FBBkUW++MK35e3WDf70p9hpRESy18iRvomYdiRNHxUU\nWSIE6NzZ7/0NGxY7jYhIdttkE9+RdPJkeOih2GlygwqKLHHXXX7P77bb/B6giIhUzgknwEknwUUX\n+QKBUjkqKLLA3Lm+1sTZZ0OHDrHTiIjkjptv9h7gCy/UrY/KUkGR4ULwWR2NG/uqmCIikj5bbOG3\nPh59FB58MHaa7KaCIsONG+e3Ou68Exo1ip1GRCT3nHKKP7p1gwULYqfJXiooMticOb54Vdeuvpuo\niIhUjVtugVq1fJtz3fpIjQqKDLV2LZxzDmy+OYwYETuNiEhu22wzGDsWnnwS7rsvdprspIIiQ40Z\nA//8p8/u0KwOEZGqd/zxcMYZcPHFWvAqFSooMtCnn0KfPn4/79BDY6cREckfo0f7KsTnnqtbH8lS\nQZFhVq2Cjh1hq61g+PDYaURE8ssmm8Add8Dzz/stEKk4FRQZZuhQmDkT7r0X6tePnUZEJP/89a++\nJHfPnt5jLBWjgiKDvPUWDBkCAwbAPvvETiMikr9GjICtt/Ye41WrYqfJDiooMsTPP8OZZ0LbttCv\nX+w0IiL5rX597ymeOROuuSZ2muyggiJDXHYZzJsH99wDtWvHTiMiIvvs4z3GV1/tPchSNhUUGeC5\n5+DWW3073R13jJ1GRETW6dfPe47PPNN7kqV0KigiW7gQ/v53OPJIXxFTREQyR+3afutj3jy49NLY\naTKbCoqI1q71YsIMxo/3f0VEJLO0aOHrU9xxBzzySOw0mUsFRUQ33uhzne++23e8ExGRzHTOOXDy\nydC5M8ydGztNZlJBEcmsWb4aZo8e0KFD7DQiIlIWM9/9uWFDX5579erYiTKPCooIfv4ZCgpgt918\nISsREcl8G2/sG4dNm6appCVRQRFB9+7w1VcwaRLUqRM7jYiIVNSBB/pU0sGD4Y03YqfJLCooqtmD\nD8Kdd/oAn512ip1GRESS1b8/tGvntz6+/z52msyhgqIaffaZD+g59VQf4CMiItmnVi24/35YuhTO\nPlu7kq6jgqKa/PILnHIKNG0Kt92mKaIiItlsm218ht7kyXDDDbHTZAYVFNWksBA+/hgeeshHCYuI\nSHb729+gVy/o3RumT4+dJj4VFNXggQdg7Fhfd2LPPWOnERGRdBk6FPbay29l5/t4ChUUVezTT33c\nREEBdOkSO42IiKRT7do+2H7ZMl/5OJ/HU6RUUJhZNzObY2YrzGy6me1dxrktzeyRxPlrzeySyl4z\nW6xY4eMmmjXzBVE0bkJEJPesG0/x1FO+yWO+SrqgMLNTgZHAQKA18D4wxcw2K+Ul9YDPgT7AN2m6\nZsYLAS68ED75xMdNbLRR7EQiIlJVjj7ax1L07Quvvho7TRyp9FAUAuNCCBNDCLOB84HlQIkTIUMI\n74YQ+oQQHgJ+Tcc1s8Ftt8GECf5vq1ax04iISFW75ho46CDvmZ43L3aa6pdUQWFmtYG2wEvrjoUQ\nAjAVaJdKgKq4ZmxvvQUXXwzdusGZZ8ZOIyIi1aFWLR+EX7u2byT2a2l/QueoZHsoNgNqAguLHV8I\nNE0xQ1VcM5pvv4WTTvJRv6NGxU4jIiLVqUkT3+L83Xd988d8UitN1zEg3WNby71mYWEhjRo1Wu9Y\nQUEBBQUFaY5SMatXw2mneVX68MOwwQZRYoiISET77efbK1xwAey7b9ye6kmTJjFp0qT1ji1ZsqRK\nPleyBcViYA2wRbHjTfh9D0OVX/P666+nTZs2KX7a9LviCnjtNXjpJdhqq9hpREQklq5dfbGrrl1h\n993jrUFU0h/ZM2fOpG3btmn/XEnd8gghrAJmAO3XHTMzS3w8LZUAVXHNGO6/H669FkaMgIMPjp1G\nRERiMoNbb4VddoHjjoNFi2InqnqpzPIYBXQxs05mtjMwFp8aOgHAzCaa2dB1J5tZbTNrZWZ7AhsA\nWyU+3r6i18x0774L554LnTrBpZfGTiMiIplgww3hiSd8TaKTTsr9QZpJj6EIITyUWB9iMH6b4j2g\nQwhhXf21NbC6yEuaAbP4bTxEr8TjVeCwCl4zY33zjVefrVpp8SoREVnfH/4Ajz0Ghx7qs//Gjs3d\n3xMpDcoMIYwBxpTy3GHFPv6SCvSElHXNTPXLL3DCCbB2rX/B1K0bO5GIiGSaAw7w2x/nned/fF54\nYexEVSNdszzyTgg+gnfWLB+I2axZ7EQiIpKpzj0X3n8funf3cRWHHho7Ufppc7AUjRrlK2HecQfs\ns0/sNCIikulGjfJB+yefDJ9/HjtN+qmgSMETT8Bll/m67R07xk4jIiLZoFYt39tpk03gqKPghx9i\nJ0ovFRRJevddOOMMHzsxbFjsNCIikk022QSeecankZ5wQm7N/FBBkYS5c+Fvf4PddoN77oEa+r8n\nIiJJatHCe7qnTfOFr0K615mORL8SK+inn3x72jp1YPJkn18sIiKSij/9Ce6808fi5Upvt2Z5VMDq\n1XDqqfDll/Dmm7BF8UXCRUREktSxI/z3v75tw/bb+++ZbKaCohwhwPnnw9Sp8Oyz0LJl7EQiIpIr\nBg70GR+dOvkfq4ccEjtR6nTLoxwDB3q31F13wRFHxE4jIiK5xMx/xxx0EBx7LHzwQexEqVNBUYZb\nb4UhQ2D48Ljbz4qISO7aYAN49FG/7XHkkX57PRupoCjFY49Bt26+qtlll8VOIyIiuaxhQ7+tXqcO\n/OUv8N13sRMlTwVFCV5/HU4/3VczGzUqdzdyERGRzNG0KUyZAosX+6zC5ctjJ0qOCopi3nvP15rY\nf3+YOFFrTYiISPVp0cJ7Kj74IPu2PNevyyJmz4Y///m3RUfq1ImdSERE8s3ee/vvoJde8qmla9bE\nTlQxKigSvvgCDj8cmjSB55/3+1kiIiIxHHEEPPigj+fr3BnWro2dqHwqKID586F9e6hbF158ETbd\nNHYiERHJd8cd5ytpTpgAhYWZv0R33i9stXixV4IrV8Ibb8CWW8ZOJCIi4jp2hGXL4IILoFEjGDw4\ndqLS5XVB8f33PmZi0SJ47TXYdtvYiURERNZ3/vmwdCn07u1j+664InaikuVtQfH9994zMXcuvPwy\n7Lxz7EQiIiIlu+wy+OUX6N/fZx9efnnsRL+XlwXFDz94MfHllz6Kdo89YicSEREp24ABPjizXz9f\nH6lv39iJ1pd3BcW6YuKLL7xnolWr2IlEREQqZuBAH5x5+eVeVPTpEzvRb/KqoPjhBx8zMWeO90yo\nmBARkWwzaJAXFet6KDKlqMibguLbb72Y+Oor34p8zz1jJxIREUnNoEH+b9++sGKF91zE3iYiLwqK\nefN80aoffoBXX4XddoudSEREJHVmcNVVsOGGfvtj6VK47rq4RUXOFxRz5viiVatX+6ZfLVrETiQi\nIpIefftCgwZw8cXw888wZky8PahyuqCYPdt7JjbcEF55BZo3j51IREQkvS66COrXh/PO86Ji/Hio\nFeG3e84WFO+8A0cd5XtzvPiiVsAUEZHcdfbZ3lNx+unw00/wwAP+x3R1ysm9PKZMgUMPhT/+0cdM\nqJgQEZFcd/LJMHmyTzw4/HBfwLE65VxBcc89cPTRcMghPjVUG32JiEi+OPJIX2Ppk0/gwAN9Nejq\nkjMFRQgwYgR06uSPJ57we0oiIiL5ZN99Ydo0n066//7w0UfV83lzoqBYswYuvdQ3TunfH+64I86A\nFBERkUyw445eVGy+ufdUvPJK1X/OrC8oli6FY4+FW27x6TJDhsRf3ENERCS2Lbf0cYR77+0LO951\nV9V+vqwuKL75xiuv11+HZ57x/eJzzaRJk2JHqDb50la1M7eonbkl19rZsCE8+yyce64/+vTxDcaq\nQkoFhZl1M7M5ZrbCzKab2d7lnH+ymX2cOP99Mzuy2PPjzWxtscez5eXo1Mmnx0ybBh06pNKSzJdr\nX9xlyZe2qp25Re3MLbnYztq14dZbYeRIH2vYu3fVfJ6kCwozOxUYCQwEWgPvA1PMbLNSzm8H3A/c\nDuwJPAE8YWYti536HLAF0DTxKCgvy1ZbwVtvwa67JtsKERGR/GEGPXr4hIU336yaz5FKD0UhMC6E\nMDGEMBs4H1gOnFPK+d2B50IIo0IIn4QQBgIzgYuKnbcyhLAohPBt4rGkvCDjxvnCVSIiIlK+Y46p\nurEUSRUUZlYbaAu8tO5YCCEAU4F2pbysXeL5oqaUcP4hZrbQzGab2Rgz26S8PHXqVDi6iIiIADvt\nVDXXTXZy5WZATWBhseMLgdIiNi3l/KZFPn4OeBSYA2wPDAOeNbN2iYKluLoAH3/8cVLhs9GSJUuY\nOXNm7BjVIl/aqnbmFrUzt+RDO4v87qyb1guHECr8ALYE1gL7Fjt+LTCtlNesBE4tduxCYH4Zn2e7\nxOc5tJTnTweCHnrooYceeuiR8uP0ZGqA8h7J9lAsBtbggyeLasLveyHWWZDk+YQQ5pjZYmAH4JUS\nTpkCnAF8AfxSbmoRERFZpy6wLf67NG2SKihCCKvMbAbQHpgMYGaW+Hh0KS97s4Tnj0gcL5GZbQ1s\nCnxTSo7v8JkjIiIikrxp6b5gKgtUjwLuThQWb+OzPuoBEwDMbCLwdQihX+L8G4FXzawH8Aw+HbQt\n0Dlxfn18CuqjeG/GDsBw4FPSXD2JiIhI1Ui6oAghPJRYc2IwfivjPaBDCGFR4pStgdVFzn/TzAqA\naxKPz4BjQwj/SZyyBtgD6AQ0BubjhcSVIYRVKbVKREREqpWVPIlCREREpOKyei8PERERyQwqKERE\nRKTSsqKgMLPmZnaHmf3PzJab2WdmNiixcmdZr6tjZreY2WIzW2pmj5hZRi/WbWb9zOxfZvazmX1f\nwdektLlaTKm0M/G6wWY2P/F18KKZ7VCVOSvLzDY2s/vMbImZ/ZD4Oq5fzmv+Wey9XGNmY6orc0Wl\ne5PATJVMO83srCLv2br3b3l15k2Fmf3JzCab2bxE5mMq8JpDzGyGmf1iZp+a2VnVkbUykm2nmR1c\nws/WNZn8e8TMLjezt83sp8Tq04+b2Y4VeF2lvz+zoqAAdgYMnxnSEp9Zcj4+yLMsNwBHAScCBwHN\n8Nkkmaw28BBwa5KvS3pztciSbqeZ9cH3gOkK7AP8jG9Mt0GVJEyP+4Fd8KnTR+Ffh+PKeU0AbuO3\n93NLoIr2B0xNFW4SmFGSbWfCEn77PmwKNK/qnGlQHx9g3w3/+iuTmW0LPI1vw9AKn813h5kdUXUR\n0yKpdiYEoAW/vZ9bhhC+rZp4afEn4CZgX+Bw/GftC2a2YWkvSNv3ZzpXyarOB9AL+G8ZzzfEV+k8\nvsixnfAVOPeJnb8C7TsL+L6C544HHouduRraOR8oLPYerwBOid2OUvLunPh6a13kWAd8FlTTMl73\nCjAqdv5y2jYduLHIxwZ8DfQu5fwHgMnFjr0JjIndljS3s8Jfz5n6SHzNHlPOOcOBD4odmwQ8Gzt/\nmtt5MD4TsWHsvJVo52aJth5Yxjlp+f7Mlh6KkjQGyuoqb4tPiy26kdknwFxK38gsmyW9uVo2MbPt\n8L8Oir6fPwFvkbnvZzvghxDCrCLHpuJ/8exbzmvPMLNFZvahmQ0t66+L6mZVu0lgxkixnQANzOwL\nM5trZhnfC5Oi/ciy97MSDHgvcav1BTPbP3agJDXGf+aU9fsyLd+fqSxsFV3ivvlFQI8yTmsK/Jr4\npVNU8Y3JckGym6tlo6b4N0V5G81lkqbAel2jIYQ1iTEjZWW+D/gS75HZA98rZ0fgpCrKmayq2iQw\n06TSzk+Ac4APgEbAZcA0M9s1hDCvqoJGUNr72dDM6oQQVkbIVBW+wW+xvgvUwW+7/9PM9gkhvBc1\nWQWYmeG3/t8Iv639VJK0fH9GLSjMbBjQp4xTArBLCOHTIq/ZCv8F+mAIIZVd3Y2K3ztLi1TamYwQ\nwkNFPvy3mX0IfA4cQsl7oVSJqm5naZ+WDH0/y7oEZWQOIdxR5MN/m9kCYKqZbRdCmJNU2OqV7HtR\n7e9dmpSaO4QwHb9N4ieavQl8DHTBx2HkMkv8m43vaYkSP6uK/ryabmbb4+P4Mn4QKjAGH3d4QAqv\nTfr7M3YPxXX4/f+y/G/df5hZM+BlvNrqWs7rFgAbmFnDYr0UZW5MVkWSamdlhfI3V6sqVdnOBfgX\n+Bas//41AWaV+IqqU9F2LsDz/T8zqwlsTHJfg2/hbd8B74WKrVo2CcwAqbRzPSGE1WY2C3/vcklp\n7+dPIYRfI+SpTm+T2i/oamVmNwN/Bf4UQihxX6wi0vL9GbWgCL7J13cVOTfRM/Ey8A7epVieGfjg\nt/bA44lr7AhsQxkbk1WFZNqZDlbO5mpVpSrbmSiSFuDv5wcAZtYQH4twS1V8zjKyVKidib9OG5tZ\n6yLjKNrjxcFbSXzK1vhfCtX6fpYmVNMmgbGl2M71mFkNYDcgo6dxp+BNoPi0wj+Twe9nGu1Jhnwv\nliZRTBwLHBxCmFuBl6Tn+zP2CNQKjlLdEt8D5EV86ucW6x5FzmmGdy3uVeTYGPwvukPwwVX/Al6P\n3Z5y2voHfBrWlfj0s1aJR/0i58zG90MBnwZ1Lf6LtXnii+LdxP+L2rHbk652Jj7ujf8i/xuwOz61\n6TNgg9jtKaOdzybej73xv2o+Ae4p7esW+CPQH2iTeD+PAf4LvBy7LcXadQo+w6YTPptlXOK92Tzx\n/ERgaJHz2wG/4uOedgIGAb8ALWO3Jc3tHID/IN4OLwQn4dObd47dlnLaWT/x/bcnPiPg0sTHf0g8\nPwy4u8j52wLL8NkeOwEXJt7fw2O3Jc3t7J74Htwe2BUfj7AKOCR2W8po4xjgB3z66BZFHnWLnHN3\nVXx/Rm98Bf8HnYV3PRZ9rAXWFDmneeL4QUWO1cHn4y4GlgIPA01it6ecto4voa3F27UG6JT477rA\n83iX1S94V/ut637gZeoj2XYWOTYIH6y4HB+FvEPstpTTzsbAvXjR9AM+z7tekefX+7rFN9f7J7Ao\n0d9fG4YAAADOSURBVMZPEj/kGsRuSwltuxD4Av+F+ybrF/MvA3cVO/9EvEhcgfcydYjdhnS3E9+N\neU7i3PnAU8AesdtQgTYevO5narHHXYnnx1OsqE28ZkairZ8BZ8ZuR7rbiQ+q/QwvChfhM34OipE9\niTaW1L71fpZW1fenNgcTERGRSsvmdShEREQkQ6igEBERkUpTQSEiIiKVpoJCREREKk0FhYiIiFSa\nCgoRERGpNBUUIiIiUmkqKERERKTSVFCIiIhIpamgEBERkUpTQSEiIiKV9n/a5h9rcwRAXgAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f6c46176d90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.mlab as mlab\n",
    "\n",
    "mu = 0.0   # mean of the distribution\n",
    "variance = 1  # standard deviation\n",
    "\n",
    "sigma = np.sqrt(variance)\n",
    "x = np.linspace(mu - 2*sigma, mu + 2*sigma, 100)\n",
    "plt.plot(x,mlab.normpdf(x, mu, sigma))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the absence of any prior knowledge about the distribution, the normal distribution is a good choice, owing to the **central limit theorem** (which states that the sum of many indepedent random variables is approximately normally distributed) and the fact that the normal distribution encodes the most uncertainty over the real numbers among all other probability distributions with the same variance.\n",
    "\n",
    "Generalizing to $\\mathbb{R}^n$, the mean $\\boldsymbol{\\mu}$ becomes a vector and the variance is replaced by the covariance matrix $\\boldsymbol{\\Sigma}$, which is often fixed to be a diagonal matrix. \n",
    "![normal multivariate](https://i.stack.imgur.com/EOyQI.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9.4 Exponential and Laplace distributions\n",
    "\n",
    "To place a sharp peak at x = 0, we can use the **exponential distribution**:\n",
    "$$ p(x; \\lambda) = \\lambda \\hspace{.05cm} exp(-\\lambda x), \\forall x \\geq 0$$\n",
    "\n",
    "Thus, the probability for $x < 0$ is 0. A similar function that allows to place a peak at x = $\\mu$ is the **Laplace distribution**: <br> <br>\n",
    "$$p(x; \\mu, \\gamma) = \\frac{1}{2 \\gamma} \\hspace{.05cm} exp(-\\frac{|x-\\mu|}{\\gamma})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9.5 The Dirac Distribution and Empirical Distribution\n",
    "\n",
    "The Dirac delta function is defined such that it is zero everywhere except 0, where its value is 1.\n",
    "$$ \n",
    "\\\\ \n",
    "\\delta(x) = \n",
    "     \\begin{cases}\n",
    "       1 &\\quad x = 0\\\\\n",
    "       0 &\\quad otherwise\n",
    "     \\end{cases}\n",
    "\\\\\n",
    "$$\n",
    "\n",
    "It can be visualized as a gaussian distribution with mean and variance as 0. \n",
    "![dirac delta](http://www.chebfun.org/examples/complex/img/Hyperfuns_01.png)\n",
    "\n",
    "Suppose we want to define a probability distribution with all the masss centered on one value (say $\\mu$), we can define $p(x)$ as:\n",
    "$$ p(x) = \\delta (x-\\mu)$$\n",
    "\n",
    "A common use of the Dirac delta distribution is as a component of an **empirical distribution**:\n",
    "$$ \\hat{p}(\\boldsymbol{x}) = \\frac{1}{m}\\sum_{i=1}^{m} \\delta(\\boldsymbol{x} - \\boldsymbol{x}^{(i)}) $$\n",
    "\n",
    "This assigns a probability mass of 1/m to each of the m points $\\boldsymbol{x}^{(i)}, i=1, 2, ... m$. This is because putting $\\boldsymbol{x} = \\boldsymbol{x}^{(i)}$ in the above equation will result in only one non-zero term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  9.6 Mixtures of distributions\n",
    "\n",
    "We can define probability distributions by combining other simpler probability distributions. One such way is to construct a **mixture distribution**. On each trial, the choice of which component distribution should generate the sample is determined by sampling a component identity from a multinoulli distribution:\n",
    "\n",
    "$$ P(x) = \\sum_{i} P(c = i) P (x \\hspace{.1cm} | \\hspace{.1cm} c = i) $$\n",
    "\n",
    "where $P (c)$ is the multinoulli distribution over component identities. An important concept introduced here is that of a **latent variable**, which is a random variable that we cannot observe directly. In the given case, $c$ is a latent variable. The distribution $P (x \\hspace{.1cm} | \\hspace{.1cm} c = i) $, relating the latent variable to the visible variable, determines the shape of the distribution $P(x)$, even thoughit is possible to describe $P(x)$ without reference to the latent variable.\n",
    "\n",
    "A very powerful type of mixture model is the **Gaussian Mixture model (GMM)**, where each of the component distributions is a gaussian, with a **prior probability** associated with each gaussian. The word “prior” indicates that it expresses the model’s beliefs about $c$ *before* it has observed $x$. <br>\n",
    "$P(c \\hspace{.1cm} | \\hspace{.1cm} x)$ is a **posterior probability**, because it is computed *after* observation of x. A Gaussian mixture model is a **universal approximator** of densities, in the sense that any smooth density can be approximated with any speciﬁc non-zero amount of error by a Gaussian mixture model with enough components."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Useful properties of Common Functions\n",
    "\n",
    "Certain functions arise often while working with the probability distributions used in deep learning models, one such being the **logistic sigmoid**, given by:\n",
    "$$ \\sigma(x) = \\frac{1}{1 + exp(-x)} $$\n",
    "\n",
    "It produces a value in the range [0, 1] and thus, often used in the last layer of a deep learning model, (e.g. in classification), to squash the final output values so that they represent a probability of a class, given the input. The function saturates when its argument is either too positive or too negative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another commonly used function is the **softplus function**:\n",
    "\n",
    "$$ \\varsigma(x) = log(1+exp(x)) $$\n",
    "\n",
    "It is useful for producing $\\sigma$ of a normal distribution as its range is $(0, \\infty)$. It is named so because it is smoothened (or \"softened) version of: $ x^+ = max(0, x) $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAAFkCAYAAABfHiNRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xd8FNX+//HXSSghAUKTJghBpCgKJIJgQRFJ6OClGTrY\n8CIClqui38stFkAFVBS9+vMCAkEQEJBQpXgBBQmCKF0pIs1QYgolZM/vjw1IIEAgu5ndzfv5eMwj\n2bMzcz7JEva9M2fOGGstIiIikr8FOV2AiIiIOE+BQERERBQIRERERIFAREREUCAQERERFAhEREQE\nBQIRERFBgUBERERQIBAREREUCERERIRrCATGmHuMMXOMMb8ZY1zGmHbnPVfAGDPCGPODMSYlc50J\nxpgKni1bREREPOlajhCEARuAAcCFN0IIBeoB/wTqAw8CNYHZuahRREREvMzk5uZGxhgX0MFaO+cy\n69wOrAGqWGv3XXNnIiIi4jV5MYagBO4jCcfzoC8RERG5BgW8uXNjTGFgODDFWptyiXVKAzHAbuCk\nN+sREREJMCFAVWChtfZIbnbktUBgjCkATMd9dOCvl1k1BpjsrTpERETyge7AlNzswCuB4LwwUBm4\n/1JHBzLtBpg0aRK1a9f2RjmSx4YMGcLo0aOdLkM8SK9pYNHrGTi2bNlCjx49IPO9NDc8HgjOCwPV\ngKbW2mNX2OQkQO3atYmMjPR0OeKA8PBwvZYBRq9pYNHrGZByfcr9qgOBMSYMqA6YzKZqxpi6wFFg\nPzAD96WHbYCCxphymesdtdam57ZgERER8bxrOUJwO7AM99gAC7yV2T4B9/wDbTPbN2S2m8zHTYGv\nc1OsiIiIeMdVBwJr7Qouf7mipkMWERHxM3rzFo+LjY11ugTxML2mgUWvp2RHgUA8Tv/ZBB69poFF\nr6dkx6sTE3nS3r17SUxMdLoMyUNlypThhhtucLoMEZF8wS8Cwd69e6lduzZpaWlOlyJ5KDQ0lC1b\ntigUiIjkAb8IBImJiaSlpWnyonzk7GQbiYmJCgQiInnALwLBWZq8SERExDs0qFBEREQUCERERESB\nQERERFAgEBERERQI/EKfPn2IiIhwuozL2rNnD0FBQUycOPGK6/rDzyMikt8oEPgBYwxBQb7/Uhlj\nrrxS5no5XVdERPKGX112mF99/PHHuFwup8u4rCpVqnDixAkKFizodCkiInINFAj8QHBwMMHBwU6X\ncUWFChVyugQREblGvn8cOh9ISUlh8ODBREREEBISQrly5YiOjmbDhg1A9ufcjx49Ss+ePQkPD6dk\nyZL07duXH3744aLz+H369KFYsWL8+uuvtGnThmLFilG5cmXef/99ADZt2kSzZs0oWrQoVatWJS4u\n7qL6du3aRefOnSldujRhYWE0btyY+Pj4LOtcagzBF198QZ06dShSpAi33XYbX3zxhUd+ZyIi4lkK\nBD7g8ccf58MPP6Rz586MGzeO5557jrCwMLZs2QJcfM7dWkubNm347LPP6Nu3L6+99hoHDhygd+/e\nF52bN8bgcrlo2bIlVapU4Y033qBq1aoMHDiQCRMm0LJlSxo0aMDIkSMpXrw4vXv3Zs+ePee2P3z4\nMI0bN2bx4sU8+eSTvPbaa5w6dYq2bdsye/bsy/5cixYtolOnThQoUIDhw4fToUMH+vbty7p16zz4\n2xMREY+w1jq6AJGATUhIsJeSkJBgr7SOPytRooQdOHDgJZ/v06ePjYiIOPd4xowZ1hhj33333Szr\nNWvWzAYFBdkJEyZk2TYoKMiOGDHiXNvx48dtaGioDQ4Otp9//vm59m3btlljjP3nP/95rm3w4ME2\nKCjIrl69+lxbSkqKrVatmq1Wrdq5tt27d1tjTJa+69WrZ6+//nqbnJx8rm3JkiXWGJPl58lOoL/m\nIiKecPb/SiDS5vL9OODGEKSlwdat3u+nVi0IDfXMvkqUKMHatWs5cOAAFSpUuOL6CxcupFChQjzy\nyCNZ2gcMGMDSpUuz3ebhhx8+9314eDg1a9bk559/pmPHjufaa9SoQYkSJfjll1/Otc2fP5+GDRvS\nuHHjc21hYWE89thjDB06lM2bN3PzzTdf1N/BgwfZuHEjQ4cOpWjRoufamzVrxs0336w7V4qI+JiA\nCwRbt0JUlPf7SUgAT91naeTIkfTp04fKlSsTFRVFq1at6NWr1yWv1d+zZw8VKlQgJCQkS3v16tWz\nXT8kJITSpUtnaQsPD6dSpUoXrRseHs6xY8ey9NWoUaOL1jt718k9e/ZkGwjOnnbIrqaaNWvy/fff\nZ1uriIg4I+ACQa1a7jfrvOjHUzp37kyTJk2YNWsWixYt4s0332TEiBHMmjWLmJiYXO//UlcoXKrd\nuk/l5MrZfWQ334An9i8iIp4VcIEgNNRzn9zzUrly5ejfvz/9+/cnMTGR+vXr8+qrr2YbCKpUqcLy\n5cs5efJklqMEO3bs8HhdVapUYdu2bRe1nx3wWKVKlWy3q1q1KgDbt2+/6Lns2kRExFm6ysBhLpeL\nP/74I0tbmTJlqFixIqdOncp2m5iYGE6fPs1HH310rs1ay3vvvefxGQBbtWrF2rVrWbNmzbm21NRU\n/vOf/xAREZHt6QKA8uXLU69ePSZMmEBycvK59sWLF7N582aP1igiIrkXcEcI/E1ycjKVKlWiU6dO\n1K1bl6JFi7J48WLWrVvHqFGjst2mQ4cONGzYkGeeeYYdO3ZQq1Yt5syZw/Hjx4GcTyGcEy+88AJx\ncXG0aNGCp556ilKlSjF+/Hj27NnDzJkzL7vt66+/Tps2bbjrrrvo168fR44cYezYsdSpU4eUlBSP\n1SgiIrmnIwQOCw0NZcCAAWzcuJF//OMfPP300+zYsYNx48YxaNCgc+ud/yYfFBREfHw8Xbt2ZeLE\nibz88stUqlSJsWPHYq29aLDhpQJCdu0XznlQtmxZvvnmG6Kjoxk7dixDhw4lJCSEL7/8knbt2l12\nfzExMUyfPh2Xy8XQoUP54osvGD9+PFFRUbqXgYiIjzFOD/AyxkQCCQkJCURe4uT/+vXriYqK4nLr\niHtWwI4dO7Jy5coslwn6I73mIiJXdvb/SiDKWrs+N/vSEQI/deH4ApfLxbvvvkvx4sX1BioiIldN\nYwj81MCBA0lLS6Nx48acOnWKGTNm8O233/L6669TuHBhp8sTERE/o0Dgp5o2bcqoUaOYN28eJ0+e\npHr16owdO5YnnnjC6dJERMQPKRD4qdjYWGJjY50uQ0REAoTGEIiIiIgCgYiIiCgQiIiICAoEIiIi\nggKBiIiIoEAgIiIiKBCIiIgICgQBZ8GCBdSvX58iRYoQHBx80a2Vr1WfPn2IiIjwyL5ERMT3aGKi\nAHL06FG6du1KnTp1eP/99ylcuDBhYWHExcVx+PDhLHdPvFoX3gVRREQCiwJBAPnuu+9ISUnhlVde\noWnTpufap0yZwk8//ZSrQCAiIoFNpwwCyKFDhwAIDw93uBIREfE3CgQ+ICUlhcGDBxMREUFISAjl\nypUjOjqaDRs2nFtn+vTp3H777YSGhnLdddfRs2dP9u/ff+75pk2b0qdPHwBuv/12goKC6NevH02b\nNmXevHns2bOHoKAggoKCqFatGgDLly8nKCiIadOmMXToUCpUqEDRokVp3749+/btu2zNK1asICgo\niK+//jpL+9l+Jk6ceK7t0KFD9O3bl8qVKxMSEkLFihXp0KEDe/fuze2vTkREPOSqTxkYY+4BngOi\ngApAB2vtnAvW+RfwCFACWAU8Ya3dmftyA9Pjjz/OzJkzGThwILVr1+bIkSOsWrWKLVu2UK9ePcaP\nH0+/fv244447GD58OIcOHWLMmDGsXr2a77//nuLFi/Pyyy9Ts2ZNPvroI1555RWqVq3KjTfeSEpK\nCklJSfz222+MGTMGay1FixYFODcm4NVXXyUoKIgXXniBw4cPM3r0aJo3b86GDRsueyvlnI4p+Mtf\n/sKWLVt46qmnqFKlCocPH2bx4sXs3buXG264Ife/QBERybVrGUMQBmwAPgFmXPikMeZ54EmgN7AL\neAVYaIypba09nYtaA1Z8fDyPPvooI0eOPNf27LPPAnDmzBleeOEFbrvtNlasWEGhQoUAuOuuu2jT\npg2jR49m2LBhNGvWjH379vHRRx/RokULIiMjz+3r+uuv5/jx45e8O+KxY8fYunUroaGhANSvX58u\nXbrw0Ucf8eSTT+bqZ0tKSuKbb77hzTff5Omnnz7X/vzzz+dqvyIi4llXHQistQuABQAm+4+Ig4B/\nW2vnZq7TCzgEdACmXXupOZOWnsbWxK3e7oZaZWoRWjDUI/sqUaIEa9eu5cCBA1SoUCHLc+vWrePw\n4cP861//OhcGAFq1akWtWrWYN28ew4YNy1X/vXv3PhcGADp16kSFChWIj4/PdSAoUqQIhQoVYvny\n5fTr148SJUrkan8iIuIdHr3KwBgTAZQHvjrbZq39wxizBmhMHgSCrYlbifpPlLe7IeGxBCIrRF55\nxRwYOXIkffr0oXLlykRFRdGqVSt69epFREQEe/bswRhDjRo1LtquVq1arFq1Ktf9V69ePdu2PXv2\n5HrfhQoVYsSIETz77LOUK1eORo0a0aZNG3r16kW5cuVyvX8REfEMT192WB6wuI8InO9Q5nNeV6tM\nLRIeS8iTfjylc+fONGnShFmzZrFo0SLefPNNRowYwaxZs7DWeqyfq3Glfi81fiAjI+OitkGDBtGu\nXTu++OILFi5cyN///ndef/11li1bRt26dT1Sr4hIfrT+wHqP7Suv5iEwuIPCJQ0ZMuSiy+ViY2Mv\ned77UkILhnrsk3teKleuHP3796d///4kJiZSv359Xn31VUaOHIm1lm3btnHfffdl2Wbbtm1UqVLl\nivu+0uC/HTt2XNT2888/X/bNumTJklhrOX78eJb23bt3Z7t+REQEQ4YMYciQIef2/dZbb2W5GkFE\nRC4tLi6OuLi4c4//OPUHK7ev9Nj+PX3Z4UHcb/4XHgsuy8VHDbIYPXo0c+bMybJcbRjwRy6X66Lp\nhcuUKUPFihU5deoUDRo0oGzZsnzwwQekp6efW2f+/Pls2bKFNm3aXLGPsLAwkpKSLvn8xIkTSUlJ\nOfd4+vTpHDhwgFatWl1ymypVqhAcHHzRZYfvv/9+lgBy4sQJTp06lWWdiIgIihUrdlG7iIhcWmxs\n7Ln3x7GfjmVnzE5u7Hajx/bv0SME1tpdxpiDQDPgBwBjTHHgDuA9T/YVKJKTk6lUqRKdOnWibt26\nFC1alMWLF7Nu3TpGjRpFcHAwI0aMoF+/fjRp0oTY2FgOHjzIO++8Q7Vq1Rg8eHCW/WV3qD8qKopp\n06bxzDPP0KBBA4oWLZolSJQqVYq7776bvn37cvDgQd5++21q1KjBI488csm6ixcvTufOnXnnnXcA\nuPHGG5k7dy6JiYlZ1tu+fTvNmjWjS5cu3HzzzRQoUICZM2dy+PDhfBH4REQ87UjaEWImxVAouBDv\ntnyXmNdiPLLfa5mHIAyojvtIAEA1Y0xd4Ki19ldgDPCyMWYnsBv4N7APmO2RigNMaGgoAwYMYNGi\nRcyaNQuXy0X16tUZN24cjz32GOC+CiAsLIzhw4fzwgsvEBYWRseOHRk+fDjFixfPsr/sTg/89a9/\nZePGjYwfP54xY8ZQpUqVc4HAGMPQoUP54YcfGD58OMnJyTRv3pz33nuPkJCQy+773Xff5cyZM3z4\n4YcULlyYrl278tZbb1GnTp1z61SuXJlu3brx1VdfMWnSJAoUKECtWrWYPn06HTp08MjvUEQkv0hL\nT6NtXFsS0xJZ1W8VKXtSrrxRTllrr2oB7gVcQMYFyyfnrfMPYD+QBiwEql9mf5GATUhIsJeSkJBg\nr7SOXL3ly5dbY4ydMWOG06VcRK+5iEhW6Rnptu2Utjb01VC7Zt8aa+2f/1cCkfYq388vXK5lHoIV\nXGHsgbX2H5mhQERERHLJWkv/L/sTvyOeubFzaXh9Q4/3obsdioiI+Lhhy4fx/77/f0zoMIGWN7X0\nSh+6uVE+l9P7EYiIiDPGfTeOf3/9b0Y8MIJedXt5rR8dIcjH7r333mwnEhIREd8wc8tMBsQPYNAd\ng3juzue82peOEIiIiPigFbtX0G1GNzrf0plRMaO8fkRXgUBERMTHbDq0ifZT23PXDXcxscNEgoz3\n364VCERERHzI3qS9tJjcgqolqjKzy0wKFyicJ/0qEIiIiPiI82chnN99PuEh4VfeyEP8alDhli1b\nnC5B8oheaxHJby6chbBCsQp52r9fBIIyZcoQGhpKjx49nC5F8lBoaChlypRxugwREa874zrDQ58/\nxMZDG1nWexk1StfI8xr8IhDccMMNbNmy5aIb50hgK1OmDDfccIPTZYiIeJW1lie+fIL5O+d7bRbC\nnPCLQADuUKA3BxERCTTDlg/j4+8/ZkKHCbSo3sKxOjSoUERExCF5NQthTigQiIiIOCAvZyHMCQUC\nERGRPHZ2FsIut3TJk1kIc0KBQEREJA+dPwvhhA4T8mQWwpzwjSpERETygfNnIZzVdVaezUKYEwoE\nIiIieeDCWQiLFy7udElZ+M1lhyIiIv7K6VkIc0KBQERExIt8YRbCnFAgEBER8RJfmYUwJxQIRERE\nvMRXZiHMCQ0qFBER8QJfmoUwJxQIREREPMzXZiHMCQUCERERD/p6z9c+NwthTigQiIiIeMiPh3+k\nXVw7n5uFMCf8p1IREREftjdpLy0m+eYshDmhQCAiIpJL589CuKDHAp+bhTAndNmhiIhILlw4C2H5\nouWdLumaKBCIiIhcozOuM3T9vCs/HPrBp2chzAkFAhERkWtgreXxuY+zYOcC5sbOpcH1DZwuKVcU\nCERERK7B35f9nU82fMLEDhN9fhbCnNCgQhERkav0/nfv88r/XmHEAyPoWben0+V4hAKBiIjIVfh8\n8+c8Gf8kg+8Y7DezEOaEAoGIiEgOrdi9gu4zu9O1TlfeinnLb2YhzAkFAhERkRzYdGgT7ae2554b\n7mF8+/F+NQthTgTWTyMiIuIFe47vocXkFlQrWY2ZXWf63SyEOaFAICIichlH0o7QYnILCgcXJr57\nvF/OQpgTuuxQRETkEtLS02gT14bEtERW91vtt7MQ5oQCgYiISDbOzkK46dAmlvVexk2lb3K6JK9S\nIBAREblAoM1CmBMeH0NgjAkyxvzbGPOLMSbNGLPTGPOyp/sRERHxlrOzEH7S7pOAmIUwJ7xxhOAF\n4HGgF7AZuB0Yb4w5bq0d64X+REREPCYQZyHMCW8EgsbAbGvtgszHe40x3YCGXuhLRETEYwJ1FsKc\n8MZlh6uBZsaYmwCMMXWBu4B4L/QlIiLiEYE8C2FOeOMIwXCgOLDVGJOBO3S8ZK2d6oW+REREci3Q\nZyHMCW8Egq5AN+Ah3GMI6gFvG2P2W2s/9UJ/IiIi1yw/zEKYE94IBCOB16y10zMf/2SMqQq8CFwy\nEAwZMoTw8PAsbbGxscTGxnqhRBEREfcshDGTYggpEML87vN9ehbCuLg44uLisrQlJSV5bP/eCASh\ngL2gzcUVxiuMHj2ayMhIL5QjIiJysdTTqbSJa8PRE0dZ/fBqyhUt53RJl5Xdh+T169cTFRXlkf17\nIxDMBV4yxvwK/AREAkOAj73Ql4iIyFU7fxbC5X2WU71UdadLcpw3AsGTwL+B94CywH5gXGabiIiI\no87OQrjw54V8Gfslt1e83emSfILHA4G1NhV4OnMRERHxKS8vfZlPNnzCxA4Tiake43Q5PiP/XVch\nIiL51ntr3+O1la8x8oGR+WoWwpxQIBARkXxh+k/TGTh/IEMaDeHZO591uhyfo0AgIiIBb/nu5fSY\n1YOH6jzEm9Fv5rtZCHNCgUBERALaxoMbaT+1PU2qNGF8h/w5C2FO6LciIiIBa/fx3bSc3JLqpaoz\ns8tMCgUXcrokn6VAICIiASkxLZEWk1pQpGAR4rvFU6xwMadL8mnemIdARETEUamnU2kzxX9mIfQF\nCgQiIhJQ0jPS6fJ5F348/CMr+qzQLIQ5pEAgIiIBw1rL418+zqKfFzGv2zyiKnpmnv/8QIFAREQC\nxstLX+a/G/7LpAcnEX1jtNPl+BUNKhQRkYAwdu1YXlv5Gm82f5Put3V3uhy/o0AgIiJ+b/pP03lq\n/lMMaTSEZ+58xuly/JICgYiI+LVlu5bRY1YPYm+N5c3oN50ux28pEIiIiN/aeHAjHT7rwL1V7uW/\n7f+rWQhzQb85ERHxS7uP76bF5BbcVOomZnSZoVkIc0mBQERE/E5iWiIxk2IIKxjGvG7zNAuhB+iy\nQxER8Supp1NpPaU1x08eZ3U/zULoKQoEIiLiN9Iz0uk8vTObf9/M8t7LubHUjU6XFDAUCERExC9Y\na3l07qMs+WWJZiH0AgUCERHxC0O/GsqEjROY/JfJNL+xudPlBBwNKhQREZ/3zpp3GL5qOG9Fv0W3\nW7s5XU5AUiAQERGfNu2naQxeMJhnGj/D042fdrqcgKVAICIiPmvprqX0nNWTbrd2Y2TzkU6XE9AU\nCERExCd9f+B7OkztwH1V7+OT9p9oFkIv029XRER8zi/HfqHl5JbULFNTsxDmEQUCERHxKYdTDxMz\nKYZihYsxr9s8ihYq6nRJ+YIuOxQREZ+RcjqFNlPakHwqmdUPr6ZsWFmnS8o3FAhERMQnpGek02la\nJ7YkbmFFnxVUK1nN6ZLyFQUCERFxnMu6eHjOwyzdtZT47vFEVoh0uqR8R4FAREQc9+KSF/n0h0+J\n6xjHA9UecLqcfEmDCkVExFFjvh3DyNUjGRMzhofqPOR0OfmWAoGIiDgmblMcQxYO4W93/o1BjQY5\nXU6+pkAgIiKOWPLLEnp/0ZtedXsx/IHhTpeT7ykQiIhInlt/YD0PfvYgzao14+O2H2OMcbqkfE+B\nQERE8tTPR3+m5eSW1C5Tm+mdp1MwuKDTJQkKBCIikofOzkIYXjhcsxD6GF12KCIieSL5VDKtJrci\nNT2V1f1Wc13YdU6XJOdRIBAREa87nXGajtM6sv3Idr7u+zURJSOcLkkuoEAgIiJe5bIu+s7uy4o9\nK1jQfQH1ytdzuiTJhgKBiIh41XOLniNuUxyfdfqMphFNnS5HLkGBQEREvObN1W8y6ttRvNvyXTrf\n0tnpcuQyvHKVgTGmojHmU2NMojEmzRiz0RijO1WIiOQjk36YxHOLn2Po3UN5suGTTpcjV+DxIwTG\nmBLAKuArIAZIBG4Cjnm6LxER8U0Ldi6g7+y+9KvXj1fuf8XpciQHvHHK4AVgr7X2kfPa9nihHxER\n8UHf/fYdnaZ1okX1FnzY9kPNQugnvHHKoC2wzhgzzRhzyBiz3hjzyBW3EhERv7f9yHZaTWnFbeVu\n47NOn1EgSEPV/IU3AkE14AlgGxANfAC8Y4zp4YW+RETERxxIPkDMpBiuC72OubFzCS0Y6nRJchW8\nEd2CgLXW2v/LfLzRGHML7pAw6VIbDRkyhPDw8CxtsbGxxMbGeqFEERHxpD9O/UHLyS1Jz0hnRZ8V\nlA4t7XRJAScuLo64uLgsbUlJSR7bv7HWemxnAMaY3cAia+1j57X1B16y1lbOZv1IICEhIYHISF2I\nICLib06dOUXLyS1Zf2A9K/utpE7ZOk6XlG+sX7+eqKgogChr7frc7MsbRwhWATUvaKuJBhaKiASc\nDFcGPWf1ZPWvq1ncc7HCgB/zRiAYDawyxrwITAPuAB4BHvVCXyIi4hBrLYMXDGbGlhnM6DKDe6rc\n43RJkgseH1RorV0HPAjEApuAl4BB1tqpnu5LRESc8/rK1xn73VjGtR5Hh1odnC5Hcskr14NYa+OB\neG/sW0REnPfJ95/w0tKX+Od9/+SxqMeuvIH4PK9MXSwiIoFr7ra5PDb3MfpH9ef/mvzflTcQv6BA\nICIiObb619V0+bwL7Wu1Z2yrsZqFMIAoEIiISI5s/n0zbaa0oeH1DZn8l8kEBwU7XZJ4kAKBiIhc\n0b4/9hEzKYZKxSsx+6HZhBQIcbok8TAFAhERuaxjJ47RYlILgkwQC3osoERICadLEi/QXSdEROSS\nTqSfoG1cWw6mHGRlv5VULFbR6ZLESxQIREQkW2dcZ3hoxkOsP7Cepb2XUqtMLadLEi9SIBARkYtY\na3niyyeYt30ec2Ln0KhSI6dLEi9TIBARkYsMWz6Mj7//mPHtx9PqplZOlyN5QIMKRUQki3HfjePf\nX/+bEQ+MoHe93k6XI3lEgUBERM6ZsXkGA+IHMPiOwTx353NOlyN5SIFAREQAWL57Od1mduOhOg/x\nVsxbmoUwn1EgEBERNh7cSPup7bm3yr2M7zCeIKO3h/xGr7iISD6369guWkxuwU2lbmJGlxkUCi7k\ndEniAAUCEZF87HDqYaInRVO0UFHiu8dTrHAxp0sSh+iyQxGRfCrldAqtp7Qm+VQyqx9eTdmwsk6X\nJA5SIBARyYdOZ5ym47SObEvcxoo+K6hWsprTJYnDFAhERPIZl3XRd3Zflu9ezoLuC6hfob7TJYkP\nUCAQEclHrLU8s/AZ4jbFMa3zNJpGNHW6JPERCgQiIvnIG6vfYMyaMbzX6j063dzJ6XLEh+gqAxGR\nfGLChgk8v+R5/q/J//HXBn91uhzxMQoEIiL5wLzt83h4zsM8Gvko/7zvn06XIz5IgUBEJMB98+s3\ndJ7embY12/J+6/c1JbFkS4FARCSAbf59M62ntOb2ircz5S9TKBCkoWOSPQUCEZEA9WvSr8RMiqFS\n8UrMiZ1DkYJFnC5JfJgCgYhIADp64igtJrcg2ASzoMcCSoSUcLok8XE6diQiEmDS0tNoG9eWw6mH\nWdVvFRWLVXS6JPEDCgQiIgEkPSOdLtO7sPHgRpb2XkqN0jWcLkn8hAKBiEiAsNby6NxHWfjzQuZ1\nm0fD6xs6XZL4EQUCEZEA8cKSF5iwcQKT/zKZ6BujnS5H/IwGFYqIBIBR34xi5OqRjIkZQ7dbuzld\njvghBQIRET836YdJPLPoGV68+0UGNRrkdDnipxQIRET82Pwd8+k7uy/96vXj1ftfdboc8WMKBCIi\nfurbfd/wVO1YAAAY1klEQVTSaXonWt3Uig/bfqgpiSVXFAhERPzQlt+30HpKayIrRDK141RNSSy5\npkAgIuJnfk36lehJ0VQsVpE5D2lKYvEMBQIRET9yJO0IMZNiCDbBLOyxkJJFSjpdkgQIHWMSEfET\nqadTaT2lNb+n/c7Kvis1JbF4lAKBiIgfSM9Ip9P0Tvz0+08s672MmmVqOl2SBBgFAhERH+eyLvrO\n7stXv3xFfPd4bq94u9MlSQBSIBAR8WHWWp5e+DRTNk1haqepPFDtAadLkgDl9UGFxpgXjTEuY8wo\nb/clIhJohq8czttr3ua9Vu/R5ZYuTpcjAcyrgcAY0wB4FNjozX5ERALRx+s/ZujSoQy7dxhPNHjC\n6XIkwHktEBhjigKTgEeA497qR0QkEM3aMovHv3ycAQ0GMOzeYU6XI/mAN48QvAfMtdYu9WIfIiIB\nZ/nu5cTOiKXzzZ15p+U7mpJY8oRXBhUaYx4C6gEaCisichW+P/A97eLa0aRKEyY+OJEgo/njJG94\nPBAYYyoBY4Dm1tr0nG43ZMgQwsPDs7TFxsYSGxvr4QpFRHzTzqM7aTG5BbWvq83MrjMpFFzI6ZLE\nh8TFxREXF5elLSkpyWP7N9Zaj+0MwBjTHpgJZABnj3MFAzazrbA9r1NjTCSQkJCQQGRkpEdrERHx\nF/uT93PXJ3cRUiCE//X9H2VCyzhdkviB9evXExUVBRBlrV2fm31545TBEuDWC9rGA1uA4dbTCURE\nxM8dO3GMmEkxnHGdYWGPhQoD4giPBwJrbSqw+fw2Y0wqcMRau8XT/YmI+LPU06m0iWvDgeQD/K/v\n/7gh/AanS5J8Kq9mKtRRARGRC6RnpNN5emc2HtzI0t5LqX1dbadLknwsTwKBtfb+vOhHRMRfuKyL\nPrP7sOSXJcR3j6fh9Q2dLknyOd3LQEQkj1lrGTR/EHGb4nR/AvEZCgQiInnsXyv+xdjvxvJB6w90\nfwLxGZrxQkQkD41dO5Z/rPgHr97/Ko/f/rjT5Yico0AgIpJHpmyawsD5A3m60dO8ePeLTpcjkoUC\ngYhIHojfEU/vL3rTu25v3oh+Q/cnEJ+jQCAi4mUr966k47SOtLqpFR+3+1j3JxCfpH+VIiJetOHg\nBtpMaUOjSo34rNNnFAjSWG7xTQoEIiJesuPIDmImxVC9VHVmPzSbkAIhTpckckkKBCIiXrDvj300\n/7Q5pYqUYn73+RQvXNzpkkQuS4FARMTDEtMSif40GotlUY9FXBd2ndMliVyRTmaJiHjQH6f+oOXk\nliSmJbKy30oqh1d2uiSRHFEgEBHxkBPpJ2gX144dR3awrPcyapSu4XRJIjmmQCAi4gHpGel0+bwL\na39by+Kei6lfob7TJYlcFQUCEZFcOnvnwoU7FzIndg533XCX0yWJXDUFAhGRXLDW8mT8k0z9cSpT\nO06lRfUWTpckck0UCEREcuGlpS8xbt04Pm77MZ1v6ex0OSLXTJcdiohco+Erh/P6ytcZFT2KhyMf\ndrockVxRIBARuQbvf/c+L371IsPuHcaQxkOcLkck1xQIRESu0qcbP2VA/ACGNBrCsHuHOV2OiEco\nEIiIXIVZW2bRd3ZfHqn/CG9Fv6XbGEvAUCAQEcmhhTsX8tCMh+h4c0c+aPOBwoAEFAUCEZEc+HrP\n1zz42YNE3xjNpw9+SnBQsNMliXiUAoGIyBWs/W0tbaa0oXHlxkzvPJ1CwYWcLknE4xQIREQu44dD\nP9BiUgtuLXcrsx+aTUiBEKdLEvEKBQIRkUvYlriN5p82J6JkBPO6zaNooaJOlyTiNQoEIiLZ+OXY\nLzSb2IwyoWVY2GMhJUJKOF2SiFcpEIiIXGBv0l7un3A/RQoWYUnPJZQJLeN0SSJep0AgInKe/cn7\naTaxGcYYlvZaSoViFZwuSSRP6OZGIiKZDqce5oGJD3DyzEm+7vM1lcMrO12SSJ5RIBARAY6eOErz\nT5tz7OQxVvRZQUTJCKdLEslTCgQiku8dP3mc6E+j2Z+8n+W9l1OjdA2nSxLJcwoEIpKvJZ1MImZS\nDLuO72Jpr6XcUvYWp0sScYQCgYjkW8mnkmk1pRXbj2znq15fUbd8XadLEnGMAoGI5Eupp1NpPaU1\nPx7+kSU9lxBZIdLpkkQcpUAgIvlOWnoabePa8v3B71nUYxENrm/gdEkijlMgEJF85WwYWPvbWuZ3\nn0/jyo2dLknEJ2hiIhHJN86GgTX71hDfPZ57qtzjdEkiPkOBQETyhbT0NNrFtTsXBppUaeJ0SSI+\nRacMRCTgpaWn0X5qe77Z9w3zu89XGBDJhgKBiAS0s0cGvtn3DfHddGRA5FI8fsrAGPOiMWatMeYP\nY8whY8wsY4ym/RKRPJdyOoXWU1rz7b5vie8Wz71V73W6JBGf5Y0xBPcA7wJ3AA8ABYFFxpgiXuhL\nRCRbyaeSaTm5Jev2r2NBjwUKAyJX4PFTBtbaVuc/Nsb0AQ4DUcBKT/cnInKhpJNJtJzckp9+/4nF\nPRfTqFIjp0sS8Xl5MYagBGCBo3nQl4jkc8dOHKPF5BZsP7KdJT2XaNIhkRzyaiAwxhhgDLDSWrvZ\nm32JiPye+jvRk6LZm7SXr3p9pemIRa6Ct48QvA/cDNzl5X5EJJ/bn7yf5p8250jaEZb3Xs6t5W51\nuiQRv+K1QGCMGQu0Au6x1h640vpDhgwhPDw8S1tsbCyxsbFeqlBEAsWe43toNrEZpzJO8XXfr6lR\nWhc2SeCJi4sjLi4uS1tSUpLH9m+stR7b2bmdusNAe+Bea+0vV1g3EkhISEggMlKH90Tk6uw4soNm\nE5tRMLggX/X6iqolqjpdkkieWb9+PVFRUQBR1tr1udmXx48QGGPeB2KBdkCqMaZc5lNJ1tqTnu5P\nRPKvTYc2ET0pmhIhJVjScwnXF7/e6ZJE/JY35iHoDxQHlgP7z1u6eKEvEcmnVv+6mibjm1C+aHlW\n9FmhMCCSS96Yh0A3TBIRr1qwcwF/+ewvNLi+AXMemkN4SPiVNxKRy9Kbt4j4lc9+/Ix2ce1oVq0Z\nC7ovUBgQ8RAFAhHxGx+s+4DYGbF0rdOVmV1mUqSgZkQX8RQFAhHxedZa/r7s7zwx7wkGNhzIhA4T\nKBhc0OmyRAKKbn8sIj7tjOsMj899nE82fMKIB0bw3J3P4Z4EVUQ8SYFARHxW6ulUunzehUU/L2Ji\nh4n0rNvT6ZJEApYCgYj4pN9Tf6dNXBt+OvwT87rNI/rGaKdLEgloCgQi4nO2Jm6l9ZTWpJxOYUWf\nFURVjHK6JJGAp0GFIuJTlu1aRuP/15iQAiGseWSNwoBIHlEgEBGfMX7DeKInRXN7xdtZ1W+V7ksg\nkocUCETEcS7r4uWlL9N3dl/61utLfLd4SoSUcLoskXxFYwhExFEpp1PoNasXs7bOYuQDI3n2zmd1\nWaGIAxQIRMQxvxz7hfZT27Pn+B5mPzSbdjXbOV2SSL6lQCAijli6aymdp3emZEhJvn3kW26+7man\nSxLJ1zSGQETylLWWd9a8Q/Sn0URViGLto2sVBkR8gI4QiEieSTmdwqNzH2Xqj1N5utHTjGg+ggJB\n+m9IxBfoL1FE8sTm3zfTcVpH9v2xj2mdptH5ls5OlyQi59EpAxHxurhNcTT8qCHBJph1j65TGBDx\nQQoEIuI1aelpPPHlE3Sb2Y0Haz/ImkfWULNMTafLEpFs6JSBiHjFxoMbiZ0Ry+7ju/mwzYc8Gvmo\n5hcQ8WEKBCLiUWevIvjbkr9Ru0xtEh5LoPZ1tZ0uSzzIWvdyNd9fqu3C5y9su9TjnLZlt05OXOt2\nud32ah075rl9KRCIiMccSD7Aw3MeZv7O+Qy+YzCvP/A6IQVCnC7L61wuSEv7c0lNdX89cQJOnnR/\nPfv9qVN/LqdPZ7+kp2ddzpyBjAz31/OXjIw/289+73Jd/PXC710u95vWhd9n9/XscvaxBC4FAhHJ\nNWstkzdN5qn5T1EouBDx3eJpeVNLp8vKsdOnITHRvRw9mnU5dgySkrIuycmQkvLn17S0nPdlDBQu\nnHUpVOjPpWDBP5cCBbI+LlLE3RYc/OfXyy3G/Pl9UNCfj4OC/lyMufjr2eXCx5dbzv5s2T0+/2fP\n7mtO27I745STs1DXul1O9+Wkn3+GZ57xzL4UCEQkVw4kH6D/vP7M2TaHbrd2450W71A6tLTTZQHu\nN+pff3Uvv/0G+/fDgQN/LocPu5ekpOy3L1HizyU83L1EREDx4lC0qHspVgzCwtzfh4a6l7Aw95v3\n+UtIiHspUMD33lTEf61f77l9KRCIyDWx1jLph0kMWjCIQsGFmNV1Fh1qdcjTGs6cgV273J+Sfvnl\nz2XXLti71/0J/3ylSkGFCu4lIgIaNYLrroOyZd1fy5SB0qXd65Uo4f40LZJfKBCIyFXbmriVAfED\nWLprKbF1Ynm35btePSpw4gRs2QI//gibN8O2be5l5073OXZwf/KuWhWqVYM77oDOnaFyZbjhBvfX\nihXdn9BFJHsKBCKSY2npabz69au8sfoNKodX9spYgYMH3YdB16+HDRtg0yb3G7/L5X6+ShWoWROa\nN4cnn3R/X706XH+9OxSIyLXRn4+IXJG1lrnb5zJowSD2J+/nxbtf5IW7X6BIwSK52m9SEnz3HXz7\nLaxZ4w4B+/e7nytZEurVgxYt4NZb3cstt7jP1YuI5ykQiMhlJexP4NnFz7J893Kib4xmUY9F3FT6\npqvej7Xu8/v/+x+sXAnffOM+DWCt+3x9w4bQpw9ERrqXqlU1+E4kLykQiEi29ibt5aWlLzHph0nc\nfN3NxHeLp0X1FjmebdBa9/n+5cthxQp3CDhwwP0mX6cONGkCf/ub+3x/jRruS9xExDkKBCKSxcGU\ng4xcNZJx68ZRvHBxPmzzIf3q97vibYqtdY/2X7IEli1zB4HDh93XzzdsCL17w913w513uk8HiIhv\nUSAQESBrECgUXIjn73qeZxo/Q7HCxS65ze+/w1dfuUPAkiWwZ497YF+DBvDII3Dffe4AEBaWdz+H\niFwbBQKRfG5v0l5GfzOaDxI+oHBwYZ6/63kG3TGIkkUu/hh/+jSsXg0LF8KiRX9OinLLLdChAzzw\nANx7r3uyHhHxLwoEIvnU2t/WMuqbUXy++XOKFS7G83c9z+BGgykRUuLcOta6L/lbuBAWLHCfBkhN\ndU/iEx0NTz3lvvyvYkXnfg4R8QwFApF85HTGaWZvnc3ba95m1a+ruLHkjbzd4m161+tN0ULu6/mS\nktxjABYudC+7drnHAdx9N7z8MsTEQN26GgQoEmgUCETyga2JW/l4/cdM2DiBxLREmlRpwhddv6BN\njTZYVzAJCe5TAAsXuucEyMiAm26C1q3dAeC++3T9v0igUyAQCVBH0o4wc8tMJv4wkZV7V1K6SGl6\n1e1Fv3oPUyjpFpYsgU4vuo8GJCW5b9zTrBm89577dEBEhNM/gYjkJQUCkQDyx6k/mL11NlN/msqi\nnxfhsi7uj7ifd+6ZSuHdHVg5uTAtH4F9+9xXAzRu7L51arNm7ksDNfWvSP6lP38RP7f7+G6+3P4l\nX27/kuW7l3Mq4xRRZe6mS7ExpP/QibWflOOpPe4JgerXhy5d4P773VcD6DSAiJylQCDiZ5JPJbNy\n70qW7lrK/J3z+en3nyhgClLV3EuNvSPYt+QvJOyuzPdB7nsBdOgATZu6ZwbUhEAicikKBCI+7uiJ\no6z9bS3/2/M/vvplGesOrCXDZhCaUYHCv0UTtOafnNnZnAMFitOgAbTvAffcA40aQfHiTlcvIv5C\ngUA8Li4ujtjYWKfL8Espp1PYdGgT6w98z/Kda/jm12/57eR2AAqcuo6MnU2xu3oRtLcpEWVrcEdD\nwx1Pud/8b7kFgoO9U5de08Ci11Oy47VAYIwZADwLlAc2AgOttd95qz/xHfrP5spOpJ9gx9EdbDqw\njW93bmH9vk1s+2MDR1w/g7GQUQAO1oN90RT6/e/UDLuDBtVv5Pamhshn4LbboEju7jx8VfSaBha9\nnpIdrwQCY0xX4C3gMWAtMARYaIypYa1N9EafIr7EZV38nvo7Ow7/SsLPu9i07xd2HtnFvpRdHDqz\ng5QCu91v/ABppeHQrRQ82pqqwXWpXaoeDaveTL1Whbn1Vvflf5oESES8zVtHCIYAH1prJwIYY/oD\nrYF+wEgv9SniVdZajp1IYvehI+w+fIRdhw+z58ghfjt+iIMphzhy8hBH038j2ezjZKHfIDj9z41P\nFodj1Sh8IoJSdOSW0NrUKlOTyCo1qXtTGWrUgPLl3VcCiIg4weOBwBhTEIgCXjvbZq21xpglQGNP\n9ydyKdZaUk6e4mjyCZJST3A85QTHU9M4lprGsZRUkk6k8seJNI6nJZN0Ipk/TiWTfDqZ1PRkUs4c\nJzXjOCc5zqmg45wJPkZGoaMQfObijtJKEXSiHIXTy1HMVqVqwbspb6/nhhKVqHZdJSKrVuPmiJJU\nqmQoXDjvfw8iIjnhjSMEZYBg4NAF7YeAmtmsHwIw6tOZlFu27pI7tdZ6qr7s98+17d+Vk82yqd11\nQVt2P9/Zms5/6sLtsmyb+dV13nZ/7sOeW+/8fVhrcVnrXs/azJ/HuveRuY3rvO0tmW0u9/cul7s3\nl3W5F5dl44+buX/Qs7hclgyb4d7/2eetiwzrwmUzMtfPIIPMdlcG1rrIIIMMewZrM8iw7u9dZODi\nDC7OkJH51UU6Lk7jIh0blI7LpGPNaWzwaQg6nfUT+pWcCYEzYQSfCSXYFUZBVzEKm2IUDS5H2eDq\nhBUsRsmQEpQsEk6ZoiUoGx5OhVIlqVq2JNeVLniFCX12k5S0m6SknJfja5KSklh/9taG4vf0egaO\nLVu2nP02JLf7Mp5+ozXGVAB+Axpba9ec1z4SuNtae+cF63cDJnu0CBERkfylu7V2Sm524I0jBIlA\nBlDugvayXHzUAGAh0B3YDZz0Qj0iIiKBKgSoivu9NFc8foQAwBjzLbDGWjso87EB9gLvWGvf8HiH\nIiIikiveuspgFDDBGJPAn5cdhgLjvdSfiIiI5IJXAoG1dpoxpgzwL9ynDjYAMdba373Rn4iIiOSO\nV04ZiIiIiH/R/GciIiKiQCAiIiIOBwJjzFBjzCpjTKox5ugl1qlsjJmXuc5BY8xIY4yCjJ8wxuw2\nxrjOWzKMMX9zui7JGWPMAGPMLmPMCWPMt8aYBk7XJNfGGDPsgr9FlzFms9N1Sc4YY+4xxswxxvyW\n+dq1y2adfxlj9htj0owxi40x1a+mD6ffWAsC04Bx2T2Z+cYfj3vwYyOgN9AH92BF8Q8WeBn34NLy\nQAXgXUcrkhw57yZlw4D6uO9aujBzwLD4px/582+xPHC3s+XIVQjDPUB/AFw8ta4x5nngSeBxoCGQ\nivvvtVBOO/CJQYXGmN7AaGttqQvaWwJzgApn75JojHkcGA5cZ63NZmJ58SXGmF24X9t3nK5Frs4l\n5hP5Ffd8IrpJmZ8xxgwD2ltrI52uRXLHGOMCOlhr55zXth94w1o7OvNxcdyTAfa21k7LyX6dPkJw\nJY2ATRfcMnkhEA7c4kxJcg1eMMYkGmPWG2OeNcYEO12QXN55Nyn76mybdX960E3K/NtNmYecfzbG\nTDLGVHa6IMk9Y0wE7iM+5/+9/gGs4Sr+Xr01MZGnlCf7mySdfW5j3pYj1+BtYD1wFLgT99Gd8sCz\nThYlV3S1NykT3/ct7lOu23CfuvsH8LUxpo61NtXBuiT3yuM+jZDd32v5nO7E40cIjDGvZzNw5cJB\nZTU80JXz5zryqat5ja21Y6y1X1trf7TW/gd4BhiY+QlU/I9Bf3t+yVq70Fo7I/NvcTHQCigJdHG4\nNPGeq/p79cYRgjeB/15hnV9yuK+DwIWjms/eNCm7GyVJ3sjNa7wG97+7qsAOD9YknnW1NykTP2Ot\nTTLGbAeuaiS6+KSDuN/8y5H177Ms8H1Od+LxQGCtPQIc8dDuvgGGGmPKnDeOIBpIAnS5jENy+RrX\nB1zAYc9VJJ5mrU3PvBdJM9wDe88OKmwGaIBoADDGFAVuBCY6XYvkjrV2lzHmIO6/zx/g3KDCO4D3\ncrofR8cQZA5oKQVUAYKNMXUzn9qZeU5rEe43/k8zL6moAPwbGGutTXeiZsk5Y0wj3P8glwHJuMcQ\njAI+tdYmOVmb5IhuUhZAjDFvAHOBPcD1wD+BM0Cck3VJzhhjwnAfzTGZTdUy3zOPWmt/BcYALxtj\ndgK7cb9X7gNm57gPJy87NMb8F+iVzVNNrbVfZ65TGfc8Bffhvq5yPPCitdaVR2XKNTLG1Afexz0I\nrTCwC/enkdEKdP7BGPNX4G/8eZOygdbadc5WJdfCGBMH3AOUBn4HVgIvWWt3OVqY5Igx5l7cH64u\nfNOeYK3tl7nOP4DHgBLA/4AB1tqdOe7DF+YhEBEREWf5+jwEIiIikgcUCERERESBQERERBQIRERE\nBAUCERERQYFAREREUCAQERERFAhEREQEBQIRERFBgUBERERQIBARERHg/wMGfSntlEBwVwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f6c46195ad0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "def softplus(x):\n",
    "    return np.log(1 + np.exp(x))\n",
    "\n",
    "def plot(x, func, label):\n",
    "    y = func(x)\n",
    "    plt.plot(x, y, label=label)\n",
    "    \n",
    "x = np.linspace(-10, 10, 100)\n",
    "plot(x, sigmoid, label='sigmoid')\n",
    "plot(x, softplus, label='softplus')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some useful properties:\n",
    "![properties](images/properties.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Bayes' Rule\n",
    "\n",
    "A lot of times, we actually have $P(y \\hspace{.1cm} | \\hspace{.1cm} x)$ and we need to find $P(x \\hspace{.1cm} | \\hspace{.1cm} y)$. **Bayes' Rule** helps us to compute that quantity:\n",
    "\n",
    "$$ P(x \\hspace{.1cm} | \\hspace{.1cm} y) = \\frac{P(x)P(y \\hspace{.1cm} | \\hspace{.1cm} x)}{P(y)} $$\n",
    "\n",
    "where $P(y) = \\sum_x P(x)P(y \\hspace{.1cm}  | \\hspace{.1cm} x)$. Bayes' Rule is one of most important concepts in probability theory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12. Technical Details of Continuous Variables\n",
    "\n",
    "A proper formal understanding of continuous random variables and probability density functions requires developing probability theory in terms of a branch of mathematics known as **measure theory**. Measure theory is more useful for describing theorems that apply to most points in $\\mathbb{R}^n$ but do not apply to some corner cases. The book describes the following three concepts:\n",
    "- **measure zero**: Measure theory provides a way to describe that a set of points is negligibly small. Such a set is said to have measure zero. For e.g., in the space of $\\mathbb{R}^2$, a line has measure zero, whereas a polygon has positive measure. An individual point also has a measure zero. The union of countably many sets each having measure zero, also has a measure zero.\n",
    "\n",
    "- **almost everywhere**: A property that holds almost everywhere holds throughout all space except for on a set of measure zero. Some results that hold for all discrete values, hold \"almost everywhere\" only for continuous values.\n",
    "\n",
    "- Handling continuous functions that are **deterministic functions** of one another. Let $\\boldsymbol{y} = g(\\boldsymbol{x})$. It may be natural to think $p_y(\\boldsymbol{y}) = p_x(g^{-1}(\\boldsymbol{y}))$. However, this is not true. E.g. $y = x/2$ where $x$ follows the uniform distribution $U(0, 1)$. Now, assigning $p_y(\\boldsymbol{y}) = p_x(2y)$ would mean that $p_y$ would be $0$ everywhere except [0, 1/2]. Thus, we get:\n",
    "$$ \\int p_y(y)dy = \\frac{1}{2}$$\n",
    "This violates the probability distribution. The reason for this is that the transformation $g$ distorts the space, which is not accounted for, above. The probability in a volume of $\\delta x$ is given by $p_x(x)\\delta x$. Taking a small volume in the $\\boldsymbol{y}$ space, the probalility would be $p_y(y)\\delta y$. Thus: <br> <br>\n",
    "$$ |p_y(y)\\delta y| = |p_x(x)\\delta x| $$\n",
    "$$ \\Rightarrow p_x(x) = p_y(y) \\hspace{.1cm} |\\frac{\\partial g(x)}{\\partial x}|$$\n",
    "In higher dimensions, the derivative generalizes to the derivative of the Jacobian Matrix, $J_{i,j} = \\frac{\\partial x_i}{\\partial y_j}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13. Information Theory\n",
    "\n",
    "**Information theory** is a branch of applied mathematics that revolves around quantifying how much information is present in a signal. It was originally invented to study sending messages from discrete alphabets over a noisy channel, such as communication via radio transmission.\n",
    "\n",
    "To give a flavour, given below is what is known as the **Binary Symmetric Channel (BSC)**:\n",
    "![bsc](https://i.stack.imgur.com/ipht5.gif)\n",
    "\n",
    "It is named so, because both the input and output can be either $0/1$. $p$ represents the probability that given that we transmitted $0$ (or $1$), we received an error, i.e. $1$ (or $0$ in case $1$ was sent). So, the probability of error in transmission is $p$. Here, we used one bit to represent one message, but we could use another coding scheme where $0$ is represented by $00$ and $1$ is represented by $11$. The advantage here is that the probability of error in transmission reduces in this case, but at the cost of increased number of transmissions. In this case, we are representing $1$ message using $2$ bits. Thus, the rate of the code is $1/2$. Information theory deals with the design of optimal **codes**, that can maximize this rate of transmission.\n",
    "\n",
    "The basic intuition behind information theory is that learning that an **unlikely** event has occurred is more informative than learning that a likely event has occurred. A message saying “the sun rose this morning” doesn't give any valuable information but a message saying “there was a solar eclipse this morning” is very informative. Thus, information needs to be quantified in a way that high probability events should have lower information, and low probability events should have higher. Also, independent events should have additive information. For example, ﬁnding out that a tossed coin has come up as heads twice should convey twice as much information as ﬁnding out that a tossed coin has come up as heads once.\n",
    "\n",
    "The **self-information** of an event x = $x$ satisfies all the above mentioned properties: <br> <br>\n",
    "$$ I(x) = - \\text{log} P(x)$$\n",
    "\n",
    "The information in an entire probability distribution is quantified using the **Shannon Entropy**. Nearly deterministic events have low entropy and those closer to uniform, have higher. The logarithm here refers to the natural log with base $e$. It is given by: <br> <br>\n",
    "$$ H(x) = \\mathbb{E}_{x \\sim P}[I(x)] = - \\mathbb{E}_{x \\sim P}[\\text{log} P(x)]$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f072482cfd0>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgsAAAFkCAYAAACuFXjcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xl4lNX5xvHvQwggiJGKgHXBHVBQSMSwWLUgilqoO0YR\nFRXFtVTcawF3pbiLqLQiolHcQasouGBVAiaIG1R+KoKoCJbGBZDt/P44kxIgCZnJzJxZ7s91zYV5\n8868Ny8x88xZzTmHiIiISHXqhQ4gIiIiqU3FgoiIiNRIxYKIiIjUSMWCiIiI1EjFgoiIiNRIxYKI\niIjUSMWCiIiI1EjFgoiIiNRIxYKIiIjUSMWCiIiI1CimYsHMLjCzL81spZnNMLPONZz7hpmtr+Ix\nOfbYIiIikixRFwtm1g8YBQwDOgFzgClm1ryapxwLtKr0aA+sAybGElhERESSy6LdSMrMZgAlzrlL\nIl8bsAi42zl3Wy2e/ydgOLCDc25l1IlFREQkqaJqWTCzXKAAmFZxzPlqYyrQtZYvMxAoVqEgIiKS\nHupHeX5zIAdYssnxJUCbLT3ZzA4E9gXO3MJ52wFHAAuAVVFmFBERyWaNgF2BKc65H+LxgtEWC9Ux\noDb9GWcBHzvnSrdw3hHAY3VOJSIikr1OBR6PxwtFWywsww9ObLnJ8RZs3tqwETPbCugH/KUW11kA\nMGHCBNq1axdlRInVkCFDuOOOO0LHyCq658mne558uufJNXfuXPr37w+R99J4iKpYcM6tMbNSoCcw\nCf43wLEncPcWnt4PaEDtWgxWAbRr1478/PxoIkod5OXl6X4nWSrfc+egvNw/aqNpU9h2W6iX4qu3\npPI9z1S658HErRs/lm6I24FHIkXDTGAI0BgYB2Bm44GvnXNXb/K8s4DnnXPLY48rInW1YgV89hnM\nmwfffgvLlsEPP/g/Kz9++AHWro3utevVg+22g+bNN/xZ+dGqFey9N7RpA9tsk5i/n4jEX9TFgnNu\nYmRNhevw3REfAEc455ZGTtkJ2OhXjJntBXQDetUtrojUhnPw3Xe+INj0sXDhhvOaNNn4zXynnaBj\nx42PbbPNllsLnIOfftq84Fi2DObM2VCM/Pjjhuf89rfQtu2GR5s2/s+ddkr91gmRbBPTAEfn3Ghg\ndDXf61HFsfn4WRQikgA//wwzZsC//uUfs2ZteGOuXx/23NO/EZ9yyoY35jZtoFmz5Ob86acNrRoV\nj+nTYexYWL3an9O4MRQUwEEH+Ue3br57Q0TCiddsCMkARUVFoSNknVjv+XffwTvv+MLg7bfhgw9g\n3Tr/5n/QQXDFFbDvvr4w2H13yM2Nc/AYNW3qC4GCgo2Pr1sHCxb44uHTT6GkBP7xD7j5ZjCD9u03\nFA8HHQS77BJ7Bv2cJ5/uefqLegXHZDCzfKC0tLRUg2JEgFWrYOpUeOEFePNN+L//88d3223jN9G2\nbTOnCd85+PzzDa0l77zjiwmAnXeGQw6Bvn3hyCNh663DZhVJJWVlZRT4irzAOVcWj9dUy4JIivrv\nf+Gll+D55+Hll+GXX/zgwN694Xe/g+7dYccdQ6dMHDPffbLnnnDGGf7Y0qXw7ru+eHj1VZgwARo2\nhMMOg2OPhT59oEWLoLFFMpKKBZEUsnixbz14/nl44w0/G6FzZ7jmGjjmGN9yYBY6ZTjbbw9//KN/\njBwJX3yx4X4NGuRbI7p39/fq2GN9F4yI1F2GNFiKpK/ly+Hee6Gw0M8EuOQS/6Z3552waBHMnAlX\nXQXt2mV3oVCV3XeHIUPgrbf8OI6xY/24jWuugT32gP32g1tvhSU1LhknIluiYkEkAOf8wMQBA/wU\nwiFDYIcd4NFH4fvv4bXX4IILfPEgtbP99jBwIEya5KdpPv20Hxg5bJi/j8cfD6+84gdTikh01A0h\nkkRLl8L48f4T8Lx5/tPvsGG+T75Vq9DpMsfWW/vi4PjjfcvNhAnw0EN+MOQuu8BZZ/nCQsWYSO2o\nZUEkwdav9zMZ+vXzAxKvvtovfDRtml9z4MorVSgkUrNmcNFFfnGoGTOgVy+47TZo3Rr+8Ac/5mHN\nmtApRVKbigWRBPn1V9+C0Latf4P66CPff754MRQXQ48emTPNMR2Y+XEhY8f6Za7vv9+PZTjmGD8F\n9Y47/IwTEdmcflWJxNkvv/jBiXvs4Ufod+jgp/p98okfm9C8eeiE0rSp/7eZNQtmz/bF3OWX+9aG\n66/3XRcisoGKBZE4Wb4cbrjBv+EMHern/n/yCTzzjJ/Op5kMqaljR3j4Yb/QVVER3HSTH9dwxRV+\nhoWIqFgQqbMlS/y4g9atfbFw8sn+jWfcOD/dUdJD69Zwzz1+2ekLL/TdFLvu6melLFgQOJxIYCoW\nRGL0zTf+TWXXXWH0aDj/fP+mcu+9/pikp5Yt/Z4UCxfCtdfCxIl+FckBAzYssy2SbVQsiETpl19g\n+HDYay8/UPGaa+Crr+CWWzSrIZNsu63/t12wAG6/HV5/HfbZx487+c9/QqcTSS4VCyK1tG6d79ve\nay//yfPCC/1yw3/5S/K3epbkadIELr4Y5s/3ReLYsb6l4c47N2yrLZLpVCyI1MK0aX5b5YED4eCD\n/YJKt94KeXmhk0mybLWVXyPj//4PTjoJLr3UbwP+3HN+RU6RTKZiQaQG8+b5nQwPOwwaN4b33oMn\nnvDz8iU7tWwJY8b4RZ723BOOO85vl/3++6GTiSSOigWRKixb5rsZ2rf30x+ffBLeeQe6dAmdTFJF\n+/Z+6/ApU/y02c6d4bTT/OZfIplGxYJIJc7BI4/A3nv7/QRuuQXmzvXNzlonQapy+OHwwQfw4IN+\nA7C2bf1qkNqwSjKJigWRiAULoHdvv6nT0Uf7AW1Dh0LDhqGTSarLyYFzzvE/M2ef7cczdO8OH38c\nOplIfKhYkKy3bh3cdZdvVp47F/75T79V9Pbbh04m6aZpU/+z9M478OOPkJ/vdxX99dfQyUTqRsWC\nZLVPPvGfAIcMgTPP9F8feWToVJLuunb1e05cdZVfPjo/3w+OFUlXKhYkK61eDSNGQKdOUF7uN3q6\n5x7/yVAkHho29D9jZWV+rYbu3eGSS+Dnn0MnE4meigXJOiUl/pPeDTf4zYJmz4Zu3UKnkkzVoYNv\nVRg1Ch56yHd3TZkSOpVIdFQsSNZYu9avtti1q19gp7TUb0fcqFHoZJLpcnJ8V9fHH/u1GXr3hnPP\nhRUrQicTqR0VC5IVFi6EQw/1UyFvuMF/0ttvv9CpJNvsvrufXjlmDIwfDwce6MfJiKQ6FQuS8Z57\nDvbf3y+WM326X7K3fv3QqSRbmflWhYoVHzt39t0TWjJaUpmKBclYq1b5VRiPOw569PAL52hsgqSK\nffeFmTP9qo+DBsHJJ/vBtiKpSMWCZKR586Cw0O8QOHo0PP20doaU1NO4MTzwAEycCK+84mfnlJSE\nTiWyORULklGcg3Hj/A6Ra9b4T26DB2upZkltJ57oW75atICDDoKRI2H9+tCpRDZQsSAZ46effJPu\nmWdCURHMmqVBjJI+dtsN3n7bLxV9+eVw1FHw/fehU4l4KhYkI3z2GRxwAEyaBI8/7rsfmjQJnUok\nOrm5fsbOlCl+/Y+OHX3rmEhoKhYk7b36qp+ClpPj104oKgqdSKRuKnaybN0aDj7YF8AiIcVULJjZ\nBWb2pZmtNLMZZtZ5C+fnmdl9ZvZN5DnzzKx3bJFFPOf8pj1HHumX0n3vPdhrr9CpROJjhx3gjTeg\nXz849VS/z4TGMUgoUc82N7N+wChgEDATGAJMMbO9nXPLqjg/F5gKfAccB3wDtAb+W4fckuVWr4bz\nz4e//91vI33LLb5lQSSTNGrkB+x26ODHMXz6KUyYoD1MJPliWZpmCPCAc248gJmdBxwNDARuq+L8\ns4BtgS7OuXWRYwtjuK4I4Ad9HX+878sdNw5OPz10IpHEMfMFcbt2voutWzc/Nme33UInk2wSVTdE\npJWgAJhWccw55/AtB12reVof4D1gtJl9Z2YfmdlVZqbxEhK1Dz/04xPmz4c331ShINnj6KNhxgxY\nudKv+vjWW6ETSTaJ9g27OZADLNnk+BKgVTXP2R04MXKtI4HrgUuBq6O8tmS555/3n6p+8xs/LbJr\ndeWpSIbaZx+/aNN++8Fhh8GDD4ZOJNkiXp/uDahuZfN6+GJikHNutnNuInAjMDhO15YM5xzceCMc\ne6wfzPj227DzzqFTiYSx3XZ+auWgQX6PiYsv9juqiiRStGMWlgHrgJabHG/B5q0NFb4FVke6KyrM\nBVqZWX3nXLU/5kOGDCEvL2+jY0VFRRRpblzWWLsWzjvPD2QcPhyuvRbqqQNLslxuLtx3H7RvDxdd\nBAsWwJNP+q3XJbsUFxdTXFy80bHyBGwyYi7Krc7MbAZQ4py7JPK14Qcs3u2cG1nF+TcCRc653Ssd\nuwS4zDm3UzXXyAdKS0tLyc/PjyqfZI5Vq+CUU/xgrn/8AwYMCJ1IJPW88orfLO2AA2DyZNjk85Vk\nobKyMgoKCgAKnHNl8XjNWD6j3Q4MMrMBZtYWGAM0BsYBmNl4M7up0vn3A9uZ2V1mtpeZHQ1cBdxb\nt+iSyX780S93+/LLfotpFQoiVevdG6ZNg48/hkMOge++C51IMlHUxUJkzMGlwHXAbGA/4Ajn3NLI\nKTtRabCjc+5r4HCgMzAHuBO4A7i1TsklYy1d6reULi31qzP26RM6kUhq69oVpk/304oPOgi+/DJ0\nIsk0sayzgHNuNDC6mu/1qOJYCdAtlmtJdlm4EHr1gvJyPzWsY8fQiUTSQ/v28M47fqno7t39IMgO\nHUKnkkyhoWKSMj791E+NXLPG/9JToSASnd12g3/9C1q29HtKvPtu6ESSKVQsSEooKYHf/c6vofDO\nO7DHHqETiaSnli39gmUVazG8/HLoRJIJVCxIcK+9Bj17Qtu2vuthhx1CJxJJb3l5fpZEr17Qty9s\nMrNOJGoqFiSop5/2y9gefLAvGpo1C51IJDNstRU884zfsfLUU/26DCKximmAo0g8PPWU3xinXz+/\nIVRubuhEIpmlfn2/RslvfgMXXuhXQ73wwtCpJB2pWJAgnn/eL7jUrx+MH6/tpUUSpV49GDXK7155\n0UW+KD/33NCpJN2oWJCke/FFOOkkv+rcI4+oUBBJNDP42982LJ9evz6cdVboVJJOVCxIUr3yChx/\nPPzhDzBhgv+lJSKJZwZ33umnJp9zjv9/T1u8S23pV7UkzdSpcMwxcMQR8MQTGqMgkmxmcO+9vmA4\n80z//+App4ROJelAxYIkxZtv+ilcPXr4gY0NGoROJJKd6tWDBx7wBcNpp/mC4cQTQ6eSVKdiQRLu\n7bf99MiDDoJnn4WGDUMnEslu9er5bd/XrvUzkurXh2OPDZ1KUpnWWZCEeu89v3tkYaGfAdGoUehE\nIgJ+YPG4cXDCCX5W0uTJoRNJKlOxIAkzc6bfPrdTJ/+LqHHj0IlEpLL69eHRR30X4QknaGloqZ6K\nBUmIDz/0Axnbt4eXXoImTUInEpGq5ObC44/7wv7YY+H110MnklSkYkHibuFCOPJIvwPeP/8JTZuG\nTiQiNWnQACZOhEMO8QXDhx+GTiSpRsWCxNXy5f4TSoMGvlDIywudSERqo2FDv1fLHnv4Yn/hwtCJ\nJJWoWJC4WbUK/vhH+P57v/hSq1ahE4lINJo29UV+gwa+YFi+PHQiSRUqFiQu1q/3c7ZnzfKDGdu0\nCZ1IRGLRqpUv9r/7zhf/q1aFTiSpQMWC1Jlz8Oc/+zUUiouha9fQiUSkLtq08UX/rFkwYID/MCDZ\nTcWC1Nntt8Ndd/llZI85JnQaEYmHbt188f/MM/7DgHOhE0lIKhakToqLYehQuPpqGDw4dBoRiadj\njoF77vEfBm6/PXQaCUnLPUvM3njD71o3YADccEPoNCKSCOefD4sW+Q8FO+4IJ58cOpGEoGJBYvLR\nR/5Tx6GHwkMP+d3sRCQz3XQTLF7sPxy0bAm//33oRJJs6oaQqC1a5KdV7bGH78/UDpIimc0Mxo6F\ngw/2HxI++ih0Ikk2FQsSlZ9/9jtI1q/vl3HW6owi2aFBA//hYPfd/eZwS5aETiTJpGJBas05OPNM\n+PJLXyjssEPoRCKSTNtsAy++CGvW+I2nVq8OnUiSRcWC1NpNN/nlYB99FPbdN3QaEQlhxx39miol\nJXDJJaHTSLKoWJBaefFFuPZaGDZMaymIZLtu3eC++2DMGHjwwdBpJBk0G0K2aN48OOUUv/TrX/8a\nOo2IpIJzzoHZs+HCC31LY/fuoRNJIqllQWr03//6ImHnnWH8eKinnxgRibjzTujSBY4/Hr7+OnQa\nSST96pdqrVsHp57qd5F84QXNfBCRjTVo4McxNWgAxx2nTacymYoFqdZf/+p3nysuhj33DJ1GRFJR\nixbw3HN+7YVzz9UeEplKxYJU6amn/OyHm2+G3r1DpxGRVFZQAH//u++qvOuu0GkkETTAUTYzZw6c\ncQYUFcFll4VOIyLp4JRT/IDHoUOhQwfo2TN0IomnmFoWzOwCM/vSzFaa2Qwz61zDuaeb2XozWxf5\nc72ZrYg9siTSsmV+amSbNn55V+35ICK1dcstvkg46SS/eJtkjqiLBTPrB4wChgGdgDnAFDNrXsPT\nyoFWlR6to48qibZ2LfTr55d0fu45aNw4dCIRSSc5OfDEE9Csmf/Q8fPPoRNJvMTSsjAEeMA5N945\nNw84D1gBDKzhOc45t9Q5933ksTSWsJJYw4bBW2/50c2tVc6JSAyaNfOzpz7/3K/BIJkhqmLBzHKB\nAmBaxTHnnAOmAl1reOrWZrbAzBaa2fNmtk9MaSVhpk71gxmvvx4OOSR0GhFJZ/vuC/ffD4884peH\nl/QXbctCcyAH2HS/sSX47oWq/Bvf6tAXODVyzXfNbMcory0JsmQJ9O8Phx0GV1wROo2IZILTToPT\nT4fBg+Gzz0KnkbqK12wIA6qcXeucmwHM+N+JZu8Bc4FB+HEP1RoyZAh5eXkbHSsqKqKoqKiueSVi\n/XoYMMD/96OPaoVGEYmfe++FGTP8WKj33oNGjUInyjzFxcUUFxdvdKy8vDzu1zEXxQoakW6IFcDx\nzrlJlY6PA/Kcc8fW8nUmAmucc6dW8/18oLS0tJT8/Pxa55Po3XorXHUVTJkCvXqFTiMimeaDD/yS\n0IMGwd13h06THcrKyigoKAAocM6VxeM1o/oc6ZxbA5QC/5tBa2YW+frd2ryGmdUD2gPfRnNtib/3\n3oNrroErr1ShICKJ0bEjjBoF99wDzz8fOo3EKpZG59uBQWY2wMzaAmOAxsA4ADMbb2Y3VZxsZtea\nWS8z283MOgGP4adOjq1zeonZ8uV+0aUDD4QRI0KnEZFMdv75cOyxMHAgLFwYOo3EIuoxC865iZE1\nFa4DWgIfAEdUmg65E7C20lOaAQ/iB0Aux7dMdI1Mu5QAnIOzz4bycj9VMjc3dCIRyWRmfjnojh39\nSo9vvgn1tX5wWonpn8s5NxoYXc33emzy9Z+BP8dyHUmM+++HZ5/1D62nICLJ0KyZ35Tu4IP9mi43\n3hg6kURDY9+zzJw58Oc/wwUX+GZBEZFk6dbNr+Vy881+bRdJHyoWssjPP/spTG3bwt/+FjqNiGSj\nK67w+0f07+/XeJH0oGIhi1x0EXz9NTz5pOY7i0gY9er5NV2c82u8rF8fOpHUhoqFLPHYYzBuHIwe\n7XeUFBEJpVUrmDABXn0VRo4MnUZqQ8VCFli0yE9d6t9/w2qNIiIh9erluySuvdaPpZLUpmIhw1VM\nk2za1C+KIiKSKkaM8GOozjgDVq8OnUZqomIhw40d65v6xo6FbbcNnUZEZIOGDX336McfayplqlOx\nkMG++spPkzz7bOjdO3QaEZHN5ef7ZedvvBHK4rKLgSSCioUMtX69X1q1WTO/LruISKq65hro0MFv\naf3rr6HTSFVULGSoMWPg9dfhH/+AbbYJnUZEpHq5ufDII/Dvf8N114VOI1VRsZCBvvgCLrsMBg+G\nww4LnUZEZMv2288vA33LLTBrVug0sikVCxlm/Xo480xo0QJuuy10GhGR2rviCujUyXdHrFoVOo1U\npmIhw9x7L0yfDg8/DFtvHTqNiEjt1a/vuyM+/xz++tfQaaQyFQsZZP58uPJKv6zzoYeGTiMiEr19\n9/XjFv72N3j33dBppIKKhQyxbp3vfvjtb/2ObiIi6WroUCgs9Is1rVgROo2AioWMcdddvgp/+GFo\n0iR0GhGR2OXk+MWaFi2Cv/wldBoBFQsZYd48P0/5T3+C3/0udBoRkbpr08Yv1HTnnfD226HTiIqF\nNLdunW+q22UXuOGG0GlEROLnkkugWzffxfrLL6HTZDcVC2nu3nth5kzf/dC4ceg0IiLxk5Pjf7d9\n8w0MHx46TXZTsZDGvvnGb+86eLCvvkVEMs1ee/lxC3fcAR99FDpN9lKxkMaGDPGtCdqtTUQy2dCh\nvmgYPNgvPCfJp2IhTb36Kkyc6DeJ0tbTIpLJGjSA0aPhnXf8LAlJPhULaWjVKrjgAujRA045JXQa\nEZHE+/3voX9/uPxy+OGH0Gmyj4qFNHTLLfDVV3DffWAWOo2ISHL87W9+BtgVV4ROkn1ULKSZ+fP9\nCo2XXw5t24ZOIyKSPC1bwk03wd//7rskJHlULKQR5+D882HHHf0iTCIi2WbQIOjcGc47D9asCZ0m\ne6hYSCNPPglTp/q1FbbaKnQaEZHky8mBMWPg00/9MveSHCoW0kR5uZ8qedxxcNRRodOIiISTn+8H\neQ8f7vePkMRTsZAmrr0WfvrJr5MuIpLtrr8emjb1S0JL4qlYSANlZX7mw4gRsPPOodOIiISXl+dX\ndXzuOXjppdBpMp+KhRS3bp0fyLPvvnDxxaHTiIikjn794LDD4MILYcWK0Gkym4qFFPfggzBrFtx/\nP+Tmhk4jIpI6zPzKjt9+q2XvE03FQgpbsgSuugrOOgu6dw+dRkQk9ey1F1x5JYwcCXPnhk6TuWIq\nFszsAjP70sxWmtkMM+tcy+edbGbrzezZWK6bbS6/HOrXh1tvDZ1ERCR1XXkltG7tuyOcC50mM0Vd\nLJhZP2AUMAzoBMwBpphZ8y08rzUwEpgeQ86s8/77MH68X61su+1CpxERSV2NGvmZYq+/DpMnh06T\nmWJpWRgCPOCcG++cmwecB6wABlb3BDOrB0wA/gp8GUvQbOIcXHqpH9Q4sNq7KiIiFY46Cnr2hMsu\n08qOiRBVsWBmuUABMK3imHPOAVOBrjU8dRjwvXPu4VhCZpsXXoDp0/2mKfXrh04jIpL6zGDUKL9/\nzgMPhE6TeaJtWWgO5ABLNjm+BGhV1RPMrDtwJnB21Omy0OrVfqzC4YdD796h04iIpI/994czzvAr\nO/73v6HTZJZ4fW41YLNhJWa2NfAocI5zbnm0LzpkyBDy8vI2OlZUVERRUVGsOVPemDHw+efwzDOh\nk4iIpJ8bbvD76Nx0E9x2W+g0iVdcXExxcfFGx8rLy+N+HXNRDB2NdEOsAI53zk2qdHwckOecO3aT\n8/cHyoB1+IICNrRmrAPaOOc2G8NgZvlAaWlpKfn5+bX/26S55cthzz39/g8PPRQ6jYhIeho+HG6+\nGebNg912C50m+crKyigoKAAocM6VxeM1o+qGcM6tAUqBnhXHzMwiX79bxVPmAh2AjsD+kcck4PXI\nf2sLkEpuvBF+/dWveS4iIrG57DI/i+yqq0InyRyxzIa4HRhkZgPMrC0wBmgMjAMws/FmdhOAc261\nc+7Tyg/gv8BPzrm5zrm18flrpL8vvoB77oErroBWVY7+EBGR2mjSZEN3xIwZodNkhqiLBefcROBS\n4DpgNrAfcIRzbmnklJ2oZrCjVO/KK2H77f2USRERqZvTT/cDHv/8Zy3UFA8xDXB0zo0GRlfzvR5b\neO6ZsVwzk737Ljz1FDzyCDRuHDqNiEj6y8nxUykPOwyefhpOPDF0ovSmvSECq1iAKT8f+vcPnUZE\nJHP07AlHH+27d3/9NXSa9KZiIbCJE32f2qhRUE//GiIicTVyJCxcCPfeGzpJetPbU0CrVvmKt29f\nOPTQ0GlERDJPu3YwaJCfZbZsWeg06UvFQkB33w2LF2fHwiEiIqEMHw7r18N114VOkr5ULASydKlf\nV+G886BNm9BpREQyV4sWcPXVcP/98NlnodOkJxULgYwY4Tc+GTYsdBIRkcz3pz/Bb3/r996R6KlY\nCOCzz/weENdcA82bh04jIpL5GjXyS0BX7Oor0VGxEMB118EOO8BFF4VOIiKSPU4+GTp1gmuv1UJN\n0VKxkGRz58Ljj/tWhUaNQqcREcke9er5LuDp0+GNN0KnSS8qFpJsxAjYeWcYODB0EhGR7POHP8AB\nB8Bf/6rWhWioWEiijz7yizD95S/QoEHoNCIi2cfMf2h75x2YOjV0mvShYiGJRoyAXXeFM84InURE\nJHsdeSQUFqp1IRoqFpJkzhx45hk/sCY3N3QaEZHsZeYHms+YAa+8EjpNelCxkCTDh8Mee8Bpp4VO\nIiIivXpB9+5qXagtFQtJUFoKzz/vF2CqH9Om4CIiEk8VrQvvvw8vvhg6TepTsZAEw4f7JZ2LikIn\nERGRCr//PRxyiFoXakPFQoLNnOmrVrUqiIikloqZER984Ft/pXoqFhJs2DDYZx846aTQSUREZFOH\nHAI9e27YmVKqpmIhgd5914+0HT4ccnJCpxERkaqMGAEffgjPPhs6SepSsZBAw4ZBhw5w/PGhk4iI\nSHW6d4fDD/e/s9etC50mNalYSJDp0/3qYMOH+/XIRUQkdY0YAZ9+Ck89FTpJatLbWIIMGwYdO8Ix\nx4ROIiIiW9KlCxx1lP+Ap9aFzalYSIA33oA33/SVqloVRETSw4gR8O9/Q3Fx6CSpR29lceacb1Uo\nKIA+fUKnERGR2jrgAOjb1y/WtHZt6DSpRcVCnE2bBm+/7X/YzEKnERGRaAwfDvPnw2OPhU6SWlQs\nxNl118GBB/pdzUREJL106gTHHgs33KCxC5WpWIijGTN8q8LVV6tVQUQkXV11Ffzf/2lVx8pULMTR\nyJF+DwjYIEhAAAAZjElEQVSNVRARSV+dO8Ohh/rf6dozwlOxECfz58Nzz8Gll2oGhIhIurvsMigp\ngX/9K3SS1KC3tTi5/XZo0QJOOy10EhERqasjj4R994XbbgudJDWoWIiD77+Hhx+Giy6CRo1CpxER\nkboyg6FD/a7Bn34aOk14Khbi4N57/fbTgweHTiIiIvFyyinw29/CqFGhk4SnYqGOfvkF7rsPzj4b\nfvOb0GlERCReGjSAP/0JJkyAb74JnSasmIoFM7vAzL40s5VmNsPMOtdw7rFmNsvMlpvZz2Y228z6\nxx45tTz8MJSXw5AhoZOIiEi8DRoEDRvC3XeHThJW1MWCmfUDRgHDgE7AHGCKmTWv5ik/ADcAXYAO\nwMPAw2bWK6bEKWTtWj+w8aSToHXr0GlERCTe8vLg3HNhzBj46afQacKJpWVhCPCAc268c24ecB6w\nAhhY1cnOuenOuRecc/92zn3pnLsb+BA4KObUKeKZZ+DLL/0UGxERyUyXXOK7nB96KHSScKIqFsws\nFygAplUcc845YCrQtZav0RPYG3grmmunGuf8gh09e/rlQUVEJDPttJMf7HjnnbBmTeg0YUTbstAc\nyAGWbHJ8CdCquieZ2TZm9pOZrQYmAxc5516P8top5c03obQULr88dBIREUm0oUNh0SJ48snQScKI\n12wIA2paFPMnYH/gAOAa4A4zOzhO1w5i5EjYf3/olfYjL0REZEs6dPALNd12W3YuAV0/yvOXAeuA\nlpscb8HmrQ3/E+mq+CLy5Ydmtg9wFTC9posNGTKEvLy8jY4VFRVRVFQUZez4+vhjePllePRRbRgl\nIpItLrsMevSAV1+FI44IncYrLi6muLh4o2Pl5eVxv465KEskM5sBlDjnLol8bcBC4G7n3Mhavsbf\ngd2ccz2q+X4+UFpaWkp+fn5U+ZLhjDPg9dfh888hNzd0GhERSQbn/CZT224LU6eGTlO9srIyCgoK\nAAqcc2XxeM1YuiFuBwaZ2QAzawuMARoD4wDMbLyZ3VRxspldaWaHmdluZtbWzC4F+gOP1j1+8n39\nNTz2mF+oQ4WCiEj2MPOtC9OmQVlc3oLTR7TdEDjnJkbWVLgO3x3xAXCEc25p5JSdgLWVntIEuC9y\nfCUwDzjVOfd0XYKHctdd0KQJnHNO6CQiIpJsxx8Pu+7qx61t0vqf0WIa4OicG+2c29U5t5Vzrqtz\n7v1K3+vhnBtY6etrnXNtnHNNnHPNnXMHpWuhUF4ODzzg94Bo2jR0GhERSbb69eHSS+Gpp2DBgtBp\nkkd7Q0ThwQfh11/h4otDJxERkVDOPNOv7HjHHaGTJI+KhVpavdp3QfTvDzvsEDqNiIiE0qQJXHAB\njB0LP/wQOk1yqFiopWeegcWLffOTiIhktwsv9PsD/eMfoZMkh4qFWhozBg49FPbZJ3QSEREJrUUL\nOPFEP45t/frQaRJPxUItfPIJTJ/uBzaKiIiAf0/4/PPUXnMhXlQs1MKYMdCyJRxzTOgkIiKSKrp1\ng/bt4f77QydJPBULW/DLLzB+PJx1FjRoEDqNiIikCjPfujB5sl+wL5OpWNiC4mL46ScYNCh0EhER\nSTX9+0OjRn5mRCZTsVAD53zz0lFHQevWodOIiEiq2WYbOPVUeOghWLMmdJrEUbFQg1mz/PrfGtgo\nIiLVGTwYvvkGXnwxdJLEUbFQgzFjfItC796hk4iISKrq2BG6dMnsgY4qFqqxfDk88QScey7k5IRO\nIyIiqWzwYHjtNZg/P3SSxFCxUI1HHvGrcw0cuOVzRUQku514IjRr5vcQykQqFqrgnO+COO44v76C\niIhITbbaym8w9fDDsGpV6DTxp2KhCm++Cf/+twY2iohI7Z17rt9Y6qmnQieJPxULVbj/fmjXDg4+\nOHQSERFJF3vvDT17+pbpTKNiYRPffQfPPQfnnedX5xIREamtwYPh3Xfhww9DJ4kvFQub+PvfITcX\nBgwInURERNJN376www6Z17qgYqGSdev8SNaiIth229BpREQk3eTmwtlnw6OP+q0CMoWKhUr++U9Y\nuFADG0VEJHbnnAMrVsBjj4VOEj8qFioZMwYOOMA/REREYrHzztCnj39PcS50mvhQsRDx5Zfw8st+\nYKOIiEhdnHcezJkDM2aEThIfKhYiHnzQ7x528smhk4iISLo7/HDYfffM2S9CxQKwerWfBXH66dCk\nSeg0IiKS7urV84s0TZzoF2pKdyoWgGefhaVL/T+siIhIPJx5ph+zMG5c6CR1p2IBGDvWr9a4zz6h\nk4iISKbYfns44QR46KH0H+iY9cXC4sXw+utwxhmhk4iISKY5/XS/11BZWegkdZP1xUJxMTRo4HeY\nFBERiacePfzuxRMmhE5SN1lfLEyY4JfnzMsLnURERDJN/fp+VeDiYli7NnSa2GV1sfDxx34ebP/+\noZOIiEim6t8fliyBadNCJ4ldVhcLjz0Gv/kN9O4dOomIiGSq/Hxo2za9l3/O2mJh/Xr/D3fSSX7M\ngoiISCKYwamn+mn6v/wSOk1ssrZYePttWLRIXRAiIpJ4p5ziC4UXXgidJDYxFQtmdoGZfWlmK81s\nhpl1ruHcs81supn9J/J4rabzk+Wxx2DXXaFbt9BJREQk0+2+O3Tvnr5dEVEXC2bWDxgFDAM6AXOA\nKWbWvJqnHAI8DhwKdAEWAa+a2Q6xBI6HVav8Epynnuqbh0RERBKtf3+YMgW+/z50kujF0rIwBHjA\nOTfeOTcPOA9YAQys6mTn3GnOuTHOuQ+dc58BZ0eu2zPW0HX1z39CebkvFkRERJLhxBP9B9Qnnwyd\nJHpRFQtmlgsUAP+bAOKcc8BUoGstX6YJkAv8J5prx9OECVBQAO3ahUogIiLZZrvt4Kij0nOBpmhb\nFpoDOcCSTY4vAVrV8jVuBRbjC4ykW74cXnpJrQoiIpJ8/fvDzJkwf37oJNGJ12wIA7a4TYaZXQmc\nBBzjnFsdp2tH5emn/SpaJ58c4uoiIpLN/vAH2Gab9BvoWD/K85cB64CWmxxvweatDRsxs6HA5UBP\n59wntbnYkCFDyNtkHeaioiKKiopqHXhTEybAYYfBDsGGV4qISLbaais4/nj/XjRsWN0H2RcXF1Nc\nXLzRsfLy8rq9aBXMRblvppnNAEqcc5dEvjZgIXC3c25kNc+5DLgaONw5N6sW18gHSktLS8nPz48q\nX02++spPlxw/Hk47LW4vKyIiUmuvvw49e8J770GXLvF//bKyMgoKCgAKnHNx2e8ylm6I24FBZjbA\nzNoCY4DGwDgAMxtvZjdVnGxmlwPX42dLLDSzlpFHkzqnj1Jxsa/qjjkm2VcWERHxDjkEdtwxvboi\noi4WnHMTgUuB64DZwH7AEc65pZFTdmLjwY6D8bMfnga+qfS4NPbY0XMOHn3UFwpNmybzyiIiIhvk\n5PgVHZ94AtasCZ2mdmIa4OicG+2c29U5t5Vzrqtz7v1K3+vhnBtY6evdnHM5VTyui8dfoLbmzIFP\nP9XyziIiEl7//rBsGbz6augktZM1e0NMmADbbw+9eoVOIiIi2W6//aB9+/RZcyErioV16/x4hX79\nIDc3dBoRERHfuvDCC/DTT6GTbFlWFAtvvgnffKMuCBERSR2nnAIrV8Jzz4VOsmVZUSxMmAB77gkH\nHhg6iYiIiLfzznDooenRFZHxxcLKlfDMM9phUkREUs+pp8K0afDtt6GT1Czji4XJk31/kPaCEBGR\nVHPCCVC/vp9GmcoyvliYMAEKC2GvvUInERER2di220KfPqnfFZHRxcIPP8DLL2tgo4iIpK7+/aGs\nDObODZ2kehldLLz0kt9h8oQTQicRERGp2pFHwtZbw/PPh05SvYwuFiZN8jMgWrXa8rkiIiIhNGwI\nhx/u37NSVcYWC7/+ClOmQN++oZOIiIjUrG9fKCmB778PnaRqGVssvPUW/PyzHzgiIiKSyo46yv/5\n0kthc1QnY4uFSZNgl12gQ4fQSURERGq2/fbQtWvqdkVkZLHgnF9foW9fLcQkIiLpoW9fvwvlqlWh\nk2wuI4uFjz6ChQvVBSEiIumjTx9YsQLeeCN0ks1lZLEwaRI0bQqHHBI6iYiISO20awd77JGaXREZ\nWSxMngxHHOGno4iIiKQDM9+6MHmy705PJRlXLHz7LcycqS4IERFJP336wOLFMHt26CQby7hi4aWX\noF69DdNQRERE0sXvfgd5eb51IZVkXLEweTJ06wbNm4dOIiIiEp3cXL/8s4qFBFq5El57TV0QIiKS\nvvr0gdJS3x2RKjKqWJg2zRcMWuJZRETS1ZFHQk4OvPhi6CQbZFSxMHky7LkntGkTOomIiEhsmjXz\nYxdSqSsiY4qF9ev9je3TR6s2iohIeuvTB6ZOhV9+CZ3Ey5hioazMT5tUF4SIiKS7Pn387slTp4ZO\n4mVMsTB5Mmy7LXTvHjqJiIhI3ey1F7RtmzpdERlTLEya5AeF5OaGTiIiIlJ3ffr4QY7r14dOkiHF\nwqJF8MEH6oIQEZHM0bcvLFkCs2aFTpIhxcLkyVC/PvTuHTqJiIhIfHTtCtttlxobS2VMsXDwwX7M\ngoiISCbIyYGjj06NcQtpXyz8/DO8/rpWbRQRkczTpw989BEsWBA2R9oXC6++CqtXq1gQEZHMc/jh\nfuB+6NaFtC8WJk+GffaBPfYInURERCS+ttkGfv97FQt1sm6d35JarQoiIpKp+vSBN9+EH38MlyGm\nYsHMLjCzL81spZnNMLPONZy7j5k9HTl/vZldHHvcjZWUwNKlKhZERCRz9ekDa9bAlCnhMkRdLJhZ\nP2AUMAzoBMwBpphZ82qe0hj4HLgC+DbGnFWaPBmaN4cuXeL5qiIiIqmjdWvYb7+wXRGxtCwMAR5w\nzo13zs0DzgNWAAOrOtk5975z7grn3ERgdexRNzdpkp9WkpMTz1cVERFJLX36+G73tWvDXD+qYsHM\ncoECYFrFMeecA6YCXeMbrWZffAGffqouCBERyXx9+sB//gPvvRfm+tG2LDQHcoAlmxxfArSKS6Ja\nmjwZGjTw00pEREQyWefO0LJluK6I+nF6HQNcnF7rf4YMGUJeXt5Gx4qKiigqKuLVV/2qjU2bxvuq\nIiIiqaVePTjqKL+20G23bTheXFxMcXHxRueWl5fH/frRFgvLgHVAy02Ot2Dz1oY6u+OOO8jPz9/s\nuHN+JsQFF8T7iiIiIqmpa1d45BH45Rdo0sQfq/gAXVlZWRkFBQVxvXZU3RDOuTVAKdCz4piZWeTr\nd+OarAZffAE//ACFhcm6ooiISFiFhX676tLS5F87ltkQtwODzGyAmbUFxuCnR44DMLPxZnZTxclm\nlmtm+5tZR6ABsGPk65jXXCwp8X8eeGCsryAiIpJe9t3XtyhUvAcmU9RjFpxzEyNrKlyH7474ADjC\nObc0cspOQOXJHb8FZrNhTMPQyOMtoEcsoUtK/PLOzatb2UFERCTD5OTAAQekSbEA4JwbDYyu5ns9\nNvn6K+K8rHRJibogREQk+xQWwuOPJ/+6abc3xK+/wuzZKhZERCT7FBbC11/D4sXJvW7aFQsffOC3\npFaxICIi2abivS/ZXRFpVyyUlPjFmDp2DJ1EREQkuXbc0T9ULGxBSQl06gQNG4ZOIiIiknxduqhY\n2CINbhQRkWxWWAjvvw/r1iXvmmlVLCxbBp9/rmJBRESyV2GhX8Xxk0+Sd820KhZmzvR/qlgQEZFs\nVVDg11xIZldEWhULJSV+Iabddw+dREREJIwmTaB9exUL1Sop8Us8m4VOIiIiEk5hoYqFKjnnuyHU\nBSEiItmusNCPWfjpp+RcL22KhfnzYflyFQsiIiKFhf5D9PvvJ+d6aVMsaKdJERERr21baNo0eV0R\naVUs7L03NGsWOomIiEhYOTnQubOKhc1oMSYREZENCgthxgzfHZFoaVEsrFoFc+aoWBAREalQWAjf\nfQeLFiX+WmlRLMyeDWvWqFgQERGpkMwdKNOiWCgp8RtH7bdf6CQiIiKpoVUr2GUXFQv/U1IC+fl+\na2oRERHxkrU4U1oUCzNmqAtCRERkU4WFUFrqu+oTKeWLhe+/hwULVCyIiIhsqrAQVq6Ejz9O7HVS\nvlioaF7p0iVsDhERkVSTnw/16ye+KyItioUWLaB169BJREREUkvjxn7wv4qFyGJM2mlSRERkc8kY\n5JjSxcL69dppUkREpCaFhTBvHpSXJ+4aKV0sfPUV/PijigUREZHqVOxAOWtW4q6R0sXCRx/57ofO\nnUMnERERSU177w15eYntikjpYuHjj/02nHl5oZOIiIikpnr14MADs7xYUBeEiIhIzSoGOSZqB8qU\nLhbmz1exICIisiWFhX4Rw6++Sszrp3SxsH69igUREZEtqXivnDEjMa+f0sVCw4bQoUPoFCIiIqlt\n++1ht90SN24hpYuFdu38MpYiIiJSs0QuzpTSxUL79qETZJfi4uLQEbKO7nny6Z4nn+55chQWQllZ\nYnagjKlYMLMLzOxLM1tpZjPMrMaVEMzsRDObGzl/jpkdWZvrqFhILv0PnXy658mne558uufJUVgI\nv/7qJwfEW9TFgpn1A0YBw4BOwBxgipk1r+b8rsDjwENAR+B54Hkz22dL11KxICIiUjudOkFubmK2\nq46lZWEI8IBzbrxzbh5wHrACGFjN+ZcALzvnbnfO/ds5NwwoAy7c0oVatYohnYiISBZq1Aj239+v\nfhxvURULZpYLFADTKo455xwwFehazdO6Rr5f2ZQazq90vWjSiYiIZLfCwsS0LEQ716A5kAMs2eT4\nEqBNNc9pVc35NbUbNAKYO3dulPGkLsrLyykrKwsdI6vonief7nny6Z4nT6tWsHDh/947G8XrdeM1\nMdGAaBaZ3NL5uwL079+/DpEkFgUFBaEjZB3d8+TTPU8+3fMgdgXejccLRVssLAPWAS03Od6CzVsP\nKnwX5fnguylOBRYAq6LMKCIiks0a4QuFKfF6QXNR7jphZjOAEufcJZGvDVgI3O2cG1nF+U8AWznn\n/ljp2DvAHOfc+XUJLyIiIokXSzfE7cAjZlYKzMTPjmgMjAMws/HA1865qyPn3wW8ZWZ/Bl4CivCD\nJM+pW3QRERFJhqiLBefcxMiaCtfhuxc+AI5wzi2NnLITsLbS+e+ZWRFwY+QxH/ijc+7TuoYXERGR\nxIu6G0JERESyS0rvDSEiIiLhqVgQERGRGgUpFpK1EZVsEM09N7OzzWy6mf0n8nhtS/9Gsrlof84r\nPe9kM1tvZs8mOmOmieF3S56Z3Wdm30SeM8/MeicrbyaI4Z7/KXKfV5jZQjO73cwaJitvujOz35nZ\nJDNbHPk90bcWzznUzErNbJWZfWZmp0d73aQXC8nciEq8aO85cAj+nh8KdAEWAa+a2Q6JT5sZYrjn\nFc9rDYwEpic8ZIaJ4XdLLn4p+l2A4/Cr0J4DLE5K4AwQwz0/Bbg5cn5b/J5C/fCD36V2muAnFlxA\nLRZDNLNdgRfx2zTsj5+hONbMekV1VedcUh/ADOCuSl8b8DVweTXnPwFM2uTYe8DoZGdP10e097yK\n59cDyoH+of8u6fKI5Z5H7vPbwJnAw8Czof8e6fSI4XfLefjZWTmhs6frI4Z7fg/w2ibH/gZMD/13\nSccHsB7ou4VzbgU+3ORYMfDPaK6V1JaFZG9EJTHf8001AXKB/8Q9YAaqwz0fBnzvnHs4sQkzT4z3\nvA+RDx5m9p2ZfWRmV5mZxnLVQoz3/F2goKKrwsx2B47Cr8EjidGFOLyHxmtviNpK1kZUskEs93xT\nt+KbZjf9gZOqRX3Pzaw7vkVh/8RGy1ix/JzvDvQAJgBHAnsBoyOvc0NiYmaUqO+5c6440kXxr8jq\nvznAGOfcrQlNmt2qew/dxswaOud+rc2LJLtYqE68N6KSLavVPTSzK4GTgEOcc6sTniqzVXnPzWxr\n4FHgHOfc8qSnymw1/ZzXw//SHBT5RDzbzHYEhqJioS6qvedmdihwNb4LaCawJ3C3mX3rnNM9Tx6L\n/Fnr99FkFwvJ2ohKNojlngNgZkOBy4GezrlPEhMvI0V7z/cAWgOTI5+2IDL42MxWA22cc18mKGum\niOXn/FtgdaRQqDAXaGVm9Z1za6t5nnix3PPrgPGVuto+iRTLD6ACLVGqew/9MZoPgEntm3POrQFK\ngZ4VxyK/HHtS/Taa71U+P6JX5LhsQYz3HDO7DLgGv5T37ETnzCQx3PO5QAf8bJ/9I49JwOuR/16U\n4MhpL8af83fwn2wrawN8q0Jhy2K8543xg/IqWx95qlVxvtRdVe+hhxPte2iA0ZsnASuBAfipMw8A\nPwDbR74/Hrip0vldgdXAn/H/Iw/Hb1u9T+iRqOnyiOGeXx65x8fiK9KKR5PQf5d0eUR7z6t4vmZD\nJPie4/exKcdPJdsLOBr/KezK0H+XdHnEcM+HAf/FT5fcFf/Bbz7weOi/S7o88APO98d/uFgP/Cny\n9c6R798MPFLp/F2Bn/Fjz9oA50feUw+L5rpJH7PgtBFV0kV7z4HB+NkPT2/yUiMiryFbEMM9lzqK\n4XfL12Z2OHAHfn2AxZH/vi2pwdNYDD/n1+Pf4K4HdgSW4lvR/pK00OnvAOAN/HgDh1/nAuAR/LoV\nrYCdK052zi0ws6PxO0ZfjJ/aepZzLqoB69pISkRERGqk+cQiIiJSIxULIiIiUiMVCyIiIlIjFQsi\nIiJSIxULIiIiUiMVCyIiIlIjFQsiIiJSIxULIiIiUiMVCyIiIlIjFQsiIiJSIxULIiIiUqP/B0sp\n+Iu8voYxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0724821ad0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# For the case of a binary random variable, let p \n",
    "# represent the probability of the value being 1. Then,\n",
    "# H(x) = -(P(0)logP(0) + P(1)log(P(1)))\n",
    "# or H(x) = -(plog(p) + (1-p)log(1-p))\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def entropy(p):\n",
    "    return -(p*np.log(p) + (1-p)*np.log(1-p))\n",
    "\n",
    "p_values = np.linspace(1e-7, 1 - 1e-7)\n",
    "entropy_values = entropy(p_values)\n",
    "plt.plot(p_values, entropy_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, given any data, we never know what is the true underlying distribution (lets call this $P(x)$). Suppose, we assume the underlying distribution to be $Q(x)$ and use a code to maximize the rate of transmission accordingly. We can measure how different the two distributions are, using the **Kullback-Leibler (KL) divergence**:\n",
    "$$ D_{KL}(P||Q) = \\mathbb{E}_{x\\sim P} [log\\frac{P(x)}{Q(x)}] = \\mathbb{E}_{x\\sim P} [log P(x) - log Q(x)]$$\n",
    "\n",
    "The KL divergence has some properties of a distance metric:\n",
    "- $D_{KL}(P||Q) \\geq 0$\n",
    "- $D_{KL}(P||Q) = 0$ only if $P$ and $Q$ are the same distribution in the case of discrete variables, or equal \"almost everywhere\" for continous variables.\n",
    "\n",
    "However, it isn't a true distance metric as it is not symmetric, i.e., $D_{KL}(P||Q) \\neq D_{KL}(Q||P)$. Thus, choosing which quantity to minimize is an important choice to make.\n",
    "\n",
    "Another similar quantity is the **cross-entropy**: $H(P, Q) = H(P) + D_{KL}(P||Q)$. <br> <br>\n",
    "$$ H(P, Q) = -\\mathbb{E}_{x\\sim P} [log{Q(x)}] $$ \n",
    "\n",
    "Since $Q$ doesn't participate in the omitted term, minimizing cross-entropy gives the same result as minimizing the KL divergence. Minimizing the KL divergence is not computationally feasible, and hence, we go for minimizing the cross-entropy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14. Structured Probabilistic Models\n",
    "\n",
    "Machine learning algorithms often involve probability distributions over a very large number of random variables. Often, these probability distributions involve direct interactions between relatively few variables. Suppose, we have 3 variables $a$, $b$ and $c$, such that $a$ can have 3 values, $b$ can have 2 and $c$ 4. Also, $a$ influences the value of $b$ which, in turn, influences the value of $c$. But given $b$, $a$ and $c$ are independent. Thus, if we wanted to use a single function to represent the joint distribution of $a$, $b$ and $c$:\n",
    "$$ P(a, b, c) = P(a)P(b | a)P(c | a,b)$$ \n",
    "\n",
    "which we get using the chain rule. Here, we are doing unnecessary computations because since $c$ is independent of $a$ given $b$, $P(c | a,b) = P(c | b)$. So, instead of computing at $3$ (# values of a) x $2$ (# values of b) cases, we just need to compute $2$ cases. Thus, it is better to factorize a probability distribution to reduce the number of parameters. Thus:\n",
    "$$ P(a, b, c) = P(a)P(b | a)P(c | b)$$ \n",
    "\n",
    "These kinds of factorizations can be described using **graphs**, where a variable represents a node of the graph, and vertices are connected to each other via edges. Such a representation of the factorization of a probability distribution is called a **Structured Probabilistic model** or **graphical model**. There are two kinds of graphical models:\n",
    "- Directed (directed edges)\n",
    "- Undirected (undirected edges)\n",
    "\n",
    "**Directed** models represent the factorization into probability distribution (like the one above), where there is a flow of influence between nodes. Directed models contain one factor for each of the variables and the factor consists of a conditional probability distribution over the factor, given its parents (denoted by $Pa_{\\mathcal{G}}$):\n",
    "$$ p(\\boldsymbol{x}) = \\prod_{i} p(x_i \\hspace{.1cm} | \\hspace{.1cm} Pa_{\\mathcal{G}}(x_i)) $$\n",
    "![directed-pgm](images/directed-pgm.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Undirected models** use graphs with undirected edges, and they represent factorizations into a set of functions, usually **not** probability distributions of any kind, i.e. there is no constraint that the factor must sum to 1, but must be non-negative. Any set of nodes that are all connected to each other in $\\mathcal{G}$ is called a clique. Each clique $C^{(i)}$ in an undirectedmodel is associated with a factor $\\phi^{(i)}(C^{(i)})$.\n",
    "\n",
    "The probability of a conﬁguration of random variables is proportional to the product of all these factors. To obtain the actual probabilities, we divide by a normalizing constant Z, deﬁned to be the sum or integral over all states of the product of the $\\phi$ functions.\n",
    "$$ p(\\boldsymbol{x}) = \\frac{1}{Z}\\prod_{i} \\phi^{(i)}(C^{(i)}) $$\n",
    "![undirected-pgm](images/undirected-pgm.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}