Navigation_SNH.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigation\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the first project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893).\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing some necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Banana.app\"`\n",
    "- **Windows** (x86): `\"path/to/Banana_Windows_x86/Banana.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Banana_Windows_x86_64/Banana.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Banana_Linux/Banana.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Banana_Linux/Banana.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Banana.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Banana.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 37\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\"Banana_Windows_x86_64/Banana.exe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "The simulation contains a single agent that navigates a large environment.  At each time step, it has four actions at its disposal:\n",
    "- `0` - walk forward \n",
    "- `1` - walk backward\n",
    "- `2` - turn left\n",
    "- `3` - turn right\n",
    "\n",
    "The state space has `37` dimensions and contains the agent's velocity, along with ray-based perception of objects around agent's forward direction.  A reward of `+1` is provided for collecting a yellow banana, and a reward of `-1` is provided for collecting a blue banana. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 1\n",
      "Number of actions: 4\n",
      "States look like: [1.         0.         0.         0.         0.84408134 0.\n",
      " 0.         1.         0.         0.0748472  0.         1.\n",
      " 0.         0.         0.25755    1.         0.         0.\n",
      " 0.         0.74177343 0.         1.         0.         0.\n",
      " 0.25854847 0.         0.         1.         0.         0.09355672\n",
      " 0.         1.         0.         0.         0.31969345 0.\n",
      " 0.        ]\n",
      "States have length: 37\n",
      "<unityagents.brain.BrainInfo object at 0x0000023D3DD4F9E8>\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents in the environment\n",
    "print('Number of agents:', len(env_info.agents))\n",
    "\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "state = env_info.vector_observations[0]\n",
    "print('States look like:', state)\n",
    "state_size = len(state)\n",
    "print('States have length:', state_size)\n",
    "\n",
    "print(env_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Define the Function to Perform the Reinforcement Learning\n",
    "A checkpoint is saved as checkpoint13.pth when the agent achieves a score of 13 averaged over 100 episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dqn(\n",
    "    agent,\n",
    "    n_episodes=2000,\n",
    "    max_t=100000,\n",
    "    epsilon_initial = 1.0,\n",
    "    epsilon_final = 0.01,\n",
    "    epsilon_rate = 0.005,\n",
    "    gamma_initial = 0.9,\n",
    "    gamma_final = 0.99,\n",
    "    gamma_rate = 0.002,\n",
    "    beta_initial = 0.4,\n",
    "    beta_rate = 0.002,\n",
    "    tau_initial = 0.001,\n",
    "    tau_final = 0.1,\n",
    "    tau_rate = 0.00001\n",
    "    ):\n",
    "    \"\"\"\n",
    "    Deep Q-Learning.\n",
    "    n_episodes (int): Maximum number of training episodes\n",
    "    max_t (int): Maximum number of timesteps per episode\n",
    "    epsilon_initial (float): Initial value of epsilon for epsilon-greedy selection of an action\n",
    "    epsilon_final (float): Final value of epsilon\n",
    "    epsilon_rate (float): A rate (0.0 to 1.0) for decreasing epsilon for each episode. Higher is faster decay.\n",
    "    gamma_initial (float): Initial gamma discount factor (0 to 1). Higher values favor long term over current rewards.\n",
    "    gamma_final (float): Final gamma discount factor (0 to 1).\n",
    "    gammma_rate (float): A rate (0 to 1) for increasing gamma.\n",
    "    beta_initial (float): For prioritized replay. Corrects bias induced by weighted sampling of stored experiences.\n",
    "        The beta parameters have no effect if the agent has prioritized experience replay activated.\n",
    "    beta_rate (float): Rate (0 to 1) for increasing beta to 1 as per Schauel et al. https://arxiv.org/abs/1511.05952\n",
    "    tau_initial (float): Initial value for tau, the weighting factor for soft updating the neural network.\n",
    "        The tau parameters have no effect if the agent uses fixed Q targets instead of soft updating.\n",
    "    tau_final (float): Final value of tau.\n",
    "    tau_rate (float): Rate (0 to 1) for increasing tau each episode.\n",
    "    \n",
    "    \"\"\"\n",
    "    # List of scores for an episode\n",
    "    scores = []\n",
    "    \n",
    "    # Most recent 100 scores\n",
    "    scores_window = deque(maxlen=100)\n",
    "    \n",
    "    epsilon = epsilon_initial\n",
    "    epsilon_scale = 1.0 - epsilon_rate\n",
    " \n",
    "    # Set final gamma to the value suggested for prioritized replay in\n",
    "    # Schaul et al., Prioritized Replay, ICLR 2016, https://arxiv.org/abs/1511.05952\n",
    "    #gamma_final = 1.0 - 1.0 / state_size\n",
    "    gamma = gamma_initial\n",
    "    gamma_scale = 1.0 - gamma_rate\n",
    "    \n",
    "    beta = beta_initial\n",
    "    beta_scale = 1.0 - beta_rate\n",
    "    \n",
    "    tau = tau_initial\n",
    "    tau_scale = 1.0 - tau_rate\n",
    "    \n",
    "    count = 0\n",
    "    level1 = False\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        # Reset environment\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        \n",
    "        # Get next state\n",
    "        state = env_info.vector_observations[0]\n",
    "\n",
    "        t = 0\n",
    "        score = 0\n",
    "        while True:\n",
    "            count += 1\n",
    "            t += 1\n",
    "            \n",
    "            # Get action\n",
    "            action = agent.act(state, epsilon)\n",
    "            \n",
    "            # Send action to environment\n",
    "            env_info = env.step(action.astype(np.int32))[brain_name]\n",
    "            \n",
    "            # Get next state\n",
    "            next_state = env_info.vector_observations[0]\n",
    "                                                      \n",
    "            # Get reward\n",
    "            reward = env_info.rewards[0]\n",
    "            \n",
    "            # Check is episode finished\n",
    "            done = env_info.local_done[0]\n",
    "                                                                                     \n",
    "            agent.step(state, action, reward, next_state, done, gamma, beta, tau)\n",
    "            score += reward\n",
    "            state = next_state\n",
    "            \n",
    "            # Exit if episode finished\n",
    "            if done or t > max_t:\n",
    "                break\n",
    "                \n",
    "        scores_window.append(score)\n",
    "        scores.append(score)\n",
    "                                                      \n",
    "        # Decrease epsilon. Limit to epsilon_final.       \n",
    "        #epsilon = max(epsilon_final, epsilon_scale * epsilon)\n",
    "        epsilon = epsilon_final + epsilon_scale * (epsilon - epsilon_final)\n",
    "\n",
    "        # Increase gamma discount factor. Limit to gamma_final.\n",
    "        gamma = gamma_final - gamma_scale * (gamma_final - gamma)\n",
    "        \n",
    "        # Increase beta for prioritized experience replay. Gradually increases to 1.\n",
    "        if agent.prioritized_replay:\n",
    "            beta = 1.0 - beta_scale * (1.0 - beta)\n",
    "        \n",
    "        tau = tau_final - tau_scale * (tau_final - tau)\n",
    "        \n",
    "        print('\\rEpisode {}\\tAverage Score: {:.4f}\\tepsilon: {:.4f}\\tgamma: {:.4f}\\tbeta: {:.4f}\\ttau: {:.4f}'\n",
    "              .format(i_episode, np.mean(scores_window), epsilon, gamma, beta, tau), end=\"\")\n",
    "        if i_episode % scores_window.maxlen == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.4f}'.format(i_episode, np.mean(scores_window)))\n",
    "            if not level1 and np.mean(scores_window) > 13:\n",
    "                level1 = True\n",
    "                print('\\rPassing score achieved in {:d} episodes!\\tAverage Score: {:.4f}'.format(i_episode, np.mean(scores_window)))\n",
    "                torch.save(agent.qnetwork_local.state_dict(), 'checkpoint13.pth')\n",
    "            if level1 and np.mean(scores_window) > 80:\n",
    "                print('\\rAverage score > 80 achieved in {:d} episodes!\\tAverage Score: {:.4f}'.format(i_episode, np.mean(scores_window)))\n",
    "                torch.save(agent.qnetwork_local.state_dict(), 'checkpoint80.pth')\n",
    "                break\n",
    "\n",
    "    return scores\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Declare the Learning Agent\n",
    "Agent has the following parameters:\n",
    "\n",
    "    state_size: Number of possible states in the environment\n",
    "    action_size: Number of possible actions\n",
    "    seed: For random number generation\n",
    "    target_update_interval (int): Interval for updating the neural network weights for fixed Q targets.\n",
    "        Set to a negative number to use soft updating instead. Default -1.\n",
    "    batch_normalize (boolean): Default False. True if batch normalization is used in the neural network\n",
    "    error_clipping (boolean): Default False. Set to True to limit the time difference error to between -1 and 1\n",
    "    reward_clipping (boolean): Default False. Set to True to limit the reward to betwen -1 and 1\n",
    "    gradient_clipping (boolean): Default False. Set to True to limit the size of the gradients to between -1 and 1\n",
    "    double_dqn (boolean): Default True. Flag to use double Q learning\n",
    "    dueling_dqn (boolean): Default True. Flag to use dueling Q networks\n",
    "    prioritized_replay (boolean): Default False. Flag for prioritized experience replay\n",
    "    learning_rate (float): initial learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "from collections import deque\n",
    "from dqn_agent import Agent, load_and_run_agent\n",
    "\n",
    "agent = Agent(\n",
    "    state_size=state_size,\n",
    "    action_size=action_size,\n",
    "    seed=0,\n",
    "    batch_normalize = False,\n",
    "    error_clipping = False,\n",
    "    reward_clipping = False,\n",
    "    gradient_clipping = False,\n",
    "    double_dqn = True,\n",
    "    prioritized_replay = True,\n",
    "    dueling_dqn = True,\n",
    "    learning_rate = 0.0005\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Train the Learning Agent\n",
    "The resulting scores are stored and plotted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: 1.0900\tepsilon: 0.6097\tgamma: 0.9571\tbeta: 0.6365\ttau: 0.0011\n",
      "Episode 200\tAverage Score: 4.4400\tepsilon: 0.3733\tgamma: 0.9779\tbeta: 0.7798\ttau: 0.0012\n",
      "Episode 300\tAverage Score: 7.1700\tepsilon: 0.2301\tgamma: 0.9856\tbeta: 0.8666\ttau: 0.0013\n",
      "Episode 400\tAverage Score: 9.4200\tepsilon: 0.1433\tgamma: 0.9884\tbeta: 0.9192\ttau: 0.0014\n",
      "Episode 500\tAverage Score: 12.3500\tepsilon: 0.0908\tgamma: 0.9894\tbeta: 0.9511\ttau: 0.0015\n",
      "Episode 600\tAverage Score: 13.7800\tepsilon: 0.0589\tgamma: 0.9898\tbeta: 0.9704\ttau: 0.0016\n",
      "Passing score achieved in 600 episodes!\tAverage Score: 13.7800\n",
      "Episode 700\tAverage Score: 14.4600\tepsilon: 0.0396\tgamma: 0.9899\tbeta: 0.9820\ttau: 0.0017\n",
      "Episode 800\tAverage Score: 15.0600\tepsilon: 0.0280\tgamma: 0.9900\tbeta: 0.9891\ttau: 0.0018\n",
      "Episode 900\tAverage Score: 14.6200\tepsilon: 0.0209\tgamma: 0.9900\tbeta: 0.9934\ttau: 0.0019\n",
      "Episode 1000\tAverage Score: 15.6900\tepsilon: 0.0166\tgamma: 0.9900\tbeta: 0.9960\ttau: 0.0020\n",
      "Episode 1100\tAverage Score: 15.1800\tepsilon: 0.0140\tgamma: 0.9900\tbeta: 0.9976\ttau: 0.0021\n",
      "Episode 1200\tAverage Score: 14.8400\tepsilon: 0.0124\tgamma: 0.9900\tbeta: 0.9985\ttau: 0.0022\n",
      "Episode 1300\tAverage Score: 15.5200\tepsilon: 0.0115\tgamma: 0.9900\tbeta: 0.9991\ttau: 0.0023\n",
      "Episode 1400\tAverage Score: 16.4800\tepsilon: 0.0109\tgamma: 0.9900\tbeta: 0.9995\ttau: 0.0024\n",
      "Episode 1500\tAverage Score: 15.8600\tepsilon: 0.0105\tgamma: 0.9900\tbeta: 0.9997\ttau: 0.0025\n",
      "Episode 1600\tAverage Score: 16.5800\tepsilon: 0.0103\tgamma: 0.9900\tbeta: 0.9998\ttau: 0.0026\n",
      "Episode 1700\tAverage Score: 16.8500\tepsilon: 0.0102\tgamma: 0.9900\tbeta: 0.9999\ttau: 0.0027\n",
      "Episode 1800\tAverage Score: 14.5900\tepsilon: 0.0101\tgamma: 0.9900\tbeta: 0.9999\ttau: 0.0028\n",
      "Episode 1900\tAverage Score: 16.0500\tepsilon: 0.0101\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0029\n",
      "Episode 2000\tAverage Score: 15.7300\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0030\n",
      "Episode 2100\tAverage Score: 15.7900\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0031\n",
      "Episode 2200\tAverage Score: 16.8400\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0032\n",
      "Episode 2300\tAverage Score: 16.3400\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0033\n",
      "Episode 2400\tAverage Score: 15.8700\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0033\n",
      "Episode 2500\tAverage Score: 16.5300\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0034\n",
      "Episode 2600\tAverage Score: 16.9700\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0035\n",
      "Episode 2700\tAverage Score: 16.8300\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0036\n",
      "Episode 2800\tAverage Score: 16.6700\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0037\n",
      "Episode 2900\tAverage Score: 16.0100\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0038\n",
      "Episode 3000\tAverage Score: 15.8500\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0039\n",
      "Episode 3100\tAverage Score: 16.0700\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0040\n",
      "Episode 3200\tAverage Score: 16.2800\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0041\n",
      "Episode 3300\tAverage Score: 15.9900\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0042\n",
      "Episode 3400\tAverage Score: 16.8100\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0043\n",
      "Episode 3500\tAverage Score: 18.0000\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0044\n",
      "Episode 3600\tAverage Score: 17.9000\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0045\n",
      "Episode 3700\tAverage Score: 17.8500\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0046\n",
      "Episode 3800\tAverage Score: 17.5700\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0047\n",
      "Episode 3900\tAverage Score: 17.3700\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0048\n",
      "Episode 4000\tAverage Score: 17.4900\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0049\n",
      "Episode 4100\tAverage Score: 16.9800\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0050\n",
      "Episode 4200\tAverage Score: 17.1100\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0051\n",
      "Episode 4300\tAverage Score: 17.1900\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0052\n",
      "Episode 4400\tAverage Score: 18.0800\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0053\n",
      "Episode 4500\tAverage Score: 17.7000\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0054\n",
      "Episode 4600\tAverage Score: 18.1400\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0055\n",
      "Episode 4700\tAverage Score: 17.3100\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0055\n",
      "Episode 4800\tAverage Score: 18.1700\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0056\n",
      "Episode 4900\tAverage Score: 17.9500\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0057\n",
      "Episode 5000\tAverage Score: 17.9100\tepsilon: 0.0100\tgamma: 0.9900\tbeta: 1.0000\ttau: 0.0058\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJztnXl4FdX5x79vErIAgbCEsBOWsIOAkWJZRGQVq9QdFbVaqXVtrbVxwbX9FVvFamu1tC64VLS1VitKpYiCGxKQfQcjO4QtLCFke39/zNybu8zcmXvvzJ25c9/P8+TJ3DNn5rznzJnznuWd9xAzQxAEQUhd0pwWQBAEQXAWUQSCIAgpjigCQRCEFEcUgSAIQoojikAQBCHFEUUgCIKQ4ogiEARBSHFEEQiCIKQ4oggEQRBSnAy7bkxE2QAWA8hS0/knMz9ERF0BzAXQEsAKANOYuTrSvVq3bs2FhYV2iSoIguBJli9ffpCZ843i2aYIAJwGMIaZTxBRIwCfEdGHAO4C8BQzzyWi5wHcCOC5SDcqLCxEaWmpjaIKgiB4DyL6zkw826aGWOGE+rOR+scAxgD4pxo+B8AUu2QQBEEQjLF1jYCI0oloJYADABYA2AbgKDPXqlF2AehgpwyCIAhCZGxVBMxcx8yDAHQEMBRAH61oWtcS0XQiKiWi0vLycjvFFARBSGkSYjXEzEcBfAJgGIA8IvKtTXQEsEfnmtnMXMzMxfn5hmsdgiAIQozYpgiIKJ+I8tTjHABjAWwAsAjApWq06wC8a5cMgiAIgjF2Wg21AzCHiNKhKJy3mPl9IloPYC4R/RrANwBesFEGQRAEwQDbFAEzrwYwWCN8O5T1AkEQBMEFyJfFgudZtOkAdh895bQYguBaRBEInudHLy3DxD8sdloMQXAtogiElOB4Va1xJEFIUUQRCIIgpDiiCARBEFIcUQSCIAgpjigCQRBShiMnq7HjUGVM167dXYHaunqLJXIHoggEQUgZRv5uEUb9flHU123cdwwX/PEzzFqw2QapnEcUgSAIKcOJ07FZj+0/dhoAsGZ3hZXiuAZRBIIgCCmOKAJBEAQDyGkBbEYUgSAIQoojikBIGKeq68CsuQ+REEBVTR3q643Lqb6eUVVTF1MaNXX1OF0b27VmqItDNjdjZfV1UxmJIhASQvnx0+jz4Hz8ZfF2p0VxNcyM3jPm4/5/rzWM+38fbEDvGfNjakwm/GExej0wPxYRTfHT15aj9wz77p9oyIa5ITeVkSgCISHsq6gCALy/WnNDOiGEN77eYRjnrdKdAIDTNdHbtm8vPxn1NdHw0fr9tt7fC7ipjEQRCAmBtbemFoSkwqv1WBSBICQ5Xm2chMQhikBICOR5AzxrkLV0wQlEEQgJQXqt9iFK1n58ZexVRS2KQDDN4ZPVKCyZh0WbDpi+5oxHPkJhyTz8bcm3NkpmjsGPfoQ/LtxiaxqXPvcF7pz7Dd5duRuFJfNQWDIP+49VhcUrP34ahSXz8NmWg3GnmWgl+4/SnSgsmWep+ek/l++K+56+Mv18a/xlGso1Lyz1H6/dXYHCknn49qD5Bfejlcq78/FG9ywQByKKQDDNWtXPygtRNOoVp2oAAO+tct5a6EhlDZ602WlY6XdH8O7KPUHOyUrLjoTFW7XzKADgpc9jV5Bkh02jCR6fvwkAUFFZY9k9f//fjQCAIydjv+c3O5RyfunzMitE0uVfK3YDABZuMN+or997DAAw26Xm06IIBNP4+p0OtT9JRXqMhZRUMw8pWA9kaihKiKgTES0iog1EtI6I7lTDHyai3US0Uv073y4ZBMEpAvVAJJ2QnO1KckptFTHpeJcXWYaN964F8AtmXkFEuQCWE9EC9dxTzPyEjWkLNiDuIcyTnubd7rKvGrh3kVrqabTYpgiYeS+AverxcSLaAKCDXekJ9tMwNeTWBsA9pJkso2QsSTunCONZ+HZ1vXSxaECC1giIqBDAYAC+pffbiGg1Eb1IRC0SIUMqsHT7Idz11krdnvsbX+/Anz6O32rGTJ3++tvD+MEfP4s7LS2+O3QSN7y8DKeq6/Dxxv0oLJmHX7y1Cs9/ui1Mhp+/uTLs+s37j+OmV0pRXVuP1776Dn/+ZKvptD/dXI57/7UmLDxS2UbTBoQ+uwf+vcbQSuu6F7/G4ZPVAID5a/fisffXa8b714pdmPXRJlNyBFo9FZbMw6eby8Pi/PzN4Lr2ypdlmL14W1g8PUreXo0lW4Lv++h/1uO/6/Zh6fZDKCyZhzFPfIJXvywzdb9oRqzPfbINp6rr0PXeef768/byXVgc8Hy3lZ/Aj+cs0/Xl5EvueFUNrn/pa2w9cALXv/Q1dh89pRFZ+ffV9sN4fP5GvLtyt//Uml0VuPX1FahTHQ3OWrAZ/yjdGVTP7cbOqSEAABE1BfA2gJ8x8zEieg7AY1CK5jEATwK4QeO66QCmA0Dnzp3tFtMTXP23paitZzx+yUA0Sg9vfnwV/LYxRbElEEVnbdoLS3G6NtwHjhXTCY+9vwEfbzyAJVvKMf3V5QCAt1fsAgDcfE53Qxl++c/VWLXzKNbuqcADqnO3W0b3MJX2dS9+DQD47cUDgsJDlcMercYgBl77agde+2oHymZO1o2zalcFXvmyDD8b2xM3v7YCADDjgr5h8e56a5Xyf3wvw3TvnBusQK978Wu/DL4Gd8mWgzhVU4fGmUoz8uC76wAA00d1hxnmLtuJuct2BuXtw7X78OHafchQp9a2HzyJGe+uw7SzC03dU8G4jj0+fyMKWzX2N+Zvr9jlr0OA8nxn/Hstvth2KMjqi8Fhd39v1R58sqkcn29djJo6xlMLNuOJy87QTfu5T4KV5a1/X4Edhytxz8Re6NKqCZ5RTZzH9y3AxxsP4NPN5ZjYv61hnuLB1hEBETWCogReZ+Z/AQAz72fmOmauB/BXAEO1rmXm2cxczMzF+fn5doopmMQ3bHfzCNwMPvHdsOThAhHiwg1lGI41QvnquVMfQyYyVTuthgjACwA2MPOsgPB2AdF+CMDY367gChoWCZMDvRcpEYoszcuLxU4LoEO0awRmowcqOzOKL1blqHddIuqrnVNDwwFMA7CGiHzjzPsATCWiQVDqUxmAn9goQ0piVy/NrwjiqJlW9q7iv5N9TVoi1EDgY0hkzzwRablB2fjdSuict3ukkMhnaqfV0GfQfh8+sCvNVMfunoPfWsTeZAyJN5+JmBoyUpZ6p93QABoRuCibDPLGiu8Z1ZusKA0dpfjSCwuP7XZRIV8Wu4AXP/sWe46ewt+X7sD28hNB545X1eCPC7f4LQoiEW3D9vLn32LXkcqgsNq6ejyzcAtOnq7Fih1HMPr3i1BTF7zgWs+MZxZuwYnTtUHhNXX1mLVgM343f6PmIi0AnKiqxR8XbjG1FWMoby3biS37j0d9XSi+RnpBwMYg/123z388f+1eLP/uMA4cq8Lsxdvw5bZDWLhhf5DlzM7Dlehx3weY+eFGTUuZwJdX6wX3PauPNx7Ac59sw2aNfP30teUB8RvK62dzv8Hlf/kSRwNcPBAhyLrl8fkbcdMrpVi3pwJHTlbj2UWRLaNe/bIMOw5VoqauHrf+fYVmHN8zN6pnG1R3CoEwM577ZBt2HanE0//bgrIQPz0hVSysvn+wZi+e/t8WVNXU4Z1vdvndnQTy1fZD/uP9x6rw18XbsXZ3BcbO+hQVp2rC/PyELojrUXm6oVwD012y5SAWby73u43wrymw8ixueqUUlzz3BbaXn8AmE/V29uLt2Bbw/v9PdWGxaV/8dd4I262GhMjsrTiFR99fj7dKd2LjvuNokpmOdY9O9J//zbwNmLtsJ3q0aYpJA9pFuFN0HDpxGg//Zz3mfPkdFt092h/+3irFT87hk9V4+YsyAIpp5LVnF/obo0WbyrFoUzkOnjiNRy/q77/2zWU7/RYPepQdqsSTCzajZ9tcTOgXnSXEPW+vBgCM61tg7gKdBsv3YgVum/mTV5f7rVd8ljdDC1vi67LDmve46m9fobaew0xWfQQ3/pH7dI/P34jH528Mswz6cG2Dctq8/wR6tc1FZXUt/r0y3G8TM4Jk8VmmLFi/H5P6tw26Vyinqusw4911KGi2FbePKcK81Xs1481asBkHT5zG8YAOgJbJ5qSnl4SFfbPzqD+fAPDU/4J9Ph08cVpXPgC45XXlmdSpnRAAYeX1wmcNfpt++tpyrNhx1P/7kffW4V/f7A6Kr9dZ8eHrMDy5oMHk9mR1nf/ZLtlyEEt0nAY+/+k2f0fjgj9+hkoTJqCvL92B15eG70y3/3i400KrkRGBw/h6PserlJfrZEiF8f2uDu0yaRDNkNTX4TpeFezky/dyBPYufVshhr7yJ08HyxrN3rlGL6Gd1NabS/tYlb4DtHgdrkU7feCrJ5EGUqd0yj905BaKb6772Klaw2dopkHTosai532qOnJefPjeJx8nTV4XiO8RnTQoPx+BOjHQ9j/WMkskoghSHDPTSXasPcQyNWQViVmEi9HpnIFskT6aivcbDTsXP626s51WO3qYLdegxXsH0o8HUQQO45Qdtu5iZQR5Qs/F03CYWfMwwp027ApWK08zZe3E9x0ufgRBxFJX3PK9TELMne1PQnAjsTWi1r32Zi0xtIj3vTCbciQRDa2CguJGd+9Y0ZPISh88oXcymw+r8mvmNprlHUNafuuykKv1ytOujkki9JEsFjvAsrLDaNUkE7uOnMLeCsUVgZZ/kq0HTvitZP5Rugvj+7ZFTma65j3f+WYXauq0a+KmfceRHqDyP1q3DwM6NteMG5XJXEiUxVHstlWlzhlvPXAc9Qz0LMj1nztWVYNVO49iZFE+9lVUYcfhShyprA67R6g1E6D4bdl99BQm9NNfUK6OMF/95bZDuueiIbCt+Gr7IbRqkomm2RkoapOL336wAS/GsCENM+Pp/2kvxn+6uRzZjbT7dbsOB1uG3f7GN/7jJVvKse2AYqlSVVOPj9ZF3mwldPev/ceqsK38RMRy27z/OLaGWMMFctnzX0RMM5Dgj7tYv1E2fUdt6uoZizaF+1cC9DeXqVOFW7u7ImJ+tdKKRCKc6YkicIDLnv/SVLyxsz71H3+29SDuf2cNZl0xKCzevooq/PzNVbr3mfCHxUG/p7+6HJcXd9SM63vR3irdpXtOi30VVVis4ZhMj/+btwHThnXB2FmKbIEWILe+vgJLthxE6QNjMebJT8IW25Z+q1jy/E7d1SqQH/xJcXT3zNTBphbYA2FmTP3rV1Fdo8f+Yw1WMC99XubfNev+8/vgb5/pK4FIU0Cfbi7XvXblzqOa4YDiryeQ/wTsFjftha+DzulZSfnYUxFswTL+qcU6Mc3HWaaxg5sZFqzfj/EalmfMSicqHgJ3jot21GPGVDQQTSd1CUamhpKIbw9p75Eaai1iZj5515FTalzzhMYN/K1nsaJHpPi+l7i6tl7T4sK3/eXOw/ov0AGNfYKNSMSaw96K2E0BfR5GBYVDUZRHtM92z1H7TTbNImsEginsrie+oWnkhWQLXUeYvFXEnb9iECeedQuzuGUBMlkJ7OQkyljA6WcmVkNCEFY6pYrmJdJbNLN7x7J4XsBYGvXwEY/1+YvnlXazlZT7ia7wAp+92U2G7EJGBIJtWKE83Nwu1cWiCFyQITfI4GZSsXzEasiDHDL4lB7Qt4bYcbgSB0+cRuumWRGv//bgSXTIy0FtHRv2ZpgZ3x06iS6tmuB0reLHJZSTp2tx4LhivRMavutIJTLS0sK+5IyWU9V1qDhVg6qaOn9vTMtnTbDssZ3Tw2fB5SNwwTcU3zpFtER6HMu/O6wr95pdFZZtdpNIQss0XgLL4OCJ06g4VYNjp2qC6t+X28MtmKJdkPatoQHxretYQSJGBJQMG5IXFxdzaWmp02JYQmHJPMM4v714AKYO7awbN9THyo5DlRj1+0VBYc1zGkVsrL7fvRW+CDD5e//2Ebhy9leG7gispGzmZH8ezypsEbP1iBa/GNcTTy7YbBwxwUwf1U3X/FCInsyMtIjmwF7gppFdcf/k8B3nzEBEy5m52CieTA25kFURTAG10OoxGPVYQ/X/zsOVCVUCoVipBADnF/j0cKlYSYvXlQCQmDUKUQSCkEhEEwguRBRBiuLWHrNVJOJrzFhIhCmg4DHEakhIFC5tNwUh5ZHvCART1CbYpbMojdjx8J72gk0k++b1Kctv5q3HX5covkpaNsn0uwYo7tLC1PVzl+3E3GU7dc/3fOBD9G3XDHNuGIozHvkofoHRsCuXGawyNDNjQeU1/vyJ9o5mgqCH7FmcpPiUABDsH6b0O2ssY6pr67Fy51Hsj8GfjiAIQiiiCJIYmWUQBO+T1C4miKgTES0iog1EtI6I7lTDWxLRAiLaov43N18ihCFz9YLgfZJ9sbgWwC+YuQ+AYQBuJaK+AEoALGTmIgAL1d+CYCmiJAWvkNQjAmbey8wr1OPjADYA6ADgIgBz1GhzAEyxSwZBEIRkxzOLxURUCGAwgKUACph5L6AoCwBtEiFDrPzl02243OSOYgAS7KYh9iryhUVbMrqV383f5LQIgmANXtiqkoiaAngbwM+Y+ZjZLz6JaDqA6QDQuXNn+wQ04Lcfhm+HGImNBh4zrUSmPwTB+yT9iICIGkFRAq8z87/U4P1E1E493w7AAa1rmXk2Mxczc3F+fr6dYiYtogcEQbACO62GCMALADYw86yAU+8BuE49vg7Au3bJ4ATud+otCEIykexfFg8HMA3AGiJaqYbdB2AmgLeI6EYAOwBcZqMMnsatjtUEQbCORJiP2qYImPkz6M9enGdXuoIgCF4iqc1HBfs594lPnBZBEASbSfrFYkEQBCE+rhjayfY0RBFYTBJsAS0IQhKRm9XI9jREEQiCILgYWSNIQsSQRxCEZEMUgcXI1JAgCMmG7FBmAQs37Ed6GqHiVA1ys6VIBUGwjmT/oCxluHFOqf+4bbNsByURBMFrJPt+BCnJoZOnnRZBEAQhKkQRCIIguBixGhIEQUhx5MviJESshgRBsJJEOJcURWASZkZ1bT2WlR0OCvt44/6geLX1ogkEQUguRBGY5I2vd+K3H27AZc9/ifV7lF3IPlizDze8XGpwpSAIQuzI1JCL+O7wSWzadxwAcKSyGgCw60ilkyIJgpACyGKxS5F1AEEQEoWsEbgJFj9CgiB4E1EEgiAIKY4oApMwgEMnlLWB7QdPYG/FKRw8IV8RC4KQ/IivIZNs3n8cG9XF4gffXYcH313nsESCIAjWICMCk2wvP+m0CIIgCLYgikAQBCHFsU0RENGLRHSAiNYGhD1MRLuJaKX6d75d6QuCIAjmsHNE8DKAiRrhTzHzIPXvAxvTtxQxHRUEwavYpgiYeTGAw4YRBUEQBEdxYo3gNiJarU4dtXAg/Zj47pC4kxAEwZuYVgRENIKIfqQe5xNR1xjSew5AdwCDAOwF8GSE9KYTUSkRlZaXl8eQlCAIgmAGU4qAiB4C8CsA96pBjQC8Fm1izLyfmeuYuR7AXwEMjRB3NjMXM3Nxfn5+tEkJgiAIJjE7IvghgAsBnAQAZt4DIDfaxIioXcg91+rFFQRBEBKD2S+Lq5mZiYgBgIiaGF1ARG8AGA2gNRHtAvAQgNFENAiKx4YyAD+JRWhBEATBOswqgreI6C8A8ojoJgA3QJna0YWZp2oEvxClfIIgCILNmFIEzPwEEY0DcAxALwAPMvMCWyUTBEEQEoKhIiCidAD/ZeaxAKTxFwRB8BiGi8XMXAegkoiaJ0AeQRCSmK6tDZcPBZXZ0850WgQ/ZtcIqgCsIaIFUC2HAICZ77BFKkEQkhLxxGKeRGxBaRazimCe+icIgqCPe9o215PmorIyu1g8h4gyAfRUgzYxc419YgmCIHgbFw0IzCkCIhoNYA4U238C0ImIrlMdy3mOvRWnMHX2V6iurXdaFEEQBNsx+2XxkwDGM/M5zDwKwAQAT9knlrPc+cZKlB2qxJ6KKqdFEYTkgu29/cii1pbeb1CnPEvvl6yYVQSNmHmT7wczb4bib8iTnKqpc1oEQXAdMy7oaxgnUA9M6FeAds2zw+I8fskAzWv1wgO5amhnwzjR8O9bh+PiwR38v8tmTkbZzMmWpqEHuWhBxexicSkRvQDgVfX31QCW2yOSIAheINqGzqmG0eZBjD7u0QOmFcFPAdwK4A4o4i8G8Ge7hHIadq5qCIJn0FsMdVNPGACY5X03qwgyADzNzLMA/9fGWbZJ5TBSLwQhHDPNd2AcIp1r4tADbrK08RJm1wgWAsgJ+J0D4H/Wi+M8K3cexfq9x5wWQxCSksA+lNt6/m7DTaVjVhFkM/MJ3w/1uLE9IjnLlGc/lxEBgH7tmzktguAyzullvEFU4DTL1KGdccOI8I0MWzTO1L7YVMtoXfM5oV9B0O/CVolt0tpqLKSH0jTL7KRNfJhVBCeJaIjvBxEVAzhlj0iCG7hnYm+nRRCiIBFTJt3zm5qO+/w1QzCiqDXG9lEa2y6tGvstcrIbObFVejh/mVYMoGEUc+fYIt24Fw1q7z8e2LE5bju3R9B5I2ujwHMdWyiTK00yMyJeM6FfAdY+MkH3vJWYVTc/A/APItoDpdzaA7jCNqkEQYgKgoPWLwH4/OeEjqoDf7t1xB1pKstKPWs2/4mcWouomonoLCJqy8zLAPQG8CaAWgDzAXybAPkEQbARq0cSoRY4vvtbZYnn1GJxqPReW7Q2GqP9BUC1enw2gPsAPAvgCIDZNsolOIzH6rmgg93PWatXq29W6gyuHaEksECMpobSmfmwenwFgNnM/DaAt4lopb2iJY5T1XXYffQUTtfKF8U+vNbj8TpE5N4WDa4WzU80dd5rr4fRiCCdiHzK4jwAHwecS8xydgK4/Y0VGDvrU0x+5jOnRRGEmIi1YfrxyG6WyhHa3uc1UTzRTB7Yzh/WpaX25jVFBbmG97ejATajo0YWGVtMmWFgx/D9vbIyGprhKQGL0onsjBkpgjcAfEpE70KxEloCAETUA0CFzbIljC+2HXJaBMGAX0/pH9N16xJkdfH8Nfq7TV08pINm+Fs/OduStH8xrifSYmw17ju/T9zpD+jQHL10GvFm2Y2w6qHx+NWEBiu0zq0a47GL+oXF1buHEc2yG/qky+4faxj/kQv7Yf2j0dWLS8/sqHuui0mz03WPTMA/b/5+WPiqh8b7j5+47Iyo5LKKiL16Zv4NES0E0A7AR9ywEpQG4Ha7hUsUXhvmWYHbPgYK7DVFQ5OsDDTLzsCxqlqLJQomkr1306wMNMlMx8nq4KnHlk107OmjTTvb2cF585xGOHjidMTzoTTTCDOD1q5ezXIa+Z9vXmPj+2ZlpKFxZpxlFiBHTqN0U5c00akj2QHXZ6Q7Y1prWBrM/JVG2Gaj64joRQAXADjAzP3VsJZQLI8KoextcDkzH4lOZOtx05ZxgvW4dXraK9VOKx9GZe6md84pX0MuKgLTH5TFwssAJoaElQBYyMxFUNxWlNiYvmlc9DyEJCWWl9qqeud0/WXW/37AaqzIa6iIvt/RKCdL5DAoK9d8RxAP6u5lh0OCL4Ky0xnU/1PsSl+IDzf1VuImAR2+WIrLTb1iN5CsxZEMFlFGJHpCqoCZ9wKA+r9NgtPXJkkroJ1YNX+dKhi1BUO6tLAt7Y4tGjvqOt2qBjyNyPTCq5vo3yHcEsgMRuXWM8bF81hwh9MPDYhoOhGVElFpeXm5vWnZevfkpEcb835lBONeYSSronh46fqzMLZvgalphHduCbZYscqhGXPDO2RWIQVKO6JHa7x/+whkZqTh3VuHR77OIJvxvMuB135y92g8e9WQoPN/vbY4TI7RvfLxmx/GZtFmxG1jehhHsohEK4L9RNQOANT/B/QiMvNsZi5m5uL8fGtsePVISxNVEEiTTHNWEEIDkRoogr7FSCjR9ojP6trSdNzBne0blcRDk6x0f686T88zqYoV0zBhj0rjnoWtmwR9+wAA+bnhW7AM7JgXZPVjBrOL0+kJbJcSrQjeA3CdenwdgHcTnL4mogbC8VKZJGLSJFJ56aVvqSOzJJsaCrwm0abKus8jltGGFxYIYKMiIKI3AHwJoBcR7SKiGwHMBDCOiLYAGKf+dhxZtAtGyiMxeLGYfY26R9pHXeJRXm58v2z7EoWZp+qcOs+uNGPFfY9FsBKn96S1s365pe5a7WU0djmiL5F4ZI7lSqfroxae8RcUC4Ul85wWwZVkxvgVr1tpl5eDrQdOGEe0ieYG895W0DQrA0cqa6K6xswOWXYR2KOOpiHO0Vi/ahznmlarJsrcfxODr419awH5TbPQPEeJ2yw7ti+kAXeNDLz1xicJHfJyjCPp8ItxPcPCHvpBXzz8g77xiBREl1aNo6qkf756iHEkB/n7j7/nP546tHPM93n/9hFRxb/0zI54bEr/sN2sfFg5Nz6ub4FxpBBeuWFo1Necq7NdpX9EEENnV68c7jivKMgi556JvfD97q38v/u1b4aZFw/A0JAF879dW4wFPx9lOv37zu+D3/ywP0Zr5O0/t43A3OnDAAC92ubi95cOxKzLB+GaYV3w2JT++NHwwqD4L//oLP9xrO/Fh3eOxOsBdTYRiCJwgBkX6DfaFw/WdlDmY3hR67Cwq77XGdcPD98bFgCmDu0UnXAq0Qxfzx/QzjhSnMQzmG7TLBvDuimNxQ/OiF3WaO3Fn7jsDEwb1iUhI6xYlEobDSsYABjWrSUm9W+rea6gWeJGEXeN64nzeheo6WbhltE9gjooTbIycKWGYh/bt8CUJ1MfOZnpuPp7XTQ7PwM6Nsewbg3K57LiTmjeuBEy0tMwbViXMN9Ao3s1fBql914Y1eU+7ZpheI/w99xORBG4DDcNF91EvKXiwmlZSxaLE11d7EjPzD3Nb+8oxIIogiRD64Www/xOFJL3iWlhVadBjrYOBpmPRvoGw+D7DMEaRBG4DGl/kxR5bkFEswAcSYlEGgn4ncUlaeG7SWpRBC7DqHKIoogPtzcaGTF+TZqenph8aX2Fn0aEjCjTTwv+oswQK+u9u2uAM4gicAT9bk6go6mbz+lu6m52KIf0NAqzdrn1XGN5HtRZCL9mWMNRw52EAAAW7ElEQVSintbuVKHMnhabb573bhuOif20FzpDad1U26wzKyMNb9w0LLqEY7GWCXluJZN642/XnYWfjDK/faRPsd0zoZdunPzcLPz24gFR39PHL8b1xN3je/rTCXyW13+/EI9fMhDPXDkYN47oiv7tm2veI5Tz+rSJ2bHhSA2DCb/sGskO6pSHXwaUzxQDg4x4eeeW7/vTe/Sifnj1xmDrLK1RTsmk3nj7p9bsWBcLoghcxNNXDvIf3ziiK0om9Y4Q237untALn5eMAQC0a56NX04wlqd3O21rjV9PaWiIpp1daLj15Ph+bdEohl7uwI55eNak2Z6eDBsfm4izA8wU9SibORlnaOxBGys3n9MdXVs3wb0xbB8ZyUfPub3yozKbDZ3WGdCxOW4bU+RPJ/BZPnxhP7Rtno1OLRtjxgV9kZamfY9QGqWn4eELjTsEfpkCbnfLaH1nbIHrHj4LvGuGdcGtAZ2aaH0DRcvgzi386V17dqGp/Y5vPqc7zuxi3m+U1YgiSDK0F4vtTI9tScMdU1zxC+FCYyTTRMp9rM8nmqk3M3XLisViV1S1ANxR94MRRSAIVuDClztWQhtzu5VdqlmoudGUWRSBS9F7NVz/zriwkkdLLA2TU4vQVtrgJxqfXBE9t5qwGkpW3PQuiyJwAKtfzET0qFKt15YIvFym0dRxM8Vg9rsDITZEEbiMC85oh9ZNs3DV97QX9yK9YBP6Re9vJpQOeTn41cSGReE2udno1roJHpuiLOxdMDCyi4aBnfLQPsSZ2QiNz+XNNBSBccb2Mc5bXmPFAVgi9vPo064ZAODeSX3QIS8HRQXR7+hmp5i92zYs2gcWtZbfo8em9Mf3ImxwM0RjQ5vzB7TFnecVxSWj4B5EEVjITSOD/f1MHdopasufds1zUPrAWHTLN9+w+BqUv0wrjhjPDJ+XjAnyrZKZkYaP7x6NMarPlz9dFdkip2lWBr64t8HTeNnMyXgtTgdaGx+biBYmTA19vl20etpaeieenuU/blZM/c7u3gqfl4xBToglipbye/ySYBPONJMCtI3Sv0/ZzMn4oY6J5N0BZpS+5KcN64I3f6JtuvjsVUPQPCfcw+afrz4TP9dwgBh670hE89FZYKdArxPxSBRWSEIwogiSjmSfGfUmXp2uiNVXfzQjvnithrxa9olEFIGFaPVEra6jmuajERNx51sS7bxw3OmZDPMiVrqGNnGhafyKIMKDNrNY7NbFcD2c3rxHC1EESUb0Vch9lc4sVr7gWrdKlpKJVyFG2/Ak3KOpmTgmFouTbWTgJncnKbVD2Y5DlaisqcXeo1Woq3d6S3PBarxa2rHky9yIy9kSs/sNTBZF7wZSakQw6veLMPEPS/Cjl5fhx6+UWn7/UMsWvc094qmgmlMcCX6hr9axaArEyLooEn1VixwjQhdoA8s1NzsDQwsjf7JvZallhmxQcllxR8NrtBZhtYj0fM08+h8MbG8qncJWjQGYk90IM3V8SOc8AIqlXCA3jtDeZMkwzSRp+X35M/v8E0FKKQK7Gdq1JcpmTvb/HtUzP+hF7ZCXE3TejHO0spmTg3a4ckNd/80PBwTlQ4s/XTUkYpxIL+0Hd440lKFs5mTMuvwM3fNrHp6At24Ot4SxSmeG3iZwp6qymZNx0SBjx2Y5memG5RhIqLIxw00ju+Lc3m2MIwL45JfnqhZH8SsCM3TLb4qymZP9Fmk+ZlzQN6xcIlkN6U4VWSGkDUwf1R1lMydr7r/sFKIIEojPIVc8FTRZej1CauPWRljQxpE1AiIqA3AcQB2AWmaO3wA+iYinLY9mL2GF5H8lzS4ORptTp+fIzRKPmPF2HGK93ur+inxZbC9OLhafy8wHHUzfcRJTod05hEj2l9kJ+bWsf/QsT+K2SLHfetQyZJQcPzI1ZDOBL6RUWCEW4hoRWCdG0pHKeY8WpxQBA/iIiJYT0XStCEQ0nYhKiai0vLw8weJFT7PshsFVdqPYitXMVEUqVW7jvAaX12ANnzg+hndXXD5E665Bj4y0xL065/Q03tgEAM7olOc/jnfE4rO4KmzVJL4bJYBo8pqfm2WfIEmMU1NDw5l5DxG1AbCAiDYy8+LACMw8G8BsACguLnZ1+/fNjHFoFGDZs+z+saitUzfd0KikeiODaOb/e7Rpiq0HTuiev+O8IjyzcAsApfHbd6wq6PyTl52BooKmuPBPn5tOM15WPTQ+ti0dTUw4LLnnXHRskaN7/vYxPXDJmR3QsUXj6AVA+NaZgZZc0XLhGe3xwAWRdyFb/fB4cD1wsroWLRpn4rWvdhjed+5Nw3DidC2A+Eef157dBWN6t0GnlrGVV/RrWUb30z6ORGitWTFjXFzPzcs4ogiYeY/6/wARvQNgKIDFka9yL6EO0XKzte2DY303tJrBlhG2JgSAgmYNPZ9WTTOx71gVMtIIteqHdJkZacjLiW3P2FiJ1m46mk6tUYOVlkZhSiCa+zfNtu5VyW6Uhja5kUcmzdQ61LxxI1TX1uvGC+xo5GSmW2aSSEQxKQE3L8DHukdyKpBw9UhETYgo13cMYDyAtYmWw0n0rWBMTA1FrUxINz0nfZ6YckpmcN7FbY4tWO3j3w6sHgn4EKshe3FiRFAA4B210csA8Hdmnu+AHI4Rz9SQv/GO4WVw9fxaBOx68VOhQUklA4XQ9yeFsh43CVcEzLwdgP4noSlETA1R1HqATbn7TTRuaIRTqZFMFG6cGnKhSK4jZVZOCkvmOZJuQYCVSk91FyvfXGXnlsEWGWZeoubqLly+jWsCd6IKupdGs290+z4mffy4hVYJnPO1wi+Mz2Il2gVr345rfduHPx+9R+qrd+3zrLGSMksXdV2hVZPEW+fovT+i8I1JKe+jTnDBwHZonJmO7EbpGNixOQBlV6uXfnQWRvRojUn92+LX89ZjWdkRU/fr17455twwFN/r2hIXD+mAHjo7mZmZ/w9VFn+PYyex928fYWoXsVB6FeRi0/7jMaVZbOBUzgizPcU5NwxFv/bN40oLAM7t1QYvXl+MUUXB5qAf3DESuREWozPS0/DWT85Gz4KmGPToAlNp+erdub3M+RmyijvHFuHMLi0woih8h7ZEIwMB84gisBkiwnka++36XtAzOuWhTZS27T678rNMNYT6H7SFKotYGnIf/TvE1lAO6dLCUBE4/UKbteM3gojCHKwB2j39UIZG2FNYLy2temc3jdLTTDu5E9xDykwNuRm7Gzpfz1dGyMHI3LG3kMXi2BFF4AJSscLKvK01uHFxNtEYlYAUkTGiCJIAOypyMrTDdtmkC4IQjOcVATNj4h/c/dGyvR2WBvPRUMuX9DTnukrRKDfp9aYuPpdOgV/BN1I36PH59zLqLkh/whjPLxYzAxv3xWaVEonpo7phWLeWqKmLv5aZvcMDkyP7pzFi1uVn4PqXlgFQTBI7tmiMh3/QFw//Z31c97UbZWTQoAxeuC6+7StevXEo0ohwurbOH3bHmB545uOtcd1XsJ42udl49KJ+QdvADumch/vP74NLzgzeSS20wyDdB/N4fkRQb1N34McjumJM7wJMMLHdpFmMKu6VQ433Cta6q+/9aNUky789ZpoaeP3w2PaHTQR6I4F4rWFGFuVjeI/WfvPZ0b3ycdf4XnHd0ylSobG79uxCtM9rcChIRLhpVLcw30EylRg7KaAIbLpxks7b+xRjWhJMt8iLLZhBryZL7TGP5xWBk47V3IhPMTq4PBA1dq8RiL7xNknQ53Ec7yuCJHrJE1Ff3TIicMVzkQZCEAB4fLF49a6j+GDNPqfFsIzY2q3gFtenCJy0GBKsQ3q7ghV4WhEkcveteLj//D6or+cgywgreeryQfjjx1vRp10uHruoP2bO34jv92jlP//T0d1xZoRtHvW4e3xPdI5xK0MzDdg/bj4bb6/Y7Z/G+uWEXmG7kN09vie6xCjD97u3wuSB7XDPBPMLxU9fOQg7D1fGlJ5gD7eNKcLeiqowKyIfrhh9uhxPK4JkoX1eDp675kzd82a2atSHUFSQi2emDgag7OT17FVDgmL8amLvmO5825iiOOQy5swuLXFmlwYfO7ee28NSGbIy0sPKwoiLBnWIOT3BHvJzszD72vhMilMdz68R2EV8jXN0yIK3IMSOTJ8ZI4ogRpxonKVCC6HIV9eCFYgiSAISOfoQBCH1EEUQI9I4C4LgFUQRGDCpv+KS4YnLlG2WH5jcB5kZaZZsXRgtZqwfxvUtwPkD2mK0uvHN1KGdbJYqNtwoX8mk3ihoFvsWi1MGtbdsExs97hrXE51a5uCxKf2RmxWbrceMC/qGuWfwMmI1ZIxYDQEomznZv6fx//1wAO57Zw2uPKsTZl4yMCjepap52o9Hdku4jGb5a4D1RNnMyQ5KEpkOeTmuk+/mc7rj5nO6x3z9H64cbKE02txxXhHuOE+xlJo2rEtM97hxRFfcOMK9PqaExCMjghB8Nutu7EXIuqAgRI+8N8Y4ogiIaCIRbSKirURU4oQMejRs6+g+TeBG5SQIQvKTcEVAROkAngUwCUBfAFOJqG+i5TDCTY2u9GgEQbATJ0YEQwFsZebtzFwNYC6AixyQQxOfNZCL9ICrlJIgCN7DCUXQAcDOgN+71LAgiGg6EZUSUWl5eXlCBBvXtwADOzUHAJzXu01C0hSAy4sbfMT0aNPUQUmCycpIQ5vc2K2IBGfx+e7q1765w5K4H0r05h9EdBmACcz8Y/X3NABDmfl2vWuKi4u5tLQ06rR8lkCR2PzrScjMSEN1bT0y0ghpacoWhlkZ6VGnZxd9H5yPyuo6rHtkAprEaDLoZurrGbX1DCLFPbZbPKPW1tUDADLSxaYiWXHbu5xoiGg5Mxs6YnKiVdkFINB4vCOAPQ7IAQDIzEgL+g8gpSuOE6SlETJd0vgHIgog+ZF32RxO1PRlAIqIqCsRZQK4EsB7DsghCIIgwIERATPXEtFtAP4LIB3Ai8y8LtFyCIIgCAqOTDgz8wcAPnAibUEQBCEYT0+C5jVOvD8gOxErUkEQ7MB7JigB9G/fHJ9tPah5rvSBsaitS46m1X3LqIIgeAlPK4L6CKaxrZsmj314cqgrQRCSFU9PDckXuYIgCMZ4WhFEGhEkEzI1JAiCnXhaEXhDDQiCINiLtxWBR0YEha2bAGjYK0EQBMFKPL5YHPz7o5+PwvGqGke2mYyHOTcMxcodR9E409OPSxAEh/B0yxI6IuhZkOuQJPHRumkWxvYtcFoMQRA8iqenhkJHBIIgCEI4nlYEogcEQRCM8bYi8MhisSAIgp14WhFkiy9yQRAEQzy9WPz01EF44+ud6Ne+GdJkB3hBEARNPK0I2jXPwV3jejothiAIgqvx9NSQIAiCYIwoAkEQhBRHFIEgCEKKI4pAEAQhxRFFIAiCkOKIIhAEQUhxRBEIgiCkOKIIBEEQUhxKBn88RFQO4LsYL28N4KCF4iQDkufUQPKcGsST5y7MnG8UKSkUQTwQUSkzFzstRyKRPKcGkufUIBF5lqkhQRCEFEcUgSAIQoqTCopgttMCOIDkOTWQPKcGtufZ82sEgiAIQmRSYUQgCIIgRMDTioCIJhLRJiLaSkQlTssTD0T0IhEdIKK1AWEtiWgBEW1R/7dQw4mInlHzvZqIhgRcc50afwsRXedEXsxARJ2IaBERbSCidUR0pxru5TxnE9HXRLRKzfMjanhXIlqqyv8mEWWq4Vnq763q+cKAe92rhm8iognO5Mg8RJRORN8Q0fvqb0/nmYjKiGgNEa0kolI1zLm6zcye/AOQDmAbgG4AMgGsAtDXabniyM8oAEMArA0I+x2AEvW4BMDj6vH5AD4EQACGAViqhrcEsF3930I9buF03nTy2w7AEPU4F8BmAH09nmcC0FQ9bgRgqZqXtwBcqYY/D+Cn6vEtAJ5Xj68E8KZ63Fet71kAuqrvQbrT+TPI+10A/g7gffW3p/MMoAxA65Awx+q2l0cEQwFsZebtzFwNYC6AixyWKWaYeTGAwyHBFwGYox7PATAlIPwVVvgKQB4RtQMwAcACZj7MzEcALAAw0X7po4eZ9zLzCvX4OIANADrA23lmZj6h/myk/jGAMQD+qYaH5tlXFv8EcB4RkRo+l5lPM/O3ALZCeR9cCRF1BDAZwN/U3wSP51kHx+q2lxVBBwA7A37vUsO8RAEz7wWUhhNAGzVcL+9JWSbq8H8wlB6yp/OsTpGsBHAAyou9DcBRZq5VowTK78+ber4CQCskWZ4B/AHAPQDq1d+t4P08M4CPiGg5EU1Xwxyr217es1hrt/pUMZHSy3vSlQkRNQXwNoCfMfMxpfOnHVUjLOnyzMx1AAYRUR6AdwD00Yqm/k/6PBPRBQAOMPNyIhrtC9aI6pk8qwxn5j1E1AbAAiLaGCGu7Xn28ohgF4BOAb87AtjjkCx2sV8dIkL9f0AN18t7UpUJETWCogReZ+Z/qcGezrMPZj4K4BMoc8J5ROTrtAXK78+ber45lOnDZMrzcAAXElEZlOnbMVBGCF7OM5h5j/r/ABSFPxQO1m0vK4JlAIpU64NMKAtL7zksk9W8B8BnKXAdgHcDwq9VrQ2GAahQh5r/BTCeiFqoFgnj1TDXoc77vgBgAzPPCjjl5TznqyMBEFEOgLFQ1kYWAbhUjRaaZ19ZXArgY1ZWEd8DcKVqYdMVQBGArxOTi+hg5nuZuSMzF0J5Rz9m5qvh4TwTURMiyvUdQ6mTa+Fk3XZ69dzOPyir7ZuhzLPe77Q8ceblDQB7AdRA6QncCGVudCGALer/lmpcAvCsmu81AIoD7nMDlIW0rQB+5HS+IuR3BJRh7moAK9W/8z2e54EAvlHzvBbAg2p4NyiN2lYA/wCQpYZnq7+3que7BdzrfrUsNgGY5HTeTOZ/NBqshjybZzVvq9S/db62ycm6LV8WC4IgpDhenhoSBEEQTCCKQBAEIcURRSAIgpDiiCIQBEFIcUQRCIIgpDiiCARPQ0R1qodH319EL7REdDMRXWtBumVE1DqG6yYQ0cOqbfgH8cohCGbwsosJQQCAU8w8yGxkZn7eTmFMMBLKx1SjAHzusCxCiiCKQEhJVJcGbwI4Vw26ipm3EtHDAE4w8xNEdAeAmwHUAljPzFcSUUsAL0L5KKgSwHRmXk1EraB89JcP5UMnCkjrGgB3QHGHvhTALaz4FAqU5woA96r3vQhAAYBjRPQ9Zr7QjjIQBB8yNSR4nZyQqaErAs4dY+ahAP4Exb9NKCUABjPzQCgKAQAeAfCNGnYfgFfU8IcAfMbMg6G4BOgMAETUB8AVUJyMDQJQB+Dq0ISY+U007DcxAMqXxYNFCQiJQEYEgteJNDX0RsD/pzTOrwbwOhH9G8C/1bARAC4BAGb+mIhaEVFzKFM5F6vh84joiBr/PABnAlimek7NQYMzsVCKoLgRAIDGrOzDIAi2I4pASGVY59jHZCgN/IUAZhBRP0R2/at1DwIwh5nvjSSIul1hawAZRLQeQDt1X4LbmXlJ5GwIQnzI1JCQylwR8P/LwBNElAagEzMvgrJpSh6ApgAWQ53aUf3nH2TmYyHhk6BsHQgozsMuVf3O+/al7RIqCDMXA5gHZX3gd1AckQ0SJSAkAhkRCF4nR+1Z+5jPzD4T0iwiWgqlQzQ15Lp0AK+p0z4E4ClmPqouJr9ERKuhLBb73AY/AuANIloB4FMAOwCAmdcT0QNQdqNKg+I99lYA32nIOgTKovItAGZpnBcEWxDvo0JKoloNFTPzQadlEQSnkakhQRCEFEdGBIIgCCmOjAgEQRBSHFEEgiAIKY4oAkEQhBRHFIEgCEKKI4pAEAQhxRFFIAiCkOL8P+pu6K3//qg+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores = dqn(\n",
    "    agent,\n",
    "    n_episodes = 5000,\n",
    "    epsilon_initial = 1.0,\n",
    "    epsilon_final = 0.01,\n",
    "    epsilon_rate = 0.005,\n",
    "    gamma_initial = 0.90,\n",
    "    gamma_final = 0.99,\n",
    "    gamma_rate = 0.01,\n",
    "    beta_initial = 0.4,\n",
    "    beta_rate = 0.005,\n",
    "    tau_initial = 0.001,\n",
    "    tau_final = 0.1,\n",
    "    tau_rate = 0.00001\n",
    ")\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Save the Trained Agent\n",
    "Another checkpoint is saved after all the training has been completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    " torch.save(agent.qnetwork_local.state_dict(), 'checkpoint13_buffer1e6_every4_tau.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Run the Trained Agent\n",
    "The trained agent is run for 3 episodes and the score for each episode is printed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 18.0\n",
      "Score: 20.0\n",
      "Score: 20.0\n",
      "Score: 14.0\n",
      "Score: 17.0\n",
      "Score: 18.0\n",
      "Score: 18.0\n",
      "Score: 18.0\n",
      "Score: 17.0\n",
      "Score: 18.0\n",
      "Score: 20.0\n",
      "Score: 17.0\n",
      "Score: 18.0\n",
      "Score: 17.0\n",
      "Score: 14.0\n",
      "Score: 19.0\n",
      "Score: 19.0\n",
      "Score: 17.0\n",
      "Score: 20.0\n",
      "Score: 20.0\n",
      "Score: 20.0\n",
      "Score: 20.0\n",
      "Score: 17.0\n",
      "Score: 17.0\n",
      "Score: 18.0\n",
      "Score: 18.0\n",
      "Score: 20.0\n",
      "Score: 12.0\n",
      "Score: 22.0\n",
      "Score: 17.0\n",
      "Score: 16.0\n",
      "Score: 15.0\n",
      "Score: 17.0\n",
      "Score: 19.0\n",
      "Score: 23.0\n",
      "Score: 18.0\n",
      "Score: 19.0\n",
      "Score: 23.0\n",
      "Score: 15.0\n",
      "Score: 17.0\n",
      "Score: 18.0\n",
      "Score: 17.0\n",
      "Score: 19.0\n",
      "Score: 17.0\n",
      "Score: 21.0\n",
      "Score: 17.0\n",
      "Score: 13.0\n",
      "Score: 14.0\n",
      "Score: 24.0\n",
      "Score: 20.0\n",
      "Score: 16.0\n",
      "Score: 23.0\n",
      "Score: 21.0\n",
      "Score: 11.0\n",
      "Score: 20.0\n",
      "Score: 16.0\n",
      "Score: 21.0\n",
      "Score: 19.0\n",
      "Score: 14.0\n",
      "Score: 19.0\n",
      "Score: 19.0\n",
      "Score: 18.0\n",
      "Score: 18.0\n",
      "Score: 14.0\n",
      "Score: 18.0\n",
      "Score: 20.0\n",
      "Score: 21.0\n",
      "Score: 17.0\n",
      "Score: 21.0\n",
      "Score: 20.0\n",
      "Score: 19.0\n",
      "Score: 16.0\n",
      "Score: 21.0\n",
      "Score: 19.0\n",
      "Score: 20.0\n",
      "Score: 22.0\n",
      "Score: 20.0\n",
      "Score: 12.0\n",
      "Score: 20.0\n",
      "Score: 13.0\n",
      "Score: 14.0\n",
      "Score: 20.0\n",
      "Score: 7.0\n",
      "Score: 17.0\n",
      "Score: 20.0\n",
      "Score: 18.0\n",
      "Score: 14.0\n",
      "Score: 18.0\n",
      "Score: 17.0\n",
      "Score: 20.0\n",
      "Score: 17.0\n",
      "Score: 21.0\n",
      "Score: 17.0\n",
      "Score: 19.0\n",
      "Score: 18.0\n",
      "Score: 15.0\n",
      "Score: 14.0\n",
      "Score: 15.0\n",
      "Score: 27.0\n",
      "Score: 23.0\n",
      "Average Score: 18.01\n"
     ]
    }
   ],
   "source": [
    "load_and_run_agent(agent, env, 'checkpoint_final_buffer1e6_every4_tau', 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}