PPO_colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e7JowRQEGGKQ"
      },
      "source": [
        "################################################################################\n",
        "> # **Clone GitHub repository**\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IyGzuEMQF6sJ"
      },
      "outputs": [],
      "source": [
        "\n",
        "################# Clone repository from github to colab session ################\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "run this section if you want to clone all the preTrained networks, logs, graph figures, gifs \n",
        "from the GitHub repository to this colab session\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "!git clone https://github.com/nikhilbarhate99/PPO-PyTorch\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mrn6rpJpF8Sc"
      },
      "outputs": [],
      "source": [
        "\n",
        "\"\"\"\n",
        "\n",
        "run this section if you want to copy all files and folders from cloned folder (PPO-PyTorch)\n",
        "to current directory (/content/ or ./)\n",
        "\n",
        "So you can load preTrained networks and log files without changing any paths\n",
        "\n",
        "**  This will overwrite any saved networks, logs, graph figures, or gifs \n",
        "    that are created in this session before copying having the same name (or number)\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "!cp -rv ./PPO-PyTorch/* ./\n",
        "\n",
        "print(\"============================================================================================\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X-7AbGA2F8Ut",
        "outputId": "c6921fe3-fa71-42df-e3fa-c9af7eee6aaf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "============================================================================================\n",
            "============================================================================================\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\"\"\"\n",
        "\n",
        "run this section if you want to delete original cloned folder and the cloned ipynb file\n",
        "(after you have copied its contents to current directory)\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "# delete original cloned folder\n",
        "!rm -r ./PPO-PyTorch\n",
        "\n",
        "# delete cloned ipynb file\n",
        "!rm ./PPO_colab.ipynb\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z4VJcUT2GlJz"
      },
      "source": [
        "################################################################################\n",
        "> # **Install Dependencies**\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rbpSQTflGlAr"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "############ install compatible version of OpenAI roboschool and gym ###########\n",
        "\n",
        "!pip install swig\n",
        "\n",
        "# !pip install roboschool==1.0.7 gym==0.15.4\n",
        "\n",
        "# !pip install box2d-py\n",
        "\n",
        "# !pip install Box2D\n",
        "\n",
        "# !pip install pybullet\n",
        "\n",
        "# !pip install gym[box2d]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pzZairIiGQ11"
      },
      "source": [
        "################################################################################\n",
        "> # **Introduction**\n",
        "> The notebook is divided into 5 major parts : \n",
        "\n",
        "*   **Part I** : define actor-critic network and PPO algorithm\n",
        "*   **Part II** : train PPO algorithm and save network weights and log files\n",
        "*   **Part III** : load (preTrained) network weights and test PPO algorithm\n",
        "*   **Part IV** : load log files and plot graphs\n",
        "*   **Part V** : install xvbf, load (preTrained) network weights and save images for gif and then generate gif\n",
        "\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s37cJXAYGrTY"
      },
      "source": [
        "################################################################################\n",
        "> # **Part - I**\n",
        "\n",
        "*   define actor critic networks\n",
        "*   define PPO algorithm\n",
        "\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UT6VUBg-F8Zm",
        "outputId": "2369a1ae-0ba8-41ab-aaa0-f216627b86f6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "============================================================================================\n",
            "Device set to : cpu\n",
            "============================================================================================\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\n",
        "############################### Import libraries ###############################\n",
        "\n",
        "\n",
        "import os\n",
        "import glob\n",
        "import time\n",
        "from datetime import datetime\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.distributions import MultivariateNormal\n",
        "from torch.distributions import Categorical\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "import gym\n",
        "# import roboschool\n",
        "import pybullet_envs\n",
        "\n",
        "\n",
        "################################## set device ##################################\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "# set device to cpu or cuda\n",
        "device = torch.device('cpu')\n",
        "\n",
        "if(torch.cuda.is_available()): \n",
        "    device = torch.device('cuda:0') \n",
        "    torch.cuda.empty_cache()\n",
        "    print(\"Device set to : \" + str(torch.cuda.get_device_name(device)))\n",
        "else:\n",
        "    print(\"Device set to : cpu\")\n",
        "    \n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "################################## PPO Policy ##################################\n",
        "\n",
        "\n",
        "class RolloutBuffer:\n",
        "    def __init__(self):\n",
        "        self.actions = []\n",
        "        self.states = []\n",
        "        self.logprobs = []\n",
        "        self.rewards = []\n",
        "        self.state_values = []\n",
        "        self.is_terminals = []\n",
        "    \n",
        "\n",
        "    def clear(self):\n",
        "        del self.actions[:]\n",
        "        del self.states[:]\n",
        "        del self.logprobs[:]\n",
        "        del self.rewards[:]\n",
        "        del self.state_values[:]\n",
        "        del self.is_terminals[:]\n",
        "\n",
        "\n",
        "class ActorCritic(nn.Module):\n",
        "    def __init__(self, state_dim, action_dim, has_continuous_action_space, action_std_init):\n",
        "        super(ActorCritic, self).__init__()\n",
        "\n",
        "        self.has_continuous_action_space = has_continuous_action_space\n",
        "\n",
        "        if has_continuous_action_space:\n",
        "            self.action_dim = action_dim\n",
        "            self.action_var = torch.full((action_dim,), action_std_init * action_std_init).to(device)\n",
        "\n",
        "        # actor\n",
        "        if has_continuous_action_space :\n",
        "            self.actor = nn.Sequential(\n",
        "                            nn.Linear(state_dim, 64),\n",
        "                            nn.Tanh(),\n",
        "                            nn.Linear(64, 64),\n",
        "                            nn.Tanh(),\n",
        "                            nn.Linear(64, action_dim),\n",
        "                            nn.Tanh()\n",
        "                        )\n",
        "        else:\n",
        "            self.actor = nn.Sequential(\n",
        "                            nn.Linear(state_dim, 64),\n",
        "                            nn.Tanh(),\n",
        "                            nn.Linear(64, 64),\n",
        "                            nn.Tanh(),\n",
        "                            nn.Linear(64, action_dim),\n",
        "                            nn.Softmax(dim=-1)\n",
        "                        )\n",
        "\n",
        "        \n",
        "        # critic\n",
        "        self.critic = nn.Sequential(\n",
        "                        nn.Linear(state_dim, 64),\n",
        "                        nn.Tanh(),\n",
        "                        nn.Linear(64, 64),\n",
        "                        nn.Tanh(),\n",
        "                        nn.Linear(64, 1)\n",
        "                    )\n",
        "        \n",
        "    def set_action_std(self, new_action_std):\n",
        "\n",
        "        if self.has_continuous_action_space:\n",
        "            self.action_var = torch.full((self.action_dim,), new_action_std * new_action_std).to(device)\n",
        "        else:\n",
        "            print(\"--------------------------------------------------------------------------------------------\")\n",
        "            print(\"WARNING : Calling ActorCritic::set_action_std() on discrete action space policy\")\n",
        "            print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "\n",
        "    def forward(self):\n",
        "        raise NotImplementedError\n",
        "    \n",
        "\n",
        "    def act(self, state):\n",
        "\n",
        "        if self.has_continuous_action_space:\n",
        "            action_mean = self.actor(state)\n",
        "            cov_mat = torch.diag(self.action_var).unsqueeze(dim=0)\n",
        "            dist = MultivariateNormal(action_mean, cov_mat)\n",
        "        else:\n",
        "            action_probs = self.actor(state)\n",
        "            dist = Categorical(action_probs)\n",
        "\n",
        "        action = dist.sample()\n",
        "        action_logprob = dist.log_prob(action)\n",
        "        state_val = self.critic(state)\n",
        "\n",
        "        return action.detach(), action_logprob.detach(), state_val.detach()\n",
        "    \n",
        "\n",
        "    def evaluate(self, state, action):\n",
        "\n",
        "        if self.has_continuous_action_space:\n",
        "            action_mean = self.actor(state)\n",
        "            action_var = self.action_var.expand_as(action_mean)\n",
        "            cov_mat = torch.diag_embed(action_var).to(device)\n",
        "            dist = MultivariateNormal(action_mean, cov_mat)\n",
        "            \n",
        "            # for single action continuous environments\n",
        "            if self.action_dim == 1:\n",
        "                action = action.reshape(-1, self.action_dim)\n",
        "\n",
        "        else:\n",
        "            action_probs = self.actor(state)\n",
        "            dist = Categorical(action_probs)\n",
        "\n",
        "        action_logprobs = dist.log_prob(action)\n",
        "        dist_entropy = dist.entropy()\n",
        "        state_values = self.critic(state)\n",
        "        \n",
        "        return action_logprobs, state_values, dist_entropy\n",
        "\n",
        "\n",
        "class PPO:\n",
        "    def __init__(self, state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, has_continuous_action_space, action_std_init=0.6):\n",
        "\n",
        "        self.has_continuous_action_space = has_continuous_action_space\n",
        "\n",
        "        if has_continuous_action_space:\n",
        "            self.action_std = action_std_init\n",
        "\n",
        "        self.gamma = gamma\n",
        "        self.eps_clip = eps_clip\n",
        "        self.K_epochs = K_epochs\n",
        "        \n",
        "        self.buffer = RolloutBuffer()\n",
        "\n",
        "        self.policy = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)\n",
        "        self.optimizer = torch.optim.Adam([\n",
        "                        {'params': self.policy.actor.parameters(), 'lr': lr_actor},\n",
        "                        {'params': self.policy.critic.parameters(), 'lr': lr_critic}\n",
        "                    ])\n",
        "\n",
        "        self.policy_old = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)\n",
        "        self.policy_old.load_state_dict(self.policy.state_dict())\n",
        "        \n",
        "        self.MseLoss = nn.MSELoss()\n",
        "\n",
        "\n",
        "    def set_action_std(self, new_action_std):\n",
        "        \n",
        "        if self.has_continuous_action_space:\n",
        "            self.action_std = new_action_std\n",
        "            self.policy.set_action_std(new_action_std)\n",
        "            self.policy_old.set_action_std(new_action_std)\n",
        "        \n",
        "        else:\n",
        "            print(\"--------------------------------------------------------------------------------------------\")\n",
        "            print(\"WARNING : Calling PPO::set_action_std() on discrete action space policy\")\n",
        "            print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "\n",
        "    def decay_action_std(self, action_std_decay_rate, min_action_std):\n",
        "        print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "        if self.has_continuous_action_space:\n",
        "            self.action_std = self.action_std - action_std_decay_rate\n",
        "            self.action_std = round(self.action_std, 4)\n",
        "            if (self.action_std <= min_action_std):\n",
        "                self.action_std = min_action_std\n",
        "                print(\"setting actor output action_std to min_action_std : \", self.action_std)\n",
        "            else:\n",
        "                print(\"setting actor output action_std to : \", self.action_std)\n",
        "            self.set_action_std(self.action_std)\n",
        "\n",
        "        else:\n",
        "            print(\"WARNING : Calling PPO::decay_action_std() on discrete action space policy\")\n",
        "\n",
        "        print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "\n",
        "    def select_action(self, state):\n",
        "\n",
        "        if self.has_continuous_action_space:\n",
        "            with torch.no_grad():\n",
        "                state = torch.FloatTensor(state).to(device)\n",
        "                action, action_logprob, state_val = self.policy_old.act(state)\n",
        "\n",
        "            self.buffer.states.append(state)\n",
        "            self.buffer.actions.append(action)\n",
        "            self.buffer.logprobs.append(action_logprob)\n",
        "            self.buffer.state_values.append(state_val)\n",
        "\n",
        "            return action.detach().cpu().numpy().flatten()\n",
        "\n",
        "        else:\n",
        "            with torch.no_grad():\n",
        "                state = torch.FloatTensor(state).to(device)\n",
        "                action, action_logprob, state_val = self.policy_old.act(state)\n",
        "            \n",
        "            self.buffer.states.append(state)\n",
        "            self.buffer.actions.append(action)\n",
        "            self.buffer.logprobs.append(action_logprob)\n",
        "            self.buffer.state_values.append(state_val)\n",
        "\n",
        "            return action.item()\n",
        "\n",
        "\n",
        "    def update(self):\n",
        "\n",
        "        # Monte Carlo estimate of returns\n",
        "        rewards = []\n",
        "        discounted_reward = 0\n",
        "        for reward, is_terminal in zip(reversed(self.buffer.rewards), reversed(self.buffer.is_terminals)):\n",
        "            if is_terminal:\n",
        "                discounted_reward = 0\n",
        "            discounted_reward = reward + (self.gamma * discounted_reward)\n",
        "            rewards.insert(0, discounted_reward)\n",
        "            \n",
        "        # Normalizing the rewards\n",
        "        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)\n",
        "        rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)\n",
        "\n",
        "        # convert list to tensor\n",
        "        old_states = torch.squeeze(torch.stack(self.buffer.states, dim=0)).detach().to(device)\n",
        "        old_actions = torch.squeeze(torch.stack(self.buffer.actions, dim=0)).detach().to(device)\n",
        "        old_logprobs = torch.squeeze(torch.stack(self.buffer.logprobs, dim=0)).detach().to(device)\n",
        "        old_state_values = torch.squeeze(torch.stack(self.buffer.state_values, dim=0)).detach().to(device)\n",
        "\n",
        "        # calculate advantages\n",
        "        advantages = rewards.detach() - old_state_values.detach()\n",
        "        \n",
        "\n",
        "        # Optimize policy for K epochs\n",
        "        for _ in range(self.K_epochs):\n",
        "\n",
        "            # Evaluating old actions and values\n",
        "            logprobs, state_values, dist_entropy = self.policy.evaluate(old_states, old_actions)\n",
        "\n",
        "            # match state_values tensor dimensions with rewards tensor\n",
        "            state_values = torch.squeeze(state_values)\n",
        "            \n",
        "            # Finding the ratio (pi_theta / pi_theta__old)\n",
        "            ratios = torch.exp(logprobs - old_logprobs.detach())\n",
        "\n",
        "            # Finding Surrogate Loss   \n",
        "            surr1 = ratios * advantages\n",
        "            surr2 = torch.clamp(ratios, 1-self.eps_clip, 1+self.eps_clip) * advantages\n",
        "\n",
        "            # final loss of clipped objective PPO\n",
        "            loss = -torch.min(surr1, surr2) + 0.5 * self.MseLoss(state_values, rewards) - 0.01 * dist_entropy\n",
        "            \n",
        "            # take gradient step\n",
        "            self.optimizer.zero_grad()\n",
        "            loss.mean().backward()\n",
        "            self.optimizer.step()\n",
        "            \n",
        "        # Copy new weights into old policy\n",
        "        self.policy_old.load_state_dict(self.policy.state_dict())\n",
        "\n",
        "        # clear buffer\n",
        "        self.buffer.clear()\n",
        "    \n",
        "    \n",
        "    def save(self, checkpoint_path):\n",
        "        torch.save(self.policy_old.state_dict(), checkpoint_path)\n",
        "   \n",
        "\n",
        "    def load(self, checkpoint_path):\n",
        "        self.policy_old.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))\n",
        "        self.policy.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))\n",
        "        \n",
        "        \n",
        "       \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-xCb_EyxF8cF"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "\n",
        "################################# End of Part I ################################\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yr-ZjT_CGyEi"
      },
      "source": [
        "################################################################################\n",
        "> # **Part - II**\n",
        "\n",
        "*   train PPO algorithm on environments\n",
        "*   save preTrained networks weights and log files\n",
        "\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "YY1-DzVCF8eh"
      },
      "outputs": [],
      "source": [
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "################################### Training ###################################\n",
        "\n",
        "\n",
        "####### initialize environment hyperparameters ######\n",
        "\n",
        "env_name = \"CartPole-v1\"\n",
        "has_continuous_action_space = False\n",
        "\n",
        "max_ep_len = 400                    # max timesteps in one episode\n",
        "max_training_timesteps = int(1e5)   # break training loop if timeteps > max_training_timesteps\n",
        "\n",
        "print_freq = max_ep_len * 4     # print avg reward in the interval (in num timesteps)\n",
        "log_freq = max_ep_len * 2       # log avg reward in the interval (in num timesteps)\n",
        "save_model_freq = int(2e4)      # save model frequency (in num timesteps)\n",
        "\n",
        "action_std = None\n",
        "\n",
        "\n",
        "#####################################################\n",
        "\n",
        "\n",
        "## Note : print/log frequencies should be > than max_ep_len\n",
        "\n",
        "\n",
        "################ PPO hyperparameters ################\n",
        "\n",
        "\n",
        "update_timestep = max_ep_len * 4      # update policy every n timesteps\n",
        "K_epochs = 40               # update policy for K epochs\n",
        "eps_clip = 0.2              # clip parameter for PPO\n",
        "gamma = 0.99                # discount factor\n",
        "\n",
        "lr_actor = 0.0003       # learning rate for actor network\n",
        "lr_critic = 0.001       # learning rate for critic network\n",
        "\n",
        "random_seed = 0         # set random seed if required (0 = no random seed)\n",
        "\n",
        "#####################################################\n",
        "\n",
        "\n",
        "\n",
        "print(\"training environment name : \" + env_name)\n",
        "\n",
        "env = gym.make(env_name)\n",
        "\n",
        "# state space dimension\n",
        "state_dim = env.observation_space.shape[0]\n",
        "\n",
        "# action space dimension\n",
        "if has_continuous_action_space:\n",
        "    action_dim = env.action_space.shape[0]\n",
        "else:\n",
        "    action_dim = env.action_space.n\n",
        "\n",
        "\n",
        "\n",
        "###################### logging ######################\n",
        "\n",
        "#### log files for multiple runs are NOT overwritten\n",
        "\n",
        "log_dir = \"PPO_logs\"\n",
        "if not os.path.exists(log_dir):\n",
        "      os.makedirs(log_dir)\n",
        "\n",
        "log_dir = log_dir + '/' + env_name + '/'\n",
        "if not os.path.exists(log_dir):\n",
        "      os.makedirs(log_dir)\n",
        "\n",
        "\n",
        "#### get number of log files in log directory\n",
        "run_num = 0\n",
        "current_num_files = next(os.walk(log_dir))[2]\n",
        "run_num = len(current_num_files)\n",
        "\n",
        "\n",
        "#### create new log file for each run \n",
        "log_f_name = log_dir + '/PPO_' + env_name + \"_log_\" + str(run_num) + \".csv\"\n",
        "\n",
        "print(\"current logging run number for \" + env_name + \" : \", run_num)\n",
        "print(\"logging at : \" + log_f_name)\n",
        "\n",
        "#####################################################\n",
        "\n",
        "\n",
        "################### checkpointing ###################\n",
        "\n",
        "run_num_pretrained = 0      #### change this to prevent overwriting weights in same env_name folder\n",
        "\n",
        "directory = \"PPO_preTrained\"\n",
        "if not os.path.exists(directory):\n",
        "      os.makedirs(directory)\n",
        "\n",
        "directory = directory + '/' + env_name + '/'\n",
        "if not os.path.exists(directory):\n",
        "      os.makedirs(directory)\n",
        "\n",
        "\n",
        "checkpoint_path = directory + \"PPO_{}_{}_{}.pth\".format(env_name, random_seed, run_num_pretrained)\n",
        "print(\"save checkpoint path : \" + checkpoint_path)\n",
        "\n",
        "#####################################################\n",
        "\n",
        "\n",
        "############# print all hyperparameters #############\n",
        "\n",
        "print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "print(\"max training timesteps : \", max_training_timesteps)\n",
        "print(\"max timesteps per episode : \", max_ep_len)\n",
        "\n",
        "print(\"model saving frequency : \" + str(save_model_freq) + \" timesteps\")\n",
        "print(\"log frequency : \" + str(log_freq) + \" timesteps\")\n",
        "print(\"printing average reward over episodes in last : \" + str(print_freq) + \" timesteps\")\n",
        "\n",
        "print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "print(\"state space dimension : \", state_dim)\n",
        "print(\"action space dimension : \", action_dim)\n",
        "\n",
        "print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "if has_continuous_action_space:\n",
        "    print(\"Initializing a continuous action space policy\")\n",
        "    print(\"--------------------------------------------------------------------------------------------\")\n",
        "    print(\"starting std of action distribution : \", action_std)\n",
        "    print(\"decay rate of std of action distribution : \", action_std_decay_rate)\n",
        "    print(\"minimum std of action distribution : \", min_action_std)\n",
        "    print(\"decay frequency of std of action distribution : \" + str(action_std_decay_freq) + \" timesteps\")\n",
        "\n",
        "else:\n",
        "    print(\"Initializing a discrete action space policy\")\n",
        "\n",
        "print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "print(\"PPO update frequency : \" + str(update_timestep) + \" timesteps\") \n",
        "print(\"PPO K epochs : \", K_epochs)\n",
        "print(\"PPO epsilon clip : \", eps_clip)\n",
        "print(\"discount factor (gamma) : \", gamma)\n",
        "\n",
        "print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "print(\"optimizer learning rate actor : \", lr_actor)\n",
        "print(\"optimizer learning rate critic : \", lr_critic)\n",
        "\n",
        "if random_seed:\n",
        "    print(\"--------------------------------------------------------------------------------------------\")\n",
        "    print(\"setting random seed to \", random_seed)\n",
        "    torch.manual_seed(random_seed)\n",
        "    env.seed(random_seed)\n",
        "    np.random.seed(random_seed)\n",
        "\n",
        "#####################################################\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "################# training procedure ################\n",
        "\n",
        "# initialize a PPO agent\n",
        "ppo_agent = PPO(state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, has_continuous_action_space, action_std)\n",
        "\n",
        "\n",
        "# track total training time\n",
        "start_time = datetime.now().replace(microsecond=0)\n",
        "print(\"Started training at (GMT) : \", start_time)\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "# logging file\n",
        "log_f = open(log_f_name,\"w+\")\n",
        "log_f.write('episode,timestep,reward\\n')\n",
        "\n",
        "\n",
        "# printing and logging variables\n",
        "print_running_reward = 0\n",
        "print_running_episodes = 0\n",
        "\n",
        "log_running_reward = 0\n",
        "log_running_episodes = 0\n",
        "\n",
        "time_step = 0\n",
        "i_episode = 0\n",
        "\n",
        "\n",
        "# training loop\n",
        "while time_step <= max_training_timesteps:\n",
        "    \n",
        "    state = env.reset()\n",
        "    current_ep_reward = 0\n",
        "\n",
        "    for t in range(1, max_ep_len+1):\n",
        "        \n",
        "        # select action with policy\n",
        "        action = ppo_agent.select_action(state)\n",
        "        state, reward, done, _ = env.step(action)\n",
        "        \n",
        "        # saving reward and is_terminals\n",
        "        ppo_agent.buffer.rewards.append(reward)\n",
        "        ppo_agent.buffer.is_terminals.append(done)\n",
        "        \n",
        "        time_step +=1\n",
        "        current_ep_reward += reward\n",
        "\n",
        "        # update PPO agent\n",
        "        if time_step % update_timestep == 0:\n",
        "            ppo_agent.update()\n",
        "\n",
        "        # if continuous action space; then decay action std of ouput action distribution\n",
        "        if has_continuous_action_space and time_step % action_std_decay_freq == 0:\n",
        "            ppo_agent.decay_action_std(action_std_decay_rate, min_action_std)\n",
        "\n",
        "        # log in logging file\n",
        "        if time_step % log_freq == 0:\n",
        "\n",
        "            # log average reward till last episode\n",
        "            log_avg_reward = log_running_reward / log_running_episodes\n",
        "            log_avg_reward = round(log_avg_reward, 4)\n",
        "\n",
        "            log_f.write('{},{},{}\\n'.format(i_episode, time_step, log_avg_reward))\n",
        "            log_f.flush()\n",
        "\n",
        "            log_running_reward = 0\n",
        "            log_running_episodes = 0\n",
        "\n",
        "        # printing average reward\n",
        "        if time_step % print_freq == 0:\n",
        "\n",
        "            # print average reward till last episode\n",
        "            print_avg_reward = print_running_reward / print_running_episodes\n",
        "            print_avg_reward = round(print_avg_reward, 2)\n",
        "\n",
        "            print(\"Episode : {} \\t\\t Timestep : {} \\t\\t Average Reward : {}\".format(i_episode, time_step, print_avg_reward))\n",
        "\n",
        "            print_running_reward = 0\n",
        "            print_running_episodes = 0\n",
        "            \n",
        "        # save model weights\n",
        "        if time_step % save_model_freq == 0:\n",
        "            print(\"--------------------------------------------------------------------------------------------\")\n",
        "            print(\"saving model at : \" + checkpoint_path)\n",
        "            ppo_agent.save(checkpoint_path)\n",
        "            print(\"model saved\")\n",
        "            print(\"Elapsed Time  : \", datetime.now().replace(microsecond=0) - start_time)\n",
        "            print(\"--------------------------------------------------------------------------------------------\")\n",
        "            \n",
        "        # break; if the episode is over\n",
        "        if done:\n",
        "            break\n",
        "\n",
        "    print_running_reward += current_ep_reward\n",
        "    print_running_episodes += 1\n",
        "\n",
        "    log_running_reward += current_ep_reward\n",
        "    log_running_episodes += 1\n",
        "\n",
        "    i_episode += 1\n",
        "\n",
        "\n",
        "log_f.close()\n",
        "env.close()\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# print total training time\n",
        "print(\"============================================================================================\")\n",
        "end_time = datetime.now().replace(microsecond=0)\n",
        "print(\"Started training at (GMT) : \", start_time)\n",
        "print(\"Finished training at (GMT) : \", end_time)\n",
        "print(\"Total training time  : \", end_time - start_time)\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UEy2qKdZF8ha"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "\n",
        "################################ End of Part II ################################\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DHhK13_1G6zX"
      },
      "source": [
        "################################################################################\n",
        "> # **Part - III**\n",
        "\n",
        "*   load and test preTrained networks on environments\n",
        "\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SZWyhkq9Gxm5",
        "outputId": "eb21c926-f866-4fb7-cbd0-4c14ba7b45b6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "============================================================================================\n",
            "loading network from : PPO_preTrained/RoboschoolWalker2d-v1/PPO_RoboschoolWalker2d-v1_0_0.pth\n",
            "--------------------------------------------------------------------------------------------\n",
            "Episode: 1 \t\t Reward: 1644.37\n",
            "Episode: 2 \t\t Reward: 1630.13\n",
            "Episode: 3 \t\t Reward: 116.13\n",
            "Episode: 4 \t\t Reward: 1600.93\n",
            "Episode: 5 \t\t Reward: 1635.95\n",
            "Episode: 6 \t\t Reward: 1647.41\n",
            "Episode: 7 \t\t Reward: 1639.05\n",
            "Episode: 8 \t\t Reward: 1629.42\n",
            "Episode: 9 \t\t Reward: 33.27\n",
            "Episode: 10 \t\t Reward: 1623.71\n",
            "============================================================================================\n",
            "average test reward : 1320.04\n",
            "============================================================================================\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "#################################### Testing ###################################\n",
        "\n",
        "\n",
        "################## hyperparameters ##################\n",
        "\n",
        "env_name = \"CartPole-v1\"\n",
        "has_continuous_action_space = False\n",
        "max_ep_len = 400\n",
        "action_std = None\n",
        "\n",
        "\n",
        "# env_name = \"LunarLander-v2\"\n",
        "# has_continuous_action_space = False\n",
        "# max_ep_len = 300\n",
        "# action_std = None\n",
        "\n",
        "\n",
        "# env_name = \"BipedalWalker-v2\"\n",
        "# has_continuous_action_space = True\n",
        "# max_ep_len = 1500           # max timesteps in one episode\n",
        "# action_std = 0.1            # set same std for action distribution which was used while saving\n",
        "\n",
        "\n",
        "# env_name = \"RoboschoolWalker2d-v1\"\n",
        "# has_continuous_action_space = True\n",
        "# max_ep_len = 1000           # max timesteps in one episode\n",
        "# action_std = 0.1            # set same std for action distribution which was used while saving\n",
        "\n",
        "\n",
        "total_test_episodes = 10    # total num of testing episodes\n",
        "\n",
        "K_epochs = 80               # update policy for K epochs\n",
        "eps_clip = 0.2              # clip parameter for PPO\n",
        "gamma = 0.99                # discount factor\n",
        "\n",
        "lr_actor = 0.0003           # learning rate for actor\n",
        "lr_critic = 0.001           # learning rate for critic\n",
        "\n",
        "#####################################################\n",
        "\n",
        "\n",
        "env = gym.make(env_name)\n",
        "\n",
        "# state space dimension\n",
        "state_dim = env.observation_space.shape[0]\n",
        "\n",
        "# action space dimension\n",
        "if has_continuous_action_space:\n",
        "    action_dim = env.action_space.shape[0]\n",
        "else:\n",
        "    action_dim = env.action_space.n\n",
        "\n",
        "\n",
        "# initialize a PPO agent\n",
        "ppo_agent = PPO(state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, has_continuous_action_space, action_std)\n",
        "\n",
        "\n",
        "# preTrained weights directory\n",
        "\n",
        "random_seed = 0             #### set this to load a particular checkpoint trained on random seed\n",
        "run_num_pretrained = 0      #### set this to load a particular checkpoint num\n",
        "\n",
        "\n",
        "directory = \"PPO_preTrained\" + '/' + env_name + '/'\n",
        "checkpoint_path = directory + \"PPO_{}_{}_{}.pth\".format(env_name, random_seed, run_num_pretrained)\n",
        "print(\"loading network from : \" + checkpoint_path)\n",
        "\n",
        "ppo_agent.load(checkpoint_path)\n",
        "\n",
        "print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "\n",
        "\n",
        "test_running_reward = 0\n",
        "\n",
        "for ep in range(1, total_test_episodes+1):\n",
        "    ep_reward = 0\n",
        "    state = env.reset()\n",
        "    \n",
        "    for t in range(1, max_ep_len+1):\n",
        "        action = ppo_agent.select_action(state)\n",
        "        state, reward, done, _ = env.step(action)\n",
        "        ep_reward += reward\n",
        "        \n",
        "        if done:\n",
        "            break\n",
        "\n",
        "    # clear buffer    \n",
        "    ppo_agent.buffer.clear()\n",
        "\n",
        "    test_running_reward +=  ep_reward\n",
        "    print('Episode: {} \\t\\t Reward: {}'.format(ep, round(ep_reward, 2)))\n",
        "    ep_reward = 0\n",
        "\n",
        "env.close()\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "avg_test_reward = test_running_reward / total_test_episodes\n",
        "avg_test_reward = round(avg_test_reward, 2)\n",
        "print(\"average test reward : \" + str(avg_test_reward))\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n6IYC_JCGxlB"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "\n",
        "################################ End of Part III ###############################\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZewQELovHFt4"
      },
      "source": [
        "################################################################################\n",
        "> # **Part - IV**\n",
        "\n",
        "*   load log files using pandas\n",
        "*   plot graph using matplotlib\n",
        "\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 629
        },
        "id": "bY-E5HGcGxiu",
        "outputId": "2dd1e86f-e14a-440e-e61e-ab2ca8b0498c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "============================================================================================\n",
            "loading data from : PPO_logs/RoboschoolWalker2d-v1//PPO_RoboschoolWalker2d-v1_log_0.csv\n",
            "data shape :  (1500, 3)\n",
            "--------------------------------------------------------------------------------------------\n",
            "loading data from : PPO_logs/RoboschoolWalker2d-v1//PPO_RoboschoolWalker2d-v1_log_1.csv\n",
            "data shape :  (1500, 3)\n",
            "--------------------------------------------------------------------------------------------\n",
            "loading data from : PPO_logs/RoboschoolWalker2d-v1//PPO_RoboschoolWalker2d-v1_log_2.csv\n",
            "data shape :  (1500, 3)\n",
            "--------------------------------------------------------------------------------------------\n",
            "============================================================================================\n",
            "figure saved at :  PPO_figs/RoboschoolWalker2d-v1//PPO_RoboschoolWalker2d-v1_fig_0.png\n",
            "============================================================================================\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm0AAAGHCAYAAADiPGXHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3hUVfrHPycJKYTQQQRULLgoNhQLKooN3XVtqNhFfiira+9tFbtrX0UXF1kFV1ddXQs2lEXRVXQtC/ZVLKhUQ0kgIXXm/P545zJ3bu6duZNMkknyfp5nnszccs65dxLul7caay2KoiiKoihKdpPT2gtQFEVRFEVRUqOiTVEURVEUpQ2gok1RFEVRFKUNoKJNURRFURSlDaCiTVEURVEUpQ2gok1RFEVRFKUNoKJNUZS0MMYMMsZYY8zwVlzDXGPM/e1hHu/9NMaMin3u3ZzzhsUYc4wxRmtDKUoWoKJNUToYxpjpMVFgjTH1xpifjDFTjDE9Wntt2Ywx5lZjzLeebQNj9/Elz/YDY9u3bNlVBmOM2dEY84Qx5mdjTJUx5mtjzGXGmGZ/Dhhj9jHGzDTGLIndl9Oae05FaY+oaFOUjsm/gI2BQcDpwGHAn1tzQW2AN4EtjTEDXdv2A34GRhpjcj3bf7LWfteSCwzCGJMP7AKUAqcAQ4FJwDXAFS2whC7A58D5QFULzKco7RIVbYrSMamx1i631i621r4OPAWMBjDG5BhjrolZZGqMMZ8ZY47wGWNrY8w7xphqY8z/jDGj3Ttj1pX/xPavMMbcExMP7v3vG2MqjDHlxpgPjDHbufbvYYx5wxhTGdv/hjGmv2uKHGPMLcaYlcaYX4wxd7qtRsaYHsaYGcaYNTHL0r+MMUM9axwTu76a2PVebYwxAffsXaAOEWQO+wGPAuuAnT3b34jNcbIx5kNjzLrYOp82xgwImKMBxpgCY8xzxpj/GmP6xraNN8Z8Gbu33xhjLvRcuzXGnG2MedYYUwncYq192Fp7nrV2rrX2e2vtk8AU4GjPfKcaY340xqyPWRA3SrG+vxtj/unZlhO7nxcBWGtfsdZeZa19BoiGvXZFURJR0aYoHRxjzBbAIYggAbGGXApcDmwPPAc8a4zZyXPq7cB9wE7AbOAFR4zEfr4KzAeGAROAE4BbY/vzgBeAd4Adgd2BPwGR2P4dEcvWt8BewB6IsMxzzX8SUA/sCZwDXAAc59o/PTbuEcBuwHpgljGmKDbHLsDTwLOx67wCuDI2VgOstZXABzQUbXOBt5ztxpguwK6x9QPkI1atHYHfAr2BJ/zm8GKM6QrMAnoCo6y1vxhjzgBuAa4FtgEuRr6r33tOnwS8Eru2BwKm6Aqscc23O3LfpiLf64vADSmW+RhwqDGmm2vbvoglN9R1KooSEmutvvSlrw70Qh7K9UAF4qqysdeFsf1LgGs958wFHou9HxQ7/mrX/hzgG+Cm2OebgYVAjuuY04AaoDMiQiywb8AaHwfeS3INc737EeE4LfZ+cGz8fVz7uwHlwOmuOd7wjHEdsNgzz/2uzzcCP7jug3M9E4FXY9sPic09MGDtQ9z7XfdzeOzzqNjnbYGPgZlAoev8n4BTPGNeAHzp+myBySl+D3YGqoGjXdv+Dsz2HDdNHhWB4+QBK4AJnnNeDzi+Ajittf8O9KWvtvhSS5uidEzeRiwpuwGTEYvMfTHLTn/EFejmHUREuHnPeWOtjQL/cR2zDfB+bLt7jHxgK2vtakQ8vmaMedkYc5ExZlPXscOIuReT8Knn81Kgr2v+qGeN5cBnnjX6XeeA2H3w4w1gkDFmEGJZ+9Baux4Rd3vHLIj7Ad9aaxcDGGN2Nsa8EHM5rgM+io21qXdwD68Bi4Ex1trq2Fh9gE2Av8TcyhXGmArgj4A36eEjAjDG/Ap4GfiTtdbt2twG1z2L8Z7rvE3d8xpjrrLW1iNW0JNixxQgLtfHUlyfoihpoqJNUTom662131prP7PWnodYi65JcU6myj6IKcja8Yj78m3gcOBrY8zBaYxT5/lsCfdvWpjrCDrmPcS6Nir2mgtgrf0GiWsbHtvuxLMVI+JrPZIAsCtiiQMRsMl4Cdgb2M61zbm+MxHR7by2Q5IL3FT6DWqMGRJb95PW2nSTEJZ65n0wtv0xYN+YW/xQ5NqeTXNsRVFSoKJNURSA65G4qC7Ig3kvz/69gS892/Zw3sSC93cDvopt+grYwySWk9gbqAU2ZFRaaz+x1t5mrR2FCIlxsV3zgf0bfzl8hfz7NsK1xq5IfNeXrmP8rnOxtXad36Axi9d7iDXNiWdzeAuJn9uFuJVwCBLDdpW19m1r7f+IWwNTcQ0iiuY48YTW2hXI97NlTHQnvFINaIzZNrbmp621F/oc8hWu7zXGhs/W2nrPnKtj2z9A4g9PQCxuL1hrK0Jep6IoIclLfYiiKO0da+1cY8yXwB+AO4AbjDELkZiqk4GRJGZHApxljPkGcTn+HtgMyUYEKR9yAfBnY8y9wBaIC+9+a+16Y8zmwO+QeK0lsf07uM6/A3jfGDMVCaKvjq3hdWvtTyGuZ6Ex5gXEjTgRKEPi7NYicVsAdwEfGmOui23bFQnqvyrF8G8ClwAFwDzX9reQ5Ixc4kkIPyGWuXOMMQ8g7scbU63fdR1ONuu/jDEHWGs/QRIMJhtjyhC3difkuxlgrb01aKxY5uwbsbXdYozp55pneeztfcA8Y8yVwDOI1fCokMt9HCkfMwgY45m7C7BV7GMOsGlMiK4O830qihKjtYPq9KUvfbXsC4kle8ln+4mIwNgMsfL8jFjGPgOOdB03CHEfnoSIlmrga+DXnvH2QeLcapBA9XuAgti+jRD32ZLY/p8QwdPJdf7eiOu0ChFd/wI2ju2biytBwO+6gB7ADCQ7sip2/lDPOWNi11cbu96rAePa7zfPyNj1v+PZ7iQYfO7ZfhxiXaxGsk8Pjh03ynM/vYkIvV1j3AqsBHaMfT4B+G9szDVILN7xruMtcIxnHdcRTzpJeHmOGx/7PqqQDOBzvMcE/F5tERtvBZDn2TcqYO7prf33oC99taWXsVa7kyiKoiiKomQ7GtOmKIqiKIrSBlDRpiiKoiiK0gZQ0aYoiqIoitIGUNGmKIqiKIrSBlDRpiiKoiiK0gZo93XaevfubQcNGtSsc9TV1dGpU6dmnaOjofc08+g9zSx6PzOP3tPMovcz87TEPf34449XWmv7+O1r96Jt0KBBfPRRYAu+jLB06VL69+/frHN0NPSeZh69p5lF72fm0XuaWfR+Zp6WuKfGmB+D9ql7VFEURVEUpQ2gok1RFEVRFKUNoKJNURRFURSlDdDuY9r8qKurY/HixVRXV2dkvEgkQnl5eUbGUoSWvKeFhYUMHDhQA3YVRVGUrKZDirbFixdTUlLCoEGDMMY0ebza2lry8/MzsDLFoaXuqbWWVatWsXjxYjbffPNmn09RFEVRGkuHdI9WV1fTq1evjAg2pW1jjKFXr14Zs7oqiqIoSnPRIUUboIJN2YD+LiiKoihtgRYRbcaYh40xvxhjPvfZd7Exxhpjesc+G2PMfcaYb40xnxpjdnYdO84YszD2GtcSa1cURVEURckGWsrSNh04xLvRGLMJMBr4ybX518Dg2GsiMCV2bE9gErA7sBswyRjTo1lX3U6ZO3cuv/3tb1tl7urqanbbbTd23HFHhg4dyqRJk1plHYqiKIrS1mgR0WatfRtY7bPrHuAywLq2HQE8aoX3ge7GmI2Bg4HZ1trV1to1wGx8hGBbxFpLNBpttvEjkUizjZ0uBQUFvPHGG3zyyScsWLCAWbNm8f7774c6N5uuQ1EURVFamlaLaTPGHAEssdZ+4tk1APjZ9XlxbFvQ9qZxwQUwalSTXnkHHZS47YILUk67aNEifvWrX3Hqqaey3XbbceONN7Lrrruyww47bLA+3XHHHdx3330AXHjhhey///4AvPHGG5x00kkAnHXWWQwfPryB1WrQoEFcfvnl7Lzzzjz99NPMmjWLIUOGsPPOO/Pss88mXdsHH3zAiBEjGDZsGHvuuSdff/01AHvssQdffPHFhuNGjRrFRx99RGlpKQcddBBDhw7l9NNPZ7PNNmPlypW+Yxtj6NKlCyClV+rq6pLGlHmvw5kTYOXKlTh9ZadPn86YMWM45JBDGDx4MJdddhkgQu+0005ju+22Y/vtt+eee+5Jeu2KoiiKkq20SskPY0xn4CrENdoc409EXKsMGDCApUuXJuyPRCLU1tYCkBuJYKxtMEY6WGtx28lsJEIkNn4QtbW1LFy4kGnTpnHcccfx7LPP8s4772CtZcyYMcyZM4c99tiDP/3pT5x55pl8+OGH1NTUUFlZydy5c9lzzz2pra1l0qRJ9OzZk0gkwiGHHMLHH3/M9ttvD0C3bt14//33qa6uZujQocyaNYutttqKk046iWg0uuEeeNliiy2YM2cOeXl5zJkzhyuuuIKnnnqKo48+mieeeIJrr72WZcuWsXTpUnbYYQfOP/989t13Xy677DJee+01/vrXv1JbWxs4fiQSYY899uC7777jzDPPZNiwYQ2Ora+v3/DeuQ6AKVOmUFdXlzB+bW0t9fX1zJ8/nw8++ICCggK23357fve731FaWsrPP//Mf//7XwDKysp81xWJRBr8nrQ3SktLW3sJ7Qq9n5lH72lmSbifdXWY+npsUVHrLagd0Nq/o61Vp21LYHPgk5iVZSDwX2PMbsASYBPXsQNj25YAozzb5/oNbq2dCkwFGD58uPU2dy0vL4/XAJs8uYmX4l9TLDfFOfn5+Wy22WaMHDmSSy65hDlz5rD77rsDUFFRwaJFizj11FOZP38+1dXVFBYWsssuu/Dpp58yb9487rvvPvLz83n++eeZOnUq9fX1LFu2jIULF7LLLrsAcNJJJ5Gfn8+XX37J5ptvztChQwE49dRTmTp1amAdtKqqKs444wwWLlyIMYa6ujry8/M58cQTGT16NDfddBPPP/88xx57LPn5+bz33ns899xz5Ofnc9hhh9GjRw/y8/OT1ln75JNPKCsr46ijjuKbb75hu+22871H7usAsdR16tQpYfz8/Hzy8vI48MAD6dOnDwDbbrsty5YtY+jQoSxatIiLL76YQw89lNGjR5OT09DAnJub2yEaK3eEa2xJ9H5mHr2nmWXD/Vy6FDp1gl69oKCgdRfVxmnN39FWcY9aaz+z1va11g6y1g5CXJ07W2uXAzOBU2NZpHsA5dbaZcBrwGhjTI9YAsLo2LY2S3FxMSCWuiuvvJIFCxawYMECvv32WyZMmECnTp3YfPPNmT59OnvuuScjR47kzTff5Ntvv2Wbbbbhhx9+4M4772TOnDl8+umnHHrooQn1xpzx0+Waa65hv/324/PPP+fFF1/cMOaAAQPo1asXn376KU899RTHHXdck66/e/fu7LfffsyaNSvpce7ryMvL2xD/562tVuD6hyg3N5f6+np69OjBJ598wqhRo3jwwQc5/fTTm7RmRVGUNo3Li6G0PVqq5McTwHvAr4wxi40xE5Ic/grwPfAt8BDwewBr7WrgRuDD2OuG2LY2z8EHH8zDDz9MRUUFAEuWLOGXX34BYOTIkdx5553ss88+jBw5kgcffJBhw4ZhjGHt2rUUFxfTrVs3VqxYwauvvuo7/pAhQ1i0aBHfffcdAE888UTS9ZSXlzNggIQLTp8+PWHfcccdx+233055eTk77LADAHvttRf/+Mc/AHj99ddZs2ZN4NilpaWUlZUBYtGbPXs2Q4YMSboeN4MGDeLjjz8G4Jlnnkl5/MqVK4lGoxx99NHcdNNNG9ykiqIoHQZ3CFATw4GU1qVF3KPW2hNS7B/kem+BswOOexh4OKOLywJGjx7NV199xYgRIwDo0qULjz32GH379mXkyJHcfPPNjBgxguLiYgoLCxk5ciQAO+64I8OGDWPIkCFssskm7LXXXr7jFxYWMnXqVA499FA6d+7MyJEjWbduXeB6LrvsMsaNG8dNN93EoYcemrDvmGOO4fzzz+eaa67ZsG3SpEmccMIJ/O1vf2PEiBH069ePkpIS37GXLVvGuHHjiEQiRKNRxo4dm1b5kUsuuYSxY8duuJ5ULFmyhPHjx2+wzt16662h51IURWkXuDPvm7FSgdL8GNvOVffw4cOtk23o8NVXX7HNNttkbI6O3nu0pqaG3Nxc8vLyeO+99zjrrLNYsGBBk8Zs6Xua6d+JbGTp0qUaL5RB9H5mHr2nmWXD/ayrAyeAvrgYunVr2sC1tbBmDfToAe5/p1evhrw86NpVjunUCdp6x5loVKyTuRKp3hK/o8aYj621w/32dciG8Upm+emnnxg7dizRaJT8/Hweeuih1l6SoiiK4pBp96hT0mnNGthoI3lfXw9OnHFuLpSXS8JDr15Nn681Wb5cfm68cVYIUBVtHZhHHnmEe++9N2HbXnvtxQMPPJDWOIMHD2b+/PkJ21atWsUBBxzQ4Ng5c+bQy+eP+KijjuKHH37Y8Nlay+23387BBx+c1loURVHaFZEIlJVBly6Nz/psrpg291iVlQ3f19Rkbq7WwHvfVLQprcn48eMZP358s4zdq1evtFykzz33XMLnju5yVhRFAcRiVVMjr8a65dxxbC0h2jIRN7d+vVjuevRoPbHkjQX0KRfV0rT+ClqJ9h7Lp4RHfxcURclaMvHvUzqWtkhEBFhT5s3EmsvKRLRVVTV9rMbSXGK3CXRI0VZYWMiqVav0Ya1grWXVqlUUFha29lIURVEakgkrUzqibeVKse7FSlC1Oq35nHZb2rJEL3RI9+jAgQNZvHhxxtpRRCIRcnNT9UBQ0qEl72lhYSEDBw5skbkURVFanGSirb5esj5LSqCoKC5UampkW7KxwszXVLJFtGVJqZQOKdqcTgOZQtPUM4/eU0VRFJrf0lZWJsJtzRoRbUHnrFkDhYXycm/PlPt25UoZ2ysUW1O0qXtUURRFUZRG0ViRlEx81NbG37u72TjnVFXBL79IfFlZWUOLU21t061Q1dVSS86v6Lta2hJQ0aYoiqIobYHly0VApUuQpc0riPyC/tesSR7btXp10wWN25qYJRYtICvbf6loUxRFUZRsxWsli0TSFxDu450K/96xU53nsHZtw2MaK2j85ncLxKA1tBQq2hRFURRFaRJeYZMKr+AII9qM8d/vLZhbUJB8nCCxU18vlsM1axqKoywRSA3Ebhagok1RFEVRMkldXfrCKgg/AdNU0eYIEHdBXD/CCJVoNH6cX9LE+vXi0vWu2XHFVlUlzuO2BDprj0blnjYHyQSiWtoURVEUpQWJRFq2nVJNjTRnz1BJKV9CWn3M2rUimBzB44gq5/z16xs/T/fu8WMcQePXMaC8XKxqfm7VVO+dz879dCdNZIKaGli2rGFNuspKuTdqaVMURVGUFqSsDFatSm1VyhSOsIhGGwqVxuBn4Qkr2tavF8HkHO/UvvR+TmduB6f0h9udmazNk3usurpEAZbKPepY6TIt2pxsVff3ZK0ITW+mrFraFEVRFKWZcaxsLSXa3NarTHQV8BMLjRUQeXn+5/fqFd/nniNoHmPiVjv3cclqyrnH8lrNUrlHHZKJwrKyxJIlqdYA/qI1qMyHijZFURRFaSHq65u/j2VVVWZi2WprxW0XtN5klrZ166QMh5ecnLjocdbojJOfD337NjwnSKjk5PiLNmf8hQvh//4P7rhD7nuysSB+jHc891qTjVFXJ2LZGx/nJhqVxAf3vUkl2rznZwEdsiOCoiiK0kEwJv6wr6lJrPrfVCIRERwFBfLZT2RFo8ktRH6sXRvvQuAnLFKJNmgYuJ+TE78Pa9dCcbF8dlvNHOrqRNw4Ysq73/mck9MwEeGFF+DCC+Vev/YaLF0Kd98dn9tPeLlF29q18Xg577UGiTZ3zGKyY6yVQr5+OPciSLTV16eOAWwB1NKmKIqidAwy7eL65ReJl3NcfW6B5Y0fS4dUrauCriNVCQ+3OHKOdQTl11+L2DriCNhpJ9h+exg4EEaNgnvuSRSkzvq8iQ0zZ8I558COO8L8+XDRRfCPf8AFF4h4C1qjVyiVlflfU9D1uQVq0L1J5WZ23iezlJaVNV8Wa0hUtCmKoijtE6+rLdOizRnPEW1ui5oj2sK4S6uqJPg92Rw9e8b7cgZdR7K2S273qHt/Tg58/z2MGAEvvQSdOsHBB8MNN8B110Hv3nDnnbDnnvD883GLFMR/fv45nHuuuER33hkee0zcrRddJNufew6GD4d7721aYeAwiQhNLfTr/b48VlLTyqJN3aOKoihK+ySoqGxzz9elS9yqFcbS5gTQ5+eL+9bPupSfnxhHlmx+aCg+jIGuXeNtsNxxbSedJD9ffx023zx+TnExnHEGfPcdnH8+nH22CDtHzP38M1x+OcyZI+s+6ywRacXF8TmvuAJOPBGuvVYsbnl5cNRRqe+JH7W1iaLR77oba2lz7rP3+zIm7gb229/CqKVNURRFaZ+0lmjLzY0Li/Ly8JmrjhXHb51h+nMmqytmjAgmbzLClCnw/vvw4IOJgs093s47izC7/HKYOxf23hvGjYP994f//Ee2f/gh/PGPcWugm003hYcfhoMOgssugyVL/NcfhlWrYMWKRHet+7rXrk10r1ZXi4B2H+PE/fmJPb/71rNnPG6xlUWbWtoURVGU9klLiTZvkL07uD8aFeHWuXPqWDVHECQrtRFmHe6xvOc6PyMREUB33QWHHQbHHy9CqKIiLh7dCQZ5eXDeeXLcrbfCo4/CfvvBTTfBFlvIscnWl5MD990n8W5//CNMnpz8WqJReOIJWLRI1vOf/4gY3n13OOEE2GqreFKJ14VaWyviMRKJZ4u6xeS6deIGDiva8vPl+6upaXXRppY2RVEUJbtoLnGV7gN37Vr/8hnesZKJtsbMHWRpa4p71LGwucXkXXdJRuTtt8u2oiLo06fhPO65+/aFqVNFwDz9NAwa1DCpIYiuXcV9+uyzklnqzhJ18913cNppYpX7859h+nQRTfn5MG0a7LsvDBkibtfy8mDXpzv+zHv/vdY3a0WwupM1nGt3/TS1ta0q3NTSpiiKomQP5eUiJPr2FctKTU3c2pEu7vph3qKtqYhG48VxI5GGpTeSWWn8RJvfGEHrTWVpc2c63n67uCzHjBELlHv9fjjC6quv4G9/k+SBIUMazpOsaK7zfQSVAknGlVfCG2/A738PgweLlc6hb1/4+99h4kSZ+8YbRbxFImIZA4nJe/ll+PJLyUz94AMRkd5rCOq3GvQZ/DtYONeUny/fn7WynnTLuGQIFW2KoihK9uDEf1VVSUD7qlXyuX//xo8ZRkx48bZZikZlbcXFiTXPwN+llklLm9e1aS2sXAlHHw1vvw39+kkSwRNPwCOPSPyV19LmdDxwxMZ114ll7aqrwq3DfT1eq53fMUEUFsoaDz8cDj1U4t0KC+V9ZaXUdNtuOzmmX7/4fCUlIuD79oXx42X7738PxxwjsXJXXCGf3fcoVRxgmMK97pp0vXsTXb8+LiBbAXWPKoqiKOGprxfB0BxN2JOVrGiMy9RbqT+dMby1v8rKJBbKyfT0s7Q1xj2aqu+mH2vWwD77SJzXI49IDbS//hXeekvixbzzFRTE479yc+Hdd0XknXeeuEO9JHOPuvf7nRcm7q5PH3jqKUlw6NNHLJrXXy+lRY46StyujmBzyMmRrFw3I0bA7Nki+G65BS6+OC62vffS7/fJK9qc35MePfyvNagsSwuiljZFURQlPE6l/NWrYeONMzu224UYiTR86KZyLzpUVIg1xM9CFRZvzJojUp2fqURbWLzXmCxz1Pl5+eXw7bcwa5ZkcIK4Od9+W8TbSSdJoD6IFatnz8SxbrxRLJcTJoTrFxok2rznBn0/ubnyfbr7m267rQgupwTJ8uViRcvL8xe4OTkNXeSRiFzblCniar37bklcmDZNLHNOlqhzrPdc73fozOtep/sazz2XPh9/LHXpvL1aWwi1tCmKoijhCdNLsrF4LW3JLCVB1NZKbNKqVf4iKuy6vaItTG2wIHdisnnDWNrcImnBAuk8cNVVccHmcM01YlG77rrgdcycCZ99BpdeKsf6iTY/kZtKtOXkJJY68Ruvvj4eN+bUP3MYMEA6MATFijnHb7RRfJvz+2KMWNnuv186MRx8sAgrN6li2oLiEZ33H3wAf/871aNHt5pgAxVtiqIoSmNoTJxYqiGTtSwKK9qCjkt3vanEVLKCrn5uwrCizVm/n9XKGHjoIbEiXXRRw/29e8v2N98UK5ZzjkN9vVjZBg+WeDjv/qC1ea/HL6YtKM7Nvc97LWGsd94xcnPjoslrPTvqKOl9WlkJp5yS6M5MVf4lmVWxvl7ac/XpQ8XZZ/uvr4VQ0aYoiqIEY60UM3ViuRyaQbQltbQ1JaYtTLkML15LWxjRliymDUREPPCAZD1WVkosmpOh6p3HLXScsUpLJXPy2GPF7ek+z+kWMH48bLklTJoUz7x1ePxx+OYbKaXhiMKmuEfdJMumDLK+NUa0ud/7tQjbYQdxEf/4o9yLl19OLNviuFiDskv9vrs//EGKBz/wANaveHALoqJNURRFCaauTh6OVVXh+mg2Bff4kUhizaywYivIFZqOaHNEULK5U4k2rwtt3TqpL3bOOXDccXDqqSIU1q9PHMdPtDnbpk+X7+OUUxITQcrKJDlk/XoRJTfeCD/+SJepUxNdkzfeKKLm17+OnxtkBUu2Lcg9GubcoG2p/hPgN3+QVXXECLjjDknUmDhRCvqOHQvPPCMiORIJTnRxi7a6OhG4t90Gp58uYrmV0UQERVEUJRi3OPEWHs0wxiuO3BYSa+Of3YH1yWhMYgBI6yM3bmHlHdv73qFTJ1nnunUisM46S2LJnnsOPv1ULGF77SUdBtzjBBWqjUbhL38RQbL11omi0t3SCUQcHn88XaZPF1foZZfBzTdL0dp//jPxfoSJzzIm0V3rJ9qS3eOwQi7Zdj/LYzKX+dixcn+XLpW6cC++KP1Tzz9fzu/ZUxIfNtoI9txT7mn//nK/jIGFC+GSS+Cjj+DCCyWzNQtQ0aYoiqIE434wNrelLRn19XEx5ZcY4P0bgEUAACAASURBVBCUvNCYDFIHt0DyGyMoe7SwMG7heuklyW488kipUfbii1Ke4/DDpdq/c77bTece/803RXRNmiTbamok89I518utt1K1dClFV18t5TAqK6WDwG9/K1Y5ECEUxtLmWJ9KSuReOELPz9LmhzFyL9xi2E+U+s3tt6Zk7lE3AwbAJpvArruKcH33Xel7+tNP4mouLYUffpD742aTTeDnn6Um3yOPSIHfLKFFRJsx5mHgt8Av1trtYtvuAA4DaoHvgPHW2rLYviuBCUAEOM9a+1ps+yHAvUAuMM1a+8eWWL+iKEq7IhKRB3deHvTqlfzYoOSA5sgedcZ0yjU4nQwgUTg55T+cXpCdOvlbjBob05Yq6cEtroLmc3jgAemzOW4cXHCBbMvJgRtugN/8RuKvzj03frxTXNgrXh56SOqHHXusxBdam5iN6SU/n/K776bohx8krmvXXeGMMxKPSdcC6Y3ncguvVBa7VO7QdH6fnHndtfSSHev0Rd17b0nWcESrw+rVIuZ++EFKhnz7rbivTzlFLG9ZREtZ2qYD9wOPurbNBq601tYbY24DrgQuN8ZsCxwPDAX6A/8yxmwdO+cB4CBgMfChMWamtfbLFroGRVGU9kFtrQg3p1ZVsod3a1janIdy0NxOX0mnWwJIMVZvpwJ3SYh0RFsYN3CqRAUQK80VV4g4u/vuxPs8fLhU8n/gATj55HhBV2dutzuytFT6dZ59tlisjAl3HcZIM/hjjvG/tmTlNdzvwxznJEYElSzxbnfG7NZNEjS6dWs4Zl6erNc7fzpiM0y7qZ495T8v228vhYhT/UemFWmRRARr7dvAas+21621zm/P+8DA2PsjgCettTXW2h+Ab4HdYq9vrbXfW2trgSdjxyqKoijpkE5WZlMzONPBLwjfwU+0uXE++4m2dHGsXUEPfD9Lm9fK9txzErx+wAFSP8w7lrVSJHfdOnjwwYZzuIXJo4/K9f3udw33+eGu6N/UmmJhs0LT7cXpnFtcLEWaHdHnHrNLF3l5uzYUF4efxxuLl8qt3hxZ0RkkW7JH/w94NfZ+APCza9/i2Lag7YqiKEo6pBJiTrxUXV3jCtw2lVQPTm8LIvC3jqWytDnu1aDrcpfV8M7vPae+XrI4lyyBGTMkwWD33eHJJ8V6473PkQhss41Ywh55JLhFUm2tjHfIIfGm6KnuT34+bLQR0V69GtZ7c58bxr2YrAtF2JId3m1OId6g/e65u3ZtuAa/7gh+4tRvnnSyVLOQVk9EMMZcDdQDj2dwzInARIABAwawdOnSTA3tS2lpabOO3xHRe5p59J5mlrZ8P01lJSbW4idaX9/ggZezfLm8WbwYW1SEcbIT8/PjsWXGEM2w5W3NypVQX0+0tpac1asDj4tWV2NqazGurE5bWYktLsasX49xV923FltcDHV1mNpaojU1UFCAqarClJfHjyksxHbvDtaSs2KFjNmtmxzjnb+ujpy1azeUQyn65z/pMmMGuc59A2qHDWPVtGnY8nJyysuxBQVY5z5GIuTEfn/yTjyR3i++yLrJk6k8/fT4HDU15JSVUfjyy3QvLWXVySdTE3uW5axalVRwRaNRyMmhtKzMt7l5jmudfnLVrFmDiZUUsQUFWG8ixoaLjH9Pzjg5se/QjS0uhkgEE0tEiG60kWR1+uC+Nue78sO9RoBor15yrpv8fGx+PsaphZeTI79b3pg291qLiuLfkw+t/XffqqLNGHMakqBwgLUb/vqXAJu4DhsY20aS7QlYa6cCUwGGDx9u+/fvn8FV+9MSc3Q09J5mHr2nmaXN3s916+Iupj59fB/sG+jUKS4QnB6SIGInVe9RJ9PQ7TqrqBDx57WUADmlpfTr00dKMfjs30CPHpKJ6H64lpTIq6KiYUZlSYlcQ3W1xC8VFkrwudNE3aF//3jQek6OxFk5x5SUiGWutlaC2Z0xTjtNWhztuae4L7t2hW23Jf+AA9i4UyeZs6goMVaqtjZuAerXDw44gJInn6TkggvEHVhUJNe4dCn87W+w9db0OuGE+H0sKEis0+Zl4403WIx8f0e7dJH7FBS/5c709PYu9VJSIt+Vk6Tg/n1xHxOJxEunJPu7cf+O9e4d/HtQVBT//rt2ld9n7+9xYaGc74j43FwZ0/2fFCemzqFLFxkvCa35d99qoi2WCXoZsK+11l0EZybwd2PM3UgiwmDgA8AAg40xmyNi7XjgxJZdtaIoSjsgmXvU6/ZzP4C9cWXJqKuTLL2cHBEmIELDeYD6PfjCxhVFow2FQbpWP7853F0Y/Fo3uV2spaVwwgnwv//BfffBmDHxorp9+8bPc4SE07HAmIb3+KKL4NBDpdH5BRfERcOUKdLB4M9/9q9Tls61uSkpkXUGuX/D1l+DhqIvN7fhd5MsmcFLWHe8d41BXRpSZa02JcmhFWiRmDZjzBPAe8CvjDGLjTETkGzSEmC2MWaBMeZBAGvtF8A/gC+BWcDZ1tpILGnhHOA14CvgH7FjFUVRlMbiFTuZKqDrCLxoVN6Xl/sXqfUj1YOzvLzhOv2atrvHC8oe/fFHaSv1xReJWaDJRFttrVjVFi4US9jRRwfHdOXmyssdB+cVIzvtBKNHS0JCWZmM8d//wq23ipg7/PCG19MUjBFrZBghle5c3bqJGPRavUpKxDqWKjPTLSSTWVvD4BVtmejM0Mq0iKXNWnuCz+a/Jjn+ZuBmn+2vAK9kcGmKoigdj3QsbanGCVPktro6npGZ7NzGZvC98opU+t9++3jF+2TU1Ynoe/55OO88EZXGSC20U06RY/wsMM64d98N8+bBPfdIdqjbsuQ3t3Nd5eXiavS7x5ddJiVAbr9duhccfbS4rm+7LbV1qDlJ97vIzZVrLC+P3xfn3rmzWoPo1k3ctgUF4TNXnfd9+si9dmLWvMLbzyLnnaMl720jyO7VKYqiKJknmWgLKruR7sMsVemNVOIwrFiYPVuKxs6fD3fdJbXMvNfkHisalXZS55wjx+6yi4i+PfeEM8+Ef/87fo63Sbkx8OWX0uHg8MPFPRpmnc61OnFizv1wx1Ztu600OJ8xA371KxEezz4rbZa81qnmtgal4x4NM0a65xUVpf5981tjp05inevcWd4XFyce55cJq5Y2RVEUJasJY2nr1Ckx2D03t6HQWrVKHpJ+gdupYpOcrgZB5/jhFFt1qK+Ha6+V0hmzZ8MTT8Cll4rIufhi6N49fqzzMP7xRyl2u3q19AO9+GIRCVOnwhFHSDP3l1+WSvhuUeWcf8MNEqx+yy2yfr/6a168xWWd+9q5czzGzxj4wx/EPVhRIf0ud9rJ/z60VOkVZ13ZMEY6uL93N7m56ce4ZRnZvTpFUZS2jLUibMrKWnsliYSxtHnrXvnVwaqthfffl7iro4+GDz/0n8/P0pZMoAUFlnvX8MIL0kfy0ktFeF18sbSLevhh2HFHef+//8XHWrNG1llZKcLsD3+IZ4d27w7PPCNCcNw4EU7emLZnnxVL3EUXiavPSTxIda2OpcwRqY7ocsds5eaKS/Dqq8XaFiTYwD+BIJNiI9MiqzlEW1hroF8B4GRFgVW0KYqidFDWrROryvr1ic2ym0J1dWbbSSVzj6ZyLX38sbRI+vBDETN77y1B/d5xk/XpDMI9d0FBvBSGe8zJk2HoUIkFc2Lk7rkHZs2SbgQffwy//rXEiU2fDvvvL30lH35YzvOy5ZaSWLBwIUycKPe5Tx+J0Vq9WuLOdtxR3JjOGr0B937X6gg7b7KEWyAkK7vipbBQhKD7fiQrgtsUWto92pjxw4q2MO7R5rqPGUJFm6IoSnPhdi+GqT6fivXrRTx4i4imSxj3qFe0eS0Qn34qVf979RLR9tVXsNtusm3WrNRdF1K1gXLjuGDda3jxRRFXl1+e2HPUWklIuOYamDsXRo2C668Xd2O/frK2vfZqOIdz7gEHiAv0tddg331h5kz4/HMp6bFuHdx5Z/zBbky4h3xQ5qr7eoqKxF0a5NrzUlAgGZnGJAraFDXGQpGJmLag8Voa9/fjZ0VrY+5RjWlTFEXJNNbKA94t1FLFIVkrxUILC4MfHE7JjPp6aTPVu3fTHzJ+7ZUgXqbCwf1wW7JEXIiOS3GTTWQdr70mrZsmTpSSFQ5NtbQ5751rrayUZIAdd4RjjxUh61fyo3dvsap9/72cO2iQiJ2gCv+xTgKcdpoIqHvvjTdbLyyEv/5VEgYcvG2SwL9MhVu0ucuKeL+7sILNIS9PhKgzflFR5i1F2RqYH1ZYphJtQWNmKdktKRVFUdoiVVUSE+UmlWirqJDYtyQtdhIESX19+Lpnycbxkkq0WSuirLJSXIkbbxw/rnNnKYfx88/wuKszYVAigt+awri6HnoIli2DBx6IuxWd8bzjGAO77irWN7/Ctt41OW7WCRPElfrqq/D00/Ddd3DyyYnNyh1LW/fu4kLt0SO4e4DjInW+s0wJhFTuv6aOmQn3aGuLoZISEeupXNBZbmUDtbQpiqJknsaUuHBcqcmK2wbVNUtFVZWIhR49Gj6YKipEUDitn5y1O22cystFlDhzz5oFCxaIOHMamLvXceCBMHy4CKqjjw5+EKaKy/N76Bsj2ZZTp0ox2r32aijW3FYsv4bxyb6HSCQurpwkg0MOSTymsDBec865Nm/bLD+crNBMFS/u6KQjCp0WWw5BfzetLS5DkP2yUlEUpT2QqbpkbsKKtjVrRBQ6PRa957mzW92WquJicb917hy3Ut11F2y+uQgyB/e1GSMFbhcuhDffDF5TZWVicoafhcw9JohIevxxuY6LL07c5xVtQVanZN9DfX1qi1+yOL9kOFYed8FZN9kaAJ+tMW2ZsOS1scxRUNGmKIrSdKxNtKB4RYx3mx+NeWCk22/TabAd5jxvHJkxUmLjq6+k5IW71IW3efnYsbDpphLQHxQ/Bum7d62Fxx4Td6dTEsOxornbRLnX7T4mFdFo8vU6Y/nNkQpvuQ9nnL59JZnDr6RKa9EW3KPNMWY6GbythIo2RVGUprJ6tSQGOJYjP/HQHJY2p69nOuIt6NigmDCH6mopKLvddnDkkYn71q1LtFLl50v7pW+/lfizINzWpWRdDJz3b74JixZJEoTf/uXL4+I5XRHsZF86oi3o+0gnsN1NkHjPy5N4q2wlW12GTRGFzv123Nrduolg87pRsxAVbYqiKE3FsTQ5liNvAL9jCUomrsIIL+8xdXWwYgWUliY/zy0ugtbhbZbu5fLLYelSuO46f7GyalWiaBo9Wl533y3Zpm6ch6afkE32AP7nP0Vc/frXidv9LCReS1sqvMIp6JycHCmr4dcUPRle0ZatYgjaRsmPdL9fNz17Spxmly7yubhY6vGpe1RRFKUDEWRNcR4GTS2K6xVbjkBKFdzufqgFWfyczEnv8QCPPgp/+Qv87ncwYoT/+Y7Fz+2CvfFGeX/ttYnH+lkfU1narIWXXoJ99mnYEcDPUuXXNzQZXvdksuO7dJEHfzpiIVX7pGylLRTXbcy5nTu3CZHmpe2tWFEUJVuprRWrm1uMFBXFBUFVlZT08Cu0m6oYrRvHQhAWb3eCsJY2a+Gqq8QdOXIkXHFFw/OC2jhZCwMHwgUXSMbpnDnxY7zxXW6CHsaffSalRA46qOE+v4evV7SlCvT37m8LrZzaCtmaiNAGUdGmKIqSKSIRcRM6lq+uXUVgOcJm3ToRdmvWNDw3HfeoXwHXsHFtbqHkrp7vZ2m79lq49VY44wz417/85/UKJq818Xe/k/ZQF10kSQwgAikSkZZXEyZIl4H33ks8z/tQfukleb///v77k63LmNRxY97raG7Rls1CI9OJCM1BG7SSZYKOedWKoigtgVMqw2vFsVZEy7x5cQHntbTV1kppCz8LnN8DK1miQ5Clrbg4HoDvjXWbPBluuklE1YMP+gs2aHht3oSG/HyYNk2OGzMGXnlFLI6nnALnnCNxapdeCvvuS/G0af4lP0AE3vDhsNFGDdcQ5gGeSkSEzTDNFNks2txkopyGWtoyhoo2RVGU5iJIgNTUwKGHSnHYX/1KGpt747tWrpRaZu7OCskSBdIRbX5rdIu5sjL4wx/g4IMlls0ripJlUHpFG8DWW0u5kP79xWq30Ubw1ltixfvf/6S7wXHHUTJlijR6X7gwccx//xs++UQ6MfjN65cQ4GwLKqXhdlv7jdkclra2IjTaQskPkN8jPxHfjlHRpiiK0lx4a505TJkifTqvvVbEwxFHiOvUIZnI8hsv6DjvWM5xbvHnjl9ztk+e3LA5OkiGXbduia5GP/eoX4bqJpuIle3BB+H//k+K5J56qqyne3eYNo2KM86A2bNhm23Erbp8uZx7111Sy+zkkyUBoFOnRNeu3/3IzZXCwH36xLf16hV/36NHQ+thS9YWy2bRFtRzNh1a4vpyc7O3KHEzoaJNURSlOQgqwlpRIaLoqKPg+uvhqaekJMbkyf7j+LlH/R6I3gK3fuc78/utMxKRBInly6W22kknSU02N506iUs1VYFZd4Fa97EFBXDYYSLcnNg055hOnag480z48EPpqDBzplj6rrlGEhkuvVQEbmGhCDGvlcyvlZQ3a9Qr0rp2lTU5/ULVpSekqikYhua+lx0UFW2KoihNIUwfQ/cD7JlnxO152WXyeY894PjjxQ25dKlscwfzr1+fvByGQ7p9Ld2dDkDWtHYt/OlPMs5114Ubx8/S4V5zUGan9xqc4/v1E8vam2+KNezhhyVz9cILk6+jW7fUazVGOhA4LrWcHLG+OSVE1NImpNtpw49svr42jIo2RVGUphDmAecIF2thxgyxYO22W3z/zTeLdeO22+SzuycnSOC+n0vTS5C1zQ+/eLtFi+Dvf4fTTpOMz1TnQqIoKy6OryOZaDMm0W3pdw+HDhVX6QcfwBtvBCdCuMcMIxTy8oJdamppEzLhcvT+p0DJCCraFEVRmkKQK8nvAf3RR/DNNyKK3EJlk00kAP+ZZ+DTTxuOGVaM+RXvTSUq3eu8915xgV5+efhz3EInP1/2uePmgjI7c3LiFq6gbgydOsGAAeEf/E0VCGppEzp3Ftdx375NG2fjjcVyqmQMFW2KoihNIR336MyZImwOOyxRmNXXw7nnSmzV9dc3FG3u/qJBGangX7Q3Fc44a9bA88/DscdKlmdYnGDwnByJD0uWkRk0d6p9KtpaHnd9wcbS0mVUOgAq2hRFUZpCGNEGIsRefRVGjZIHovu8+nqxbFx6Kbz/Prz8csNzU40PUhbj5JPhrLPihWyDLF5e8ffMM5JAcPLJ4Wqaud87cWLezgOpHtp+matNIdszCduSaFOyEhVtiqIoTSGsaJs/X+qRHXpow/Mct+aJJ8JWW4m1ra4unhHp163AGNn23Xfw+uuSZTl6tNRDmzFDenR+9118jpyceMyZd53Wwt/+BsOGSbxduoLCLc68cWHusUpKEl1ubtHm3dYYgdNU0dbcVfZVtClNREWboihKUwgr2mbPFneT0zvTT7Tl50tR2++/h8ceS+zR6Z3nuedg331FnI0fD9Onw+GHww8/iMWtrk7qv33xRfwcvwxLY+A//xGBd/LJ/msPe83QUPi4P3sL2iaztDVG1DRVdBUXi4u3R4+mjROEijaliahoUxSl/RGJZKbWVNi5/HALCGvF5XnAAfHAbPf6nPc5OXDggTBihJS9WLu2oTXKGLj/fnGBFhXBLbfAu++K0Lv/fhEcgwfDI4/ItgMPhF9+aSgSnM+5uWJlKykRkefeF4RfBwK/607VBSDTlramCiGnBIjT2ivTqGhTmoiKNkVR2hfRKKxYIUKluamrk/6gfrhLVHzzjQioI4+Mi5r6+nhWqCNacnPlYT5pkiQG3HZb3H352Wfw5JNSePbcc6Xw7AsvwLhxsNNOcaHhCMADD5T969ZJTTivJctZx+rVIiiPOSY+RipBUVgocXl+lrtkos1LMktbY1yd2d5EXEWb0kSamBqiKIqSZTjV+FvC0uau/O/FLTpee01+Hnxw/GFdUSGvPn0aJgtsv710JJg8WfqALlok/TdBrFyXXNKw2Ky7FpzD0KFwxRUSI/f003D22fF9jrXs0UdFfJ5ySnxfGEHhbiPltw5nnMZa2hqTuZjtQki7BChNREWboijtC7e70trmfTiGje167TVxWW6+eWKPURCLm9s96nDTTfEEgb594cYbYe+9pRzH1ltDaWm8xIdbHHnXNGGC9Py88kqx9PXpIx0LSkrkXk2dCrvvLkLREaFNuWfJrF1B+/zuo+OqTGctTS1R0dyopU1pIlluS1YURUkTt4UtE2UkkhFGtNXVwdy58QQEP2uTX1mO/HyYNg0WL4aPP5Ym61tvHe+T6R3HOde5frfL9Z57ZB0TJ4qFrXt3Of+OO2DhQik10rt3WpceiNealOwe+blH3ddVUJC6E4KbggJx2WbqWjKNWtqUJqKiTVGU9oVfgH9zEUa0ffqpWLb22Uc++9Vvc7b7PcgLC1PHaiWztIFY+CZNEovbtGmybf58uPZaKaY7ZkzD8RqLV5g4GaN+7tRka24sxcXpCb2WJKiThKKEJMttyYqiKGnSkqItaPySkvhDed48+TlihPz0CiLHnesnlHJzG2anBnVEcD5HIhIr5z1vwgT4178k6/R//xMB17s3TJkSP7d3bzmvKfXOvC7AnJzgdkjuY/0sbe2NTp3inSNUtCmNQEWboijti9Z2j3btKpmVDu+9J/0zN9lEPnsf1u54NvfaHddgmGtwxBE0jJlzyMmR2m4TJ8Ldd4sF7MUXJW7MIRMWqnREV3NY2rKZ3FzpHNGehanSrLSI1DfGPGyM+cUY87lrW09jzGxjzMLYzx6x7cYYc58x5ltjzKfGmJ1d54yLHb/QGDOuJdauKEobwy0AmlsMOCIrL0+sSd26New6MG+eWNmCLGRuS5vbwuWIKW9ZjVSWtiCMEQvgE0/AkiVSFuWAA5Kf09xkuo1VW0AFm9IEWso+Ox04xLPtCmCOtXYwMCf2GeDXwODYayIwBUTkAZOA3YHdgEmO0FMURdlASz78nbm6dxfhVlyc+FBetgx+/BH23DO+LUi05eRI26quXcUa4xDG+pVuY+7+/UXANTepvotkJT8URWlAi4g2a+3bwGrP5iOAGbH3M4AjXdsftcL7QHdjzMbAwcBsa+1qa+0aYDYNhaCiKB2dlrS0pYrDeu89+enEs/kd67a0GSOuVbfFLVWj96Bx2wId0dKmKE2gNSMhN7LWLou9Xw44/7UcAPzsOm5xbFvQdkVRFH9ayj0aFFQ+b55YyoYNi28LimlLJroKCpKvI4ylrTVEXaq6aWppU5S0yIpEBGutNcZk7F9XY8xExLXKgAEDWLp0aaaG9qW0tLRZx++I6D3NPB3lnuaUlm6wXtmqKmwz9ZEsLS0lx1qIRolGo77Crfdbb8EOO7By1arENS5f3uBY27kztqrKdy6zfj1m7Vo5rqgIW1WFKS/HxI6P1tVBfT05QS21AFtYGDh+xqmtxVRVYaNRqKwMPi4Ske8rNxfbqROrV63CLlvWbN9ZR6Oj/M23JK19T1tTtK0wxmxsrV0Wc386jQKXAJu4jhsY27YEGOXZPtdvYGvtVGAqwPDhw23//v0zu3IfWmKOjobe08zTIe6pOwvTLzEgGY6bLmQ5hn7Om403bmghqqmRfqHnnBPuvpeUBMeZrV8v8W4g19Otm3xev1629e0rnRWSiZ3OnSX2LpuIRsUVHCuDYaqr6de/f/M1bO+AdIi/+RamNe9pa7pHZwJOBug44AXX9lNjWaR7AOUxN+prwGhjTI9YAsLo2DZFURR/0nWPrlgBy5eHq+/mjmfzc+l9/LEIt732arjPyTR1k27drvYU0xaNQktZARWlDdMiljZjzBOIlay3MWYxkgX6R+AfxpgJwI/A2NjhrwC/Ab4F1gPjAay1q40xNwIfxo67wVrrTW5QFKWj05Q4Nkes1dWljiNz+poGiaV33pGf7sxRh7w8sTC53ZnJCtr69az0bmuLos2P9nIditIMtIhos9aeELCrQZEga60Fzg4Y52Hg4QwuTVGU9kaY7NFIRCxbQQIhjPBLdcy770qTeHf5DjdBvUPDHOvdlq2JCKnIxjUpShajfTQURWn7rF3r716zVvZVVEBtrXyuqRE36Jo1mZnbT3hEo5I56ucaDTo3mWjzazTu1y6qLdIe3LyK0kJkRfaooihKo6mtFVEG0lzdTTQaD9YHKb/huCGrq4PHTMfS5icyPvsMVq6EffdNPoYx8XHCijb3ue73XmHnvYZsFUPetWq9NkUJpI3+10xRFCVGsge+N6GgtjY4ySCMcPA7xk8Mvf66/DzoIP9xHJpiaUs2VrYKtDCESQJRlA6KWtoURWk/OBY3B6fbgJswgsxPONTUwKpVUjqjqgpTWSklNPwE0uzZMHSoNIpPht/6/HCLtjCWqNzctiN+2rLAVJQWRi1tiqK0bdwPfa9oq69veHwYS5sfq2PJ6uvXg7UYx73qFR2VlfD226mtbI0lmRjr0kUyU3v0kB6m7ozUbBVHXguh1mhTlEDU0qYoStsmmdhKJz4qlXs0aCyvGHr1VbHKHX546jm7dxeR17Nn+HUma3vVtau8QARcJJK8I0E24LqOaI8e2SsuFSULUNGmKEr2U1MDnTr5x31lKnDdK9oiEXnl58s2d6cFN16R8c9/Qu/eMHJk6jk7d453OgiLs4aiIsmY9SZfJFtbNtJeYvEUpQVQ96iiKNlNdbXEkqXT868xD3+v+FuxQjJA6+rCj1ldDS+9BEcdlbpZero47bi6dImvp1ev8G262oIgagtrVJRWRC1tiqJkNzU18jMoaN/P0paTk97x3u3u97W1YuULwm39e/11ias7+ujg4xtLt27Sm7St1mMLQi1tihKadvbXryhKuyNVFmSQaEuXMGLOD/dcTz8tcWr77Zf+/GFousmWnQAAIABJREFUan/SbERFm6KERkWboijZTWNi1prqHvUr/5EqEWHVKnjmGRg7Nh4Hl01kqyBS0aYooVHRpihK5qivlyboTa0RVl8Py5aJqzHVg9xPTCU7x1rpC3rBBXDiifCf/4QbP0i0OdavRx6RmLazfVsnK2FQ0aYoSdGYNkVRMsfq1SK46uslSL6xrFsX7xtaUBDfbm3DB3sqS5y3TdK8eWIN695dtj/9tGR8Hnhg+DHdOFmlU6bA3nvDDjuEP7e5USuWorQr1NKmKErmcIrZOskDjcVtqXOPFbZ+WlB7KGvhmmtgs83EwvbDD7DddnDWWSIU/eZPZWkzRmqzff+9Wtkag/YaVZTQqGhTFCVzuCvwh6G+Hn75pWHz9kw8yHv0ECtdSUl820cfwZdfirjq0kUyMh94AJYuhcmT/ee3NvFzly6J1j+AG26ATTeFMWOavu6Ohoo2RQmNijZFUVoPx526Zk3i9nQyOYMsbUVF4qJ1C8np00V0HXVU/NzNN4dDDoE//UmSCcDf0uaM27VrovXu2Wfhgw/EgpdtCQhtwT3aVnqkKkoWoKJNUZTWw683KDS+/ZQfjmhbuVIK344dG+9C4FjRrrhCeoree2/DMaur4+t0hI/zc948OOMM2GUXGDcu/JqVOCraFCU0KtoURck+mvog94tp+8c/RHydemp8X22t/Bw8GI47DmbMgM8+azj/ypWJ45aV0W3SJBGAm24KM2cmL8DbWrQFS5vT0SFsZwdF6cCoaFMUpfWxNt4uCoJFW2MSEZz3zz8Pw4aJQHMoK4u/v/xy2GgjOOEEmD/ff46nnpLCuYMHU/jqq3D66fDWW9C/f/C1tSbZKtTcdOkCffpIfKGiKElR0aYoSnbg9BYNaj8VRJiSH2vXwhdfwJFHBh/Xt6+U/iguljZU//53fF9VFVx9NZx/vqzz7LNZ9fjjcN11TStt0ty0BdEG2WmlVJQsROu0KYqSPdTVJRdhySxtTr00aChWXntNfh50UPL5t9hC4t6OOQaOPx422USyUL//Xgr9nnkm3H8/VFVR/803/nNlE23BPaooSmhUtCmK0np4C9+6rWy5uelZ3ZKJkldeEbfoZpslHyM3V1x1zz0nMXCffiqZrWPGwGGHwahRcow7ezSbxVA2r01RlLRR0aYoSvbgFnD5+eKWDHtOkFWpshLmzpX4s1Q4YqxnT7GqFRVJ8oI73g4gr43806mWNkVpV2hMm6Io2YO1iS5Ob7HeZO7RIFHyxhuSJTp6dOr5czz/JHrX4MyRny+voqLUY7YmQZ0hFEVpk+hfsaIozUNjK927Y9TSsQ4FWZVeeUUyFEeMSG8M57OfaAOiPXtKvFs2476edLtVKIqSdahoUxQlc6Qr1LzHu1tG+Qm2VIkIfvteeUWawRcWpl6PMfHCu875bldoW3YxtuW1K4oCqGhTFKW5CBJYK1ZIcH/QfrdoCyM0ksW0ffUV/PQT/OY3wWO5i7oaA927J35uyxaq3Fwpp5HtblxFUULRRqJpFUVpE3gbrXuprZWM0Koq/2KqqSxtyfATbTNnys9f/zrceQ5du0JNjTSbdxf6bYvWqj59WnsFiqJkCLW0KYrScqQSPaksbekkIlgLjz0msWwDBwbP7Sf2unSRork5OYExbYqiKC2NijZFUTJHKkub37G5uWLR8m4PEkjWwk03wfDhcNdd/gV1jYEFC6QLgrvXaCr85nRva2xyhaIoSgZQ0aYoSmbwSypIdoxbbDlJBF5Lm3eMaBQuuQSuuQYWL5b3t94q+7yJCH/+MxQUSFN3ZzyHnByJ8+rZM5z1zGmzpLFhiqK0IiraFEVpHhzBFY3C+vX+AszBEU5e0eZtHH/LLXD33XDuubBsGUycCJMni8XNLb6+/hpmzIAzzhBh5p4DRLT16CEZpWEK0PbsCb17iwhUFEVpJTQRQVGUzBDkOiwvl8SDmppES5Xb0hYk2txj/vOfcPPNMG4c3Huv7H/gAclEvftuSXA491zpXnDKKZLocPXVqdcdxtKWm9u2s0gVRWkXqGhTFKV5cASX04qqqiqxVloY0eYcM38+XHwxjBwJU6bAypUyVkkJ/OlP0p3g3nvh4YelztrKlfDii9CvX3y+IIuatnpSFKWNoO5RRVEyQ6qYNq/lzC+BwCvaOnWCigpxg/btC48/LmVD6upg3TrpCwpw220wa5aIuq23htdfT17mw7suv/eKoihZRqtb2owxFwKnAxb4DBgPbAw8CfQCPgZOsdbWGmMKgEeBXYBVwHHW2kWtsW5FUTyEyawMyi71y9A0RuLObroJli6Vmms9e4qb1cERbcZI14Ptt5fPvXs3nFvFmaIobZxWtbQZYwYA5wHDrbXbAbnA8cBtwD3W2q2ANcCE2CkTgDWx7ffEjlMUpbmorRVLV2NYvz4uqiDRiuZ8huTu0W++kSzQU06BXXbxX59Dbq50N8jLS2w95ZBOnTZFUZQsJLRoM8Zsa4zZKPa+izHmemPMJGNM51TnpiAPKDLG5AGdgWXA/sAzsf0zgCNj74+IfSa2/wBj9F9ZRWk2Vq6EtWuhujr1sV5LW3U1/PJL8DFhRNuVV0qh2+uui+93Z5Q663LO79ZN3Kh+fUjdeDNJ/bYriqJkGem4R58AxgIrgDuBXwHVwF+AUxozubV2iTHmTuAnoAp4HXGHlllrnf+iLwYGxN4PAH6OnVtvjClHXKgr3eMaYyYCEwEGDBjA0qVLG7O80JSWljbr+B0RvaeZpzH3NGf5cgBsVRU2VY2y2lpyVq9OeogtL8fEEhNsQQGmpgZbWIjt0oWclSvFQhazzpmff6bviy9Scd55VNTXY5Yvx1ZUYKqrE92i1oIxRFO5Z13rs/n5WMfNGomQE7s30fp6fyudD/o7mnn0nmYWvZ+Zp7XvaTqibZC19uuYZWsMsC0itH5o7OTGmB6I9WxzoAx4GjikseM5WGunAlMBhg8fbvv379/UIVPSEnN0NPSeZp607qlbBPXokbqwbG2tZHEmo7Awbh0rKJD4tM6dJQs0L0+sXtGoiLF77gFjKLnoIkp69hSLH4jlzUtOTmKmaKr1FRbG67dZGy/n0bdvaNEG+jvaHOg9zSx6PzNPa97TdGLaqo0xJcBuwE/W2pVADVCY/LSkHAj8YK0ttdbWAc8CewHdY+5SgIHAktj7JcAmALH93ZCEBEVRMo3bDZmsMG7QManGjETkp9s96eyvqIC//AWOPRY23TQzbkuNaVMUpY2Tjmj7O/AGElM2PbZtZ5pgaUPconsYYzrHLHgHAF8CbwLHxI4ZB7wQez8z9pnY/jes1WaAitIsBGV61tbC8uVSNDfo+CDcos1xceblNRRLjzwilrWLL5bPqcRUGLEVFMcGYnXr1k0L6CqKktWE9gNYay80xowG6qy1b8Y2R4ELGzu5tfY/xphngP8C9cB8xK35MvCkMeam2La/xk75K/A3Y8y3wGok01RRlObAW1Nt7VoRNU7GZmWlCJ3GjumQmxsXVNEovPqqFMo94gjYdVfZnmkLmHe8wqY4DBRFUVqGtOq0WWtf93z+qKkLsNZOAiZ5Nn+PuGG9x1YDxzZ1TkVR0qSqKm4Zc8et1dTIvq5dw1naVq2C6dOl7lq/fjB8OHz/PcydK5a7Zcvg559h8GDpKerglw3ao4e0sIL0LW3qBlUUpQ2SVLQZY/6NFL1NirV2n4ytSFGU7MAtwtz11urq4u9XxUJKne4FyVi/Ho47Dj7/HHr1ip8LsNNOkpCwzTZw6aUSyxbUgsr57BaPKtoURekApLK0TXO93xL4PySm7UdgUyS+7OHmWZqiKK1KkOXMb3s0Gtzo3TnnvPPgiy8kXm30aHG3zpsHe+wB224rVjbnPG9smZ+lzX1MuiJMRZuiKG2QpKLNWusUssUY8z5wsLX2C9e2vyOizeveVBSlo+EIrs6xett1dfH4t2nTJFbt2mtFsIG4VA85BDbeWD67xZ7bmgcNRZxXdPlls3pRS5uiKG2cdLJHtwG+82z7ARiSueUoipI1pJOY7RVE3brFg/sXLJD+oaNHS+P3oHPdY5SUBI/v9zmMaEt2vqIoShsgHdH2FjDdGDPYGFNkjNkayeb8d/MsTVGUViVd0eZ2j4JY2aZOlb6hffvC3XcnF1/u9519uuN17drw2IKCxJ+p1uiQRgFdRVGUbCEd0XZa7OcXQAXwGWCA8Rlek6Io2UBTRNu8eTBiBFx/PQwZAo89JtmeXoJ6hPpt9xNyPXqImAtbeqSkRDoqpOrcoCiKkoWE+u+mMSYXuAARbicCfYBSa22aPglFUdoF7nZUIGU/nA4HDzwAV10FAweKWNtvP4lJc/YH4RaJfu5LP6tcTo5/W6sgvG5XRVGUNkQoS5u1NgL8Hqi11kattStUsClKOyeZpc1b3sMRZC+8AFdcIYVxP/pIBBsEW9TcQiyVqAs6T1EUpYOQjnv0UeDM5lqIoihZRjLR5ifC3n0Xzj8f9t4bnngi0WUZRrSlc6yKNkVROiDpROPuBpxrjLkM+BlX0V0trqsoHQyvsFqyBCZMgK22gpkzJTHAXbYjjBArKhI3q7owFUVRfElHtD0UeymK0hFIZmnLzRVLWnm5HHf55SLSnnsunnDgFmpBjdjdoq1bNygu1iQBRVGUANJpGD8j9VGKorQpnJ6iftatVDFt+fkixh56CN58UzJFt9wyfkyYEhvuY3Jywgu2dDJbFUVR2glpFSsyxmyEuEl7I+U+ALDWaisrRWmLOA3Xi4rC1S7r2lUEmyO2KipErA0bBuPHJ1rXvILML4O0sbFpKtoURemAhBZtxpgjgceAhcBQpF7bdsA7aP9RRWnb+Ikgv22dOycKs7vuksbvM2aIKAtKFjBGRZuiKEoTSSd79CZgvLV2GFAZ+zkR+LhZVqYoSvOSSvj47XeLrPXrYcoU+M1vxNLm3Z/s3GTbkuFYA70lRxRFUToA6Yi2Ta21T3u2zQBOzeB6FEVpKdz9Or0CraJCRJkXt8h68klJRJgwIdx8fhmk6Yq2Xr0k/i5sBwRFUZR2RDqi7ZdYTBvAImPMCGBLICAtTFGUrCZItEWjsHZt8nOtlc4H224Lu+8u25IJsLy8zNRWy80V0RZUQkRRFKUdk86/fA8Be8fe3wO8CXwC/DnTi1IUpQVwCzX3+5qa1Od++CH8979w1llxMeYnyvr1k2bxYUp+KIqiKElJp+THba73jxpj5gLF1tqvmmNhiqJkmEgEKislmcDd4B2gtla2FRRICZBUTJkiNdVOPjnuRg3qbuBYxQoLG7pcVbQpiqKEJrSlzRhzuDGmu/PZWvuTCjZFaTvkrFolMWiVlbLBLdoqKiQLNBJJnaCwapXEs51ySnqxZYWF0Lt3vPguqGhTFEVJg3Tco5cAS4wxC4wx9xpjxhhjejfXwhRFyTBODFt9vXQv8LOoRaOJsW5+TJ8O1dXiGnUTRoDl5ydmfqpoUxRFCU1o0RbrL9oLuABYDZyNJCR83kxrUxSlObAWSkuDkw28ljbHvWmMWOL+/GfYay/YYYemN3FX0aYoihKatDoiIJmi+UABUAiUAeoiVZS2RDJLmrUNRVvPnuJS7dIFnn0Wvv8e7rij8fM3VegpiqJ0UNLpiPABsDHwLjAXOMNa+2UzrUtRlEziLemR7DivaMvLkzi0SARuuQW22gqOOKLxa1HRpiiK0ijSsbSVA5sCPWKv7saYPGttiFQzRVFaFbdQSyXavPsd9+iUKbBggSQh+JXwCNtayl1jLVX8nKIoirKBdGLaDgIGAtcC9cAVwGJjzL+aaW2KomSKoJpsfsf57X/nHbjkEhg9GsaO9T83HatZSYlY8AoKwp+jKIrSwUk3pq0r4iIdCGwGdAeKMr0oRVEyTGPdowUFsGgRHH44bLYZPP54ZlyaJSXyUhRFUUKTTkzbp8BWwIfA28DFwDxrrU+DQkVRsoqwrku3aOvXTwTaMcdIeZBXX5U6a4qiKEqrkI6l7TzgfWttdXMtRlGUZiId0eZY4oyBmTNh7lwp87HFFsnP1aQCRVGUZiWdmLa5QLEx5hRjzGUAxpj+xpiBzbU4RemQlJfLK5OEFW1uwVZfD5dfDkOGwBlnZHY9iqIoStqk08ZqX+Br4CT+v707j6+rrvM//vqkSZoutKEtFJoia9VBkMUCFTcWZYeOzoDwQ1BZiiP4A0GHisOMIAzgCI5oBQqWFn4MILjVglAQFFBAdlDWUmW6AE2bpm3SZv/8/viewz25uTe5t725S/J+Ph555N5zzj33my+35J3vChdHh6cB1w1CuUSGp46OsCZaa2vuQSsXud6ruzt8N4ObboLXXoOrrgqTBkREpKTy2cbqv4HPu/sRhNmjAE8C+xe8VCLDVRyaoLChLd/3b22F73wHPvlJOPbY4pdDRET6yOfP553c/XfR4/i3SUee9xCRUsg1AHZ0hO/XXw+rVsGiRbmPVavK529AERHJVz7/l33ZzA5PO/Zp4KUClkdkeMt1PbXB1NoaukZnzoT99hv4+gkTwkbw48cPftlERIaxfELbBcBtZrYAGGVmNwDzgW9uSQHMrN7M7jazV83sFTP7qJlNMLMHzOyN6PvW0bVmZtea2RIze9HM9t2S9xYpO4MV1JL3raoK+4hmc/vt0NwMs2fndu+6urAUSKZdEkREpGDymT36BPBh4K/APGApcCLwr1tYhh8C97n7B4G9CBvQzwZ+5+7TgN9FzwGOJEx+mAbMQpMgZKgZ7Ja2MWPC+ms1Ndnf/5ZbYPp0mDGj8O8vIiKbbcDQZmajzey7ZvYb4BvA9cA1wAzgHmDV5r65mY0HPgn8FMDdO9y9GZgJLIguWwD8Y/R4JnCLB08Q9j/dfnPfX6TsDHZLWzw+Lds4taefhjffhFNOGZxyiIjIZstlEsEcYB/gfkJL157ABwldo2e6++oteP+dgUbgZjPbC3gGOBeY7O5vR9e8A0yOHjcAyxKvXx4deztxDDObRWiJo6GhgZUrV25BEQfW2Ng4qPcfjoZrnVpLC9bSAkBPZ2f2FrHNuG/TmjX4mDF4Swu0tVHV3Nz7otpaxs+bx8hRo2jcf3+6B/nfTaUbrp/RwaQ6LSzVZ+GVuk5zCW2HA3u7+yoz+xHwv8BB7v5Igd5/X+Br7v6kmf2QVFcoAO7uZpZX84O7zwXmAkyfPt2nTJlSgKL2rxjvMdwMyzpdvx6i0MakSWGAfyFs2IC1tLDdlClhz8/29jAWDVITCLq7YfFi+NznmLzzzjB5cvb7CTBMP6ODTHVaWKrPwitlneYypm2su68CcPflQEuBAhuElrLl7v5k9PxuQoh7N+72jL7HXbArgB0Sr58aHRMZGooxEQFSi+WawejRYazbokXQ1gYnnQT19YNTDhER2Wy5tLRVm9nBwHuDYNKfu/tDm/Pm7v6OmS0zsw+4+2vAocDL0dcXgSuj77+OXrIQOMfM7gAOANYlulFFKt9gT0SIx7KNGBFa0sxSx+67DyZOhCOP1ExQEZEylEtoW0WYLRpbk/bcgQF2ku7X1whLidQSZqR+mdAC+DMzOx14CzghuvZe4ChgCbAxulZk6ChWSxv0DmY9PXD//XD44QpsIiJlasDQ5u47DWYB3P15YHqGU4dmuNaBswezPCJlwz2EqXj8WbYZnz094SuX/UGz3eO558IOCEccsfnlFRGRQaV9Z0TKSXqL2Nq14SuenJBJY2MIXF1d2a9JX/Ij3X33he+HHZZ7WUVEpKgU2kTKSfqYtvb28HjTpuyviTd57+zc/Pe97z74yEc0Y1REpIwptImUg56e8D3bRIRsLWRxYOvvmuS9Ml3T3AyPP66uURGRMqfQJlJqTU3wzjub11KW62zT/kLbgw+G8KfQJiJS1hTaRErJPayNBiG05dvSFrfQpT/O9D7Z7nPPPWGBXe01KiJS1nJZ8kNEBksymPX05L7kR3t76NZM7piwOS1t7e3wq1/BccflNvtURERKRi1tIqWUbB3r7s7e0pYeyNrawvXJCQrJa5qbw5ZY6efSQ9vixeHak07avPKLiEjR6E9rkVLqr6Ut34V2e3qgoyO8buPGcGzcuN73Sg9tt98edkH49Kfzey8RESk6hTaRQunoCMEp3og9F/2FtHwDXE8PrF7d9x5mmUPb+vWwcCGcfDLU1OReZhERKQl1j4oUyurVYSZopgkBnZ1hEdx43bVY8lr3vi1vme7T1JR5pmmm6+P7ZQptt94Kra1wxhmZfx4RESkrCm0ihZDcjSBTeGpuDkFrzZrex9Nb0wYa07Z2bRjP1tHR9z3yCW3NzXDZZfDRj8J++2X+mUREpKyoe1SkEJKhLVNXZrbuzf5a1tJb4aD3YrrpMp3r6QkbwKe//+zZYeure+7Jfj8RESkramkTKYRMASupKss/tfTZo0mZWtqy3Sf9Xslj6eu9PfYY3HADnHce7Ltv9vuJiEhZUWgTKYT+FraF3EJbetjLdM/+tqrKpLW1d9fo8uVh4sH73geXXJLfvUREpKQU2kQKIdlKlqmlLRm2BppskOlcvst/pN/HneoXXwzj15qa4Be/gLFjN/+eIiJSdBrTJlII+SzP0d2d2n0g19AW33ftWnjgATjgAGhoyP7a5maYOxf++Ed47TXo7GRSWxvsumvYa/RDH+q/jCIiUnYU2kQKYaDQlq11rb+JBemvX7QITj89LB1SUwP/9E/wiU/AIYekFtHt6YEXXoBzz4U334Tp0+GEE6C7m5a6OsZefjlMmJD/zyciIiWn0CZSaAOFtuTjTKEtuRguwKuvwoUXwtNPwwc/CNdcE2Z9LlwId9wRrt9333Cvt94KrXH19XD33WFJj5oa6OykZc0axiqwiYhULIU2kUJID2Xd3WESwOjRoSs0U2jr6Qlf8Xi35AzROMz97//CzJkwciRce23Y2L2mJrSufe97oRt0wQJ4/PGwtMdnPgP77w8HHwzbbRfuES/Em+8kBhERKSsKbSKFkB7KNm6ElpbwNWVK/2uujRjRe2mOOLR1dMBXvxqOLVoUujpXr07da8QI2HNPuPLKsOZaurFjw/vHb1ugH1VEREpDoU2kENJDW/q4tUwtbXFoq67uvS1VvDzIJZfAc8+FCQXve1/fHRMGkt6yppY2EZGKptAmUgj9TURID21xoEu2tCV3VKiqgl/+EubPh1mz4OijU/cdaDkRCCGwrq7veLkRI3L+cUREpPwotIkUQn97iHZ3Z+4ejY9VVfUOXq+9Bt/4RlhT7aKLwvmentA1mkl6aJswIQS3jo7QTZvtOhERqShaXFekEPoLbclWNIBNm8L3OLSZpQJVSwucemoYj3b99WHSQX9bV8WvT4qvr61NTUYQEZGKp9AmUmjpoa2jo/f5uNsyvaVtyRI4/viwvtp116UCV77dmskQV1UFW28NNTX46NH53UdERMqKQptIIaR3f2YKbXH4cg8TD9raUq/9j/+AT30K3ngD7rwTDjwwnKuqyr+lLf35qFGwzTYa0yYiUuEU2kS2VHK5Dugb2pJLedTWhsdr1oTX/fnPcNhhMGcOfOEL8MQTcOyxqdf2F7Tie4mIyLCgiQgiWyp9zFp/szzjCQLvvhvWYPvTn8LuBTfcAMcck7oulj5JYcyYMN5t48bwONN7iIjIkKTQJrKlkkt3xMt7ZAttI0aEvUOPPx6WLYMrroCvfS20um3YEK5Ldoemh7aRI8M9ttoqc1kU2kREhiyFNpEtFbe01dT0DW1xkIMQwNatg89/PgS2W2+Fz30uHE/sXNArtJn1DXH9UWgTERmyNKZNZEvFEwpqasL39NCWvO6EE2Dp0rBw7oEHpkJWciJDMnjFrXOxgSYTKLSJiAxZCm0iWypuaRs1KnzPFtouvDBs7H7ttfCJT/Ren23MmDDerb6+75i2XEJbXV34rmU9RESGLHWPimyJOKAlW8SSrWbxsWefDd2h3/gGHHdcOJbs6hwxArbdtu/9zcI4tnHj+p8tuvXWITxW65+0iMhQpZY2kS2RaVeDWDxbFODmm8Osz29/O3V+oPFpSWPH9h/azEL3rLpHRUSGrLIIbWY2wsyeM7NF0fOdzexJM1tiZneaWW10fGT0fEl0fqdSlluk1xpsye8QAlRtbViT7Te/CdtTJWd99hew4rCntdhERCRSFqENOBd4JfH8KuAH7r4bsBY4PTp+OrA2Ov6D6DqR0kluRQV9JxFUV8OvfhV2QDj77Nxngk6aFLaxUmgTEZFIyUObmU0FjgZuip4bcAhwd3TJAuAfo8czo+dE5w+NrhcpjWT3KPSeKGAWlvuYNw8OOgh23z337stctq8SEZFhpRxGLf838K9A3G80EWh293iZ+eVAQ/S4AVgG4O5dZrYuun518oZmNguYBdDQ0MDKlSsH9QdobGwc1PsPR5VSp7ZpE7ZuHT5qFN7ejjU3Y/ESILW11Nx3HxP//neaZs+mLfocVr3zDgBeV4fH1xZBpdRppVB9Fp7qtLBUn4VX6jotaWgzs2OAVe7+jJkdVKj7uvtcYC7A9OnTfcqUKYW6dVbFeI/hJu86jWdxDqSnBzZtCrMyt3S25YYNYamPrbYKX2PHphbKHTkybP6+3XZMOO201DputbVhK6uJE8M1RaTPaWGpPgtPdVpYqs/CK2Wdlrql7WPAcWZ2FFAHjAN+CNSbWXXU2jYVWBFdvwLYAVhuZtXAeGBN8YstZaejIwz4Hzs2+xZPsQ0boLU1BLZMy2zEOjvDllMA22+fORAmdzuA3t2jb70Fv/0tXHxxKrABTJgQlufQeDUREclDSQfNuPu33H2qu+8EnAg85O4nAw8D/xxd9kXg19HjhdFzovMPuWfa5FGGnY6O0NIW79850LWQWhR348awvVS6jRtTj7N1Y8Zj2uKwlgxt8+eHMDdrVu/XVFUpsImISN7KdaTzhcD5ZraEMGbtp9HxnwITo+PnA7NLVD4ZKtyhuTm0vLW39z6XnAh5OElGAAAYzklEQVSQ7W+DbC1tra2wYEHYW7ShIfNrRURE8lDq7tH3uPvvgd9Hj5cC+2e4pg04vqgFk8qQDFW5jm2D0AUa6+joPcYsec84nMV6esJ7xK116S1tP/sZrF8PX/96buUQEREZQNmENpEtkk9oS16b3HIqPZhlu66rC1atCq1r7r33B62qCi12P/kJ7LcfzJiR/88iIiKSgUKbDA3poS1uQUtOABjodclglv48blGD1Ji4+Hz6DNDbb4eVK+G667StlIiIFEy5jmkTyU96+GpsDF/x8XXroKmp7+uSrWttbfD226lJB8l7JrtR08e3Jce+tbXBj38cWtiOPnrzfhYREZEM1NImQ0O28WeNjaG1Kw5dHR0Dd4k2NcGUKb2vy/YYeoe2G24Iwe+223rPJBUREdlCCm0yNGQLbcluzfh5fxMMkjIFuvb2sDBvUhza1q6F734XDjkEDj4497KLiIjkQKFNhoZ8glgu17r3Xs6jpyd8ZepijVvULr88nL/66vzKLiIikgONaZOhob8JBUnpIS1TaDPrvWhu3JKW3moXGzkSHn0UfvhDOO002Hvv3MstIiKSI7W0ydCwuV2e2QJeHNBWrYK5c+H3vw+7GIwbBzvuCJMnw3PPwQsvhL1HV6yAnXaC739/S34KERGRrBTaZGjY3NAWi7tAm5vhllvg+efDpIVHHw2vOfDAEM7eeSeEtXXrQkg78sjwukmT4KKLoL6+oD+WiIhITKFNhoZcQ1umLs6uLrj/fvjDH+CXvwx7jn7wg2HG6Zlnhr1Dt902hLZ4EkJrK+yyi2aIiohI0Si0ydCQvrhuLjo7w6buc+eGxXBHj4bDDoOvfhX22iuEufr60OK2cWPvMDhmjAKbiIgUlUKbDA25BrXYX/4C554Lr74auj7nzIFPfSoEs46O3nuKxhMRkqFt3LjClFtERCRHCm1SGdxh9eowUzNTYMontL35Jpx4YrjXvHlwzDGwzTbhXGdnWJA3lim0jR0bvkRERIpIoU0qQ1tbCFSdnX1DWz6BrbERTj45BLGf/zxMJkjuD1pTA9XVvVva0vcPrdJKOSIiUnwKbVIZ0sesJYNUrqGtuTkEttWr4Z57QmCD/kOYWd/zCm0iIlIC+u0jlSE5nizT9lIDaWuDU06B118PM0SnT0+dS29JS1ed9reNQpuIiJSAfvtIZehvw/ZcQtv3vw/PPgt33gmHH947qA0U2tJniaaHOBERkSJQaJPKkAxq6SEtU2hLBrEnn4TrroMzzoDPfrbv+fSWs/r6cGzrrVPnx49PnddSHyIiUgJqMpDKkG9oGzEiTCZYtSqsu7bjjr23mOqvpa22FrbbrvexeF02s4Fb5kRERAaBQptUhlxCW7wVVfx41Cj4ylfCllOLF/duLeuvpS2burr8yy0iIlIg6h6V8tXRAevXhxazXENbbMMGOP740DU6fz58/OO9X5MMberuFBGRCqCWNilfa9eGmaItLb2PZ5uIEIe2l18OXaJ//3sIbCec0PfeCm0iIlJhFNqkfGXb+N09BLeurtCi1t6eOnfDDXDFFWESwQMPhK2pMkm2ymk2qIiIVAD9tpLy1N2dvQWssxPeeaf3sbY2+Jd/gYULw6bv118PO++c/f41NbDVVmHSgSYWiIhIBVBok/IUh7aamrAER1NTqoWtra33tevXw2mnweOPw8UXw1ln5bah+1ZbDU7ZRUREBoFCm5Qli7tGq6tDcJs8OYS1pqbeF27aBKeeGhbOvemmsBF8a2tYokNERGQIUWiT8hRv2J4cb5ZpaY5/+zd46qmweO7/+T9hmQ8FNhERGYK05IeUp7ilLTmuLX3ywB//CHfcAeecA8cdpz1BRURkSFNLm5Qli5f1SIa25OOaGrjsMpg2DS66KEwoqK0tbiFFRESKSKFNylOmlrbkLM/774cXX4Rbb4Xtty9u2UREREpA/UlSnjK1tAGMHh3C21VXwW67hYkHIiIiw4Ba2qT89PSE5T2qqvquoVZfD489Bs8/DzffrIVxRURk2FBLm5SfuGs008QCd7j00rBw7sknF7dcIiIiJaRmCik/2bpGIYxle+opuPHGMBlBRERkmChpS5uZ7WBmD5vZy2b2VzM7Nzo+wcweMLM3ou9bR8fNzK41syVm9qKZ7VvK8ssgiHc9gL4tbe5wySWw445hQV0REZFhpNTdo13ABe6+OzADONvMdgdmA79z92nA76LnAEcC06KvWcB1xS+yDJrVq+Hdd8PeotB3PNuDD8ITT8Ds2VreQ0REhp2ShjZ3f9vdn40ebwBeARqAmcCC6LIFwD9Gj2cCt3jwBFBvZlrvYSjo6YGOjvC9pSUcS7a0xa1sU6fCl79cmjKKiIiUUKlb2t5jZjsB+wBPApPd/e3o1DvA5OhxA7As8bLl0TGpdHHrWlKype3hh8MOCLNnw8iRxSuXiIhImSiLiQhmNhb4OXCeu6+3xC9rd3cz8zzvN4vQfUpDQwMrV64sZHH7aGxsHNT7DwfW2opt2PDe8zVr1uDjxuEbNoA7E7/9baq32453jzwSBvm/51Clz2lhqT4LT3VaWKrPwit1nZY8tJlZDSGw3ebuv4gOv2tm27v721H356ro+Apgh8TLp0bHenH3ucBcgOnTp/uUKVMGrfyxYrzHkLZ2bdjovbY2dJMC202dCnV1cNddYSzbnDlM2WWXEhe0sulzWliqz8JTnRaW6rPwSlmnpZ49asBPgVfc/ZrEqYXAF6PHXwR+nTh+ajSLdAawLtGNKuXMB2gsjddmS3Z9VleHyQnnnQf77ANnnTV45RMRESlzpW5p+xhwCvCSmT0fHbsIuBL4mZmdDrwFnBCduxc4ClgCbAQ0Ir3cucOqVWGCQX09jBrV+3xraxi7Foe2UaOgqwuvrQ2vPflkWLMGFi3KvG6biIjIMFHS0ObujwGW5fShGa534OxBLZQU1qZNqUDW3AxdXaEbtKoKNm6Edet6Xz9iBGy9dRjLdvLJsHgx3HRTaGkTEREZxkrd0iZD3caNqcfusGEDtLfDxImwfn3va2tqQqvbM88w6ayz4Jln4Oqr4fTTi1tmERGRMqTQJoOnqytMKqiqgtGjU+uvdXSEiQfxzgcQukAXL4Zf/xoefZTq8ePh9tvhxBNLU3YREZEyo9Am+ensDOPQxo3LvKF7+rUQWtBGjUqFNoC2tvC9uRm++c2wp2h3N+y+O1x1Fe/OnMn2H/jA4PwMIiIiFUihTXLX1QVNTSFcdXeHLs5s1qwJ3aAQZoHW1MCECeH1EF6/YAFccUUIf+efD6ecAnvuCYBrLTYREZFeFNokNx0dYfmNWBzIMmlv730+nvVZVxda3F56KYS0Z56Bo4+G668P21OJiIhIVgptkptM20y5995qqrk5hLXqtI9V/LytDa65Br73vTDG7cYbwySD9I3hRUREpI+y2XtUyly8bEdSV1fqcU9PmCna3d23Fc4MbrstjFe77DL49KfDXqKnnKLAJiIikiO1tEluMoW2jo4wVg1CK1p3N7zySuj+XLs2fL37Ljz2WPi+xx5hwsGHPxzCWm1tcX8GERGRCqbQJtl1dYVu0WiXAiDsatDdHdZbW7cujFfr6IA5c+C662DZstTrR46EbbeFQw4JS3ccc0yYdBAv9aFWNhERkZwptElmPT3Q2BjGra1dmzpeVxeCl3tYHHfOHLj88nDNfvvBpZfCxz8OU6aEcWuZDLRUiIiIiPSh0CaZtbVl3uQ9Dlx/+xuceSY89VQIaxdfHMaqpe8tKiIiIgWhJg/pG86amsJMUOjdhTluXOgWveAC+MhHYOlS+MlP4A9/gKOOUmATEREZRGppG47i3Qjq6kIXZ2trCGf19eF7fB5g0qQw2WD58jDzc+7c0BU6axb853/2v8CuiIiIFIxC23DT3p7alWDcuNTWUu6hda2uLnXtVlvB88/DD34Ad90Vxrl99rNw4YWhS1RERESKRqFtOOjoCAvcVlXBpk2p4+vXh+9jxoRrOjvDWmvu8MIL8F//Bb//fQhvX/ta+Np555L8CCIiIsOdQttQ194e9gEdOTJ0ZWbafqquLoS6pia4994wTu3FF6GhAa6+Gs44I7TKiYiISMkotA1169aF7+3tqR0LRowIm7c3N6cWx739drjySnjzTdhtN7jpJvjCF0LYExERkZJTaBvKNm7svdVUPCN05MgQ1kaNChMLrrkGVqyAffcNY9c++9nUJu8iIiJSFhTahhr3MDatpiYszwEhnCXHsjU2hvFq118fukQPOgjmzYPPfEa7FIiIiJQphbahwD18dXeH5TiSrWvV1WEpj7/8Be67Dx55BP70p3DuuOPCTNAZM0pTbhEREcmZQlsl6e5OTSSIdyyIW9biBXJbW+GNN+C11+D118MYtRdfDBu2Q9is/aKLwuSCHXcszc8hIiIieVNoKyfd3SF8VVWF9dPiUNbVFbotOztTm61v3Ng7nMXfly9P3a+2Fv7hH0K356GHwmGHhT1BRUREpOIotJWLdetCK1kmmzaFRW6fey58vfQSLFuWOl9TE8LZJz4BH/pQ+Np9d9hll9A9KiIiIhVPv9FLoa0tzOSMZ2h2daW6N6uqQmvasmWweDE89FAYg9bREc7vuisceCDssUcqoCmciYiIDHn6TV8snZ2hNa2qKoxLc091dUI4Xl8PixaFJTgefzwc32MPOOccOOQQOOCAsBeoiIiIDDsKbYOtpyeEteSSGwCjR4elOMxCK9nrr8Mpp8CDD8L73w/f+x6cdBJMnVqacouIiEhZUWgbTF1dYQup7u7wvLY2hLWamtROBC0t8O//HlrXRo+GH/8YvvIVLW4rIiIivSi0DRb3sGZad3cIa+PHp4JafP7nP4evfz3M+PzSl+Cqq2DbbUtWZBERESlfVaUuwJC1YUMYx1ZdHTZqTwa2116Dww+H448P5x57DG6+WYFNREREslJoGwwdHaHbE8LkgnhrqLVr4fzzYc894c9/hh/9CJ5+Gj72sdKVVURERCqCukcLrasr7OcJMHZs6BpdsQJuvDGEtOZmOO00uOwymDy5tGUVERGRiqHQVkg9PSGw9fSE7tAnngibsi9cGMa2HXUUXHFF2EpKREREJA8KbYUQbzW1cmVYX+23v4V774XGxrCu2gUXwFlnhUVwRURERDaDQtuW6uxk4syZYeLB0qUhvI0aBcceGyYaHHssjBxZ6lKKiIhIhavI0GZmRwA/BEYAN7n7laUsT8+ECWHm55FHwsEHhw3aR48uZZFERERkiKm40GZmI4A5wGeA5cBTZrbQ3V8uSYFqalg7fz6jpkwpyduLiIjI8FCJS37sDyxx96Xu3gHcAcwscZlEREREBlUlhrYGYFni+fLomIiIiMiQVXHdo7kws1nALICGhgZWrlw5qO/X2Ng4qPcfjlSnhac6LSzVZ+GpTgtL9Vl4pa7TSgxtK4AdEs+nRsfe4+5zgbkA06dP9ylFGG9WjPcYblSnhac6LSzVZ+GpTgtL9Vl4pazTSuwefQqYZmY7m1ktcCKwsMRlEhERERlUFdfS5u5dZnYOcD9hyY957v7XEhdLREREZFBVXGgDcPd7gXtLXQ4RERGRYqnE7lERERGRYUehTURERKQCKLSJiIiIVACFNhEREZEKoNAmIiIiUgEU2kREREQqgLl7qcswqMysEXhrkN9mErB6kN9juFGdFp7qtLBUn4WnOi0s1WfhFaNOd3T3bTKdGPKhrRjM7Gl3n17qcgwlqtPCU50Wluqz8FSnhaX6LLxS16m6R0VEREQqgEKbiIiISAVQaCuMuaUuwBCkOi081WlhqT4LT3VaWKrPwitpnWpMm4iIiEgFUEubiIiISAVQaMuDmR1hZq+Z2RIzm53h/EgzuzM6/6SZ7VT8UlaWHOr0S2bWaGbPR19nlKKclcLM5pnZKjP7S5bzZmbXRvX9opntW+wyVpIc6vMgM1uX+Hz+e7HLWGnMbAcze9jMXjazv5rZuRmu0ec0RznWpz6neTCzOjP7s5m9ENXpJRmuKcnve4W2HJnZCGAOcCSwO3CSme2edtnpwFp33w34AXBVcUtZWXKsU4A73X3v6Oumohay8swHjujn/JHAtOhrFnBdEcpUyebTf30CPJr4fF5ahDJVui7gAnffHZgBnJ3h370+p7nLpT5Bn9N8tAOHuPtewN7AEWY2I+2akvy+V2jL3f7AEndf6u4dwB3AzLRrZgILosd3A4eamRWxjJUmlzqVPLj7I0BTP5fMBG7x4Amg3sy2L07pKk8O9Sl5cve33f3Z6PEG4BWgIe0yfU5zlGN9Sh6iz11L9LQm+kqfAFCS3/cKbblrAJYlni+n7z+M965x9y5gHTCxKKWrTLnUKcA/RV0kd5vZDsUp2pCVa51L7j4adaP81sw+VOrCVJKoS2kf4Mm0U/qcboZ+6hP0Oc2LmY0ws+eBVcAD7p71M1rM3/cKbVLufgPs5O4fBh4g9ZeNSDl4lrDlzF7Aj4Bflbg8FcPMxgI/B85z9/WlLk+lG6A+9TnNk7t3u/vewFRgfzPbo9RlAoW2fKwAkq08U6NjGa8xs2pgPLCmKKWrTAPWqbuvcff26OlNwEeKVLahKpfPseTI3dfH3Sjufi9QY2aTSlyssmdmNYSAcZu7/yLDJfqc5mGg+tTndPO5ezPwMH3Htpbk971CW+6eAqaZ2c5mVgucCCxMu2Yh8MXo8T8DD7kWwuvPgHWaNo7lOMJ4Ddl8C4FTo9l5M4B17v52qQtVqcxsu3gci5ntT/h/qv5Q60dUXz8FXnH3a7Jcps9pjnKpT31O82Nm25hZffR4FPAZ4NW0y0ry+756sN9gqHD3LjM7B7gfGAHMc/e/mtmlwNPuvpDwD+dWM1tCGLx8YulKXP5yrNP/a2bHEWZINQFfKlmBK4CZ3Q4cBEwys+XAfxAG0eLu1wP3AkcBS4CNwJdLU9LKkEN9/jPwL2bWBWwCTtQfagP6GHAK8FI0ZgjgIuB9oM/pZsilPvU5zc/2wIJohYMq4Gfuvqgcft9rRwQRERGRCqDuUREREZEKoNAmIiIiUgEU2kREREQqgEKbiIiISAVQaBMREREpADObZ2arzOwvOV5/gpm9HG1M/z8DXa/QJiJDVvQ/woNKXQ4RGTbm03ch3ozMbBrwLeBj7v4h4LyBXqN12kSkYplZS+LpaKAd6I6enxX9j7DYZXJgmrsvKfZ7i0hpufsj0R6w7zGzXYE5wDaEdQfPdPdXgTOBOe6+NnrtqoHur9AmIhXL3cfGj83s78AZ7v5g6UokItLHXOAr7v6GmR0A/AQ4BHg/gJn9kbDA/Hfc/b7+bqTuUREZsszs72b26ejxd8zsLjP7f2a2wcxeMrP3m9m3ojEoy8zssMRrx5vZT83sbTNbYWaXRSukY2a7mdkfzGydma02szuj449EL3/BzFrM7PPR8WPM7HkzazazP5nZh9PK+K1oXMtaM7vZzOqic5PMbFH0uiYze9TM9P9tkQphZmOBA4G7oh0rbiDsuACh4WwaYdeVk4Ab4+2zstE/fhEZTo4FbgW2Bp4jbKFWBTQAlxL+hxqbT9g+bTdgH+Aw4Izo3HeBxdF9pgI/AnD3T0bn93L3se5+p5ntA8wDzgImRu+x0MxGJt7rZOBwYFfCX9//Fh2/AFhO6FaZTNieSNvYiFSOKqDZ3fdOfP1DdG45sNDdO939b8DrhBDX781ERIaLR939fnfvAu4ihKEr3b0TuAPYyczqzWwyYe/L89y9NRpr8gNS+wt2AjsCU9y9zd0f6+c9ZwE3uPuT7t7t7gsIY+9mJK75sbsvc/cm4HLCX93x+2wP7Bj9j/1R7RkpUjncfT3wNzM7HsCCvaLTvyK0smFmkwh/sC3t734KbSIynLybeLwJWO3u3YnnAGMJgawGeDvqmmwmtJBtG13zr4ABf45mqJ7Wz3vuCFwQ3ye61w7AlMQ1yxKP30qc+y/CpumLzWypmc3O54cVkeIys9uBx4EPmNlyMzud0JJ+upm9APwVmBldfj+wxsxeBh4Gvunua/q7vyYiiIj0tYzQGjYpapXrxd3fIcz8wsw+DjxoZo9kmTG6DLjc3S/v5/12SDx+H7Ayep8NhC7SC8xsD+AhM3vK3X+3OT+UiAwudz8py6k+y4BErebnR185UUubiEgad3+bMGbtajMbZ2ZVZrarmX0KwMyON7Op0eVrCePMeqLn7wK7JG53I/AVMzsg6hoZY2ZHm9lWiWvONrOpZjYB+DYQT2w4Jpr0YMA6wnImPYjIsKTQJiKS2alALfAyIZjdTWrW137Ak9E6cQuBc909HovyHWBB1BV6grs/TWiV+3F0nyXAl9Le638IIXEp8CZwWXR8GvAg0ELocvmJuz9c2B9TRCqFaUyriEjpaH05EcmVWtpEREREKoBCm4iIiEgFUPeoiIiISAVQS5uIiIhIBVBoExEREakACm0iIiIiFUChTURERKQCKLSJiIiIVACFNhEREZEK8P8B2xH049smpPwAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "\n",
        "import os\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "env_name = 'CartPole-v1'\n",
        "# env_name = 'LunarLander-v2'\n",
        "# env_name = 'BipedalWalker-v2'\n",
        "# env_name = 'RoboschoolWalker2d-v1'\n",
        "\n",
        "\n",
        "fig_num = 0     #### change this to prevent overwriting figures in same env_name folder\n",
        "\n",
        "plot_avg = True    # plot average of all runs; else plot all runs separately\n",
        "\n",
        "fig_width = 10\n",
        "fig_height = 6\n",
        "\n",
        "\n",
        "# smooth out rewards to get a smooth and a less smooth (var) plot lines\n",
        "window_len_smooth = 50\n",
        "min_window_len_smooth = 1\n",
        "linewidth_smooth = 1.5\n",
        "alpha_smooth = 1\n",
        "\n",
        "window_len_var = 5\n",
        "min_window_len_var = 1\n",
        "linewidth_var = 2\n",
        "alpha_var = 0.1\n",
        "\n",
        "\n",
        "colors = ['red', 'blue', 'green', 'orange', 'purple', 'olive', 'brown', 'magenta', 'cyan', 'crimson','gray', 'black']\n",
        "\n",
        "\n",
        "# make directory for saving figures\n",
        "figures_dir = \"PPO_figs\"\n",
        "if not os.path.exists(figures_dir):\n",
        "    os.makedirs(figures_dir)\n",
        "\n",
        "# make environment directory for saving figures\n",
        "figures_dir = figures_dir + '/' + env_name + '/'\n",
        "if not os.path.exists(figures_dir):\n",
        "    os.makedirs(figures_dir)\n",
        "\n",
        "\n",
        "fig_save_path = figures_dir + '/PPO_' + env_name + '_fig_' + str(fig_num) + '.png'\n",
        "\n",
        "\n",
        "# get number of log files in directory\n",
        "log_dir = \"PPO_logs\" + '/' + env_name + '/'\n",
        "\n",
        "current_num_files = next(os.walk(log_dir))[2]\n",
        "num_runs = len(current_num_files)\n",
        "\n",
        "\n",
        "all_runs = []\n",
        "\n",
        "for run_num in range(num_runs):\n",
        "\n",
        "    log_f_name = log_dir + '/PPO_' + env_name + \"_log_\" + str(run_num) + \".csv\"\n",
        "    print(\"loading data from : \" + log_f_name)\n",
        "    data = pd.read_csv(log_f_name)\n",
        "    data = pd.DataFrame(data)\n",
        "    \n",
        "    print(\"data shape : \", data.shape)\n",
        "    \n",
        "    all_runs.append(data)\n",
        "    print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "\n",
        "ax = plt.gca()\n",
        "\n",
        "if plot_avg:\n",
        "    # average all runs\n",
        "    df_concat = pd.concat(all_runs)\n",
        "    df_concat_groupby = df_concat.groupby(df_concat.index)\n",
        "    data_avg = df_concat_groupby.mean()\n",
        "\n",
        "    # smooth out rewards to get a smooth and a less smooth (var) plot lines\n",
        "    data_avg['reward_smooth'] = data_avg['reward'].rolling(window=window_len_smooth, win_type='triang', min_periods=min_window_len_smooth).mean()\n",
        "    data_avg['reward_var'] = data_avg['reward'].rolling(window=window_len_var, win_type='triang', min_periods=min_window_len_var).mean()\n",
        "\n",
        "    data_avg.plot(kind='line', x='timestep' , y='reward_smooth',ax=ax,color=colors[0],  linewidth=linewidth_smooth, alpha=alpha_smooth)\n",
        "    data_avg.plot(kind='line', x='timestep' , y='reward_var',ax=ax,color=colors[0],  linewidth=linewidth_var, alpha=alpha_var)\n",
        "\n",
        "    # keep only reward_smooth in the legend and rename it\n",
        "    handles, labels = ax.get_legend_handles_labels()\n",
        "    ax.legend([handles[0]], [\"reward_avg_\" + str(len(all_runs)) + \"_runs\"], loc=2)\n",
        "\n",
        "\n",
        "else:\n",
        "    for i, run in enumerate(all_runs):\n",
        "        # smooth out rewards to get a smooth and a less smooth (var) plot lines\n",
        "        run['reward_smooth_' + str(i)] = run['reward'].rolling(window=window_len_smooth, win_type='triang', min_periods=min_window_len_smooth).mean()\n",
        "        run['reward_var_' + str(i)] = run['reward'].rolling(window=window_len_var, win_type='triang', min_periods=min_window_len_var).mean()\n",
        "        \n",
        "        # plot the lines\n",
        "        run.plot(kind='line', x='timestep' , y='reward_smooth_' + str(i),ax=ax,color=colors[i % len(colors)],  linewidth=linewidth_smooth, alpha=alpha_smooth)\n",
        "        run.plot(kind='line', x='timestep' , y='reward_var_' + str(i),ax=ax,color=colors[i % len(colors)],  linewidth=linewidth_var, alpha=alpha_var)\n",
        "\n",
        "    # keep alternate elements (reward_smooth_i) in the legend\n",
        "    handles, labels = ax.get_legend_handles_labels()\n",
        "    new_handles = []\n",
        "    new_labels = []\n",
        "    for i in range(len(handles)):\n",
        "        if(i%2 == 0):\n",
        "            new_handles.append(handles[i])\n",
        "            new_labels.append(labels[i])\n",
        "    ax.legend(new_handles, new_labels, loc=2)\n",
        "\n",
        "\n",
        "\n",
        "# ax.set_yticks(np.arange(0, 1800, 200))\n",
        "# ax.set_xticks(np.arange(0, int(4e6), int(5e5)))\n",
        "\n",
        "\n",
        "ax.grid(color='gray', linestyle='-', linewidth=1, alpha=0.2)\n",
        "\n",
        "ax.set_xlabel(\"Timesteps\", fontsize=12)\n",
        "ax.set_ylabel(\"Rewards\", fontsize=12)\n",
        "\n",
        "plt.title(env_name, fontsize=14)\n",
        "\n",
        "\n",
        "fig = plt.gcf()\n",
        "fig.set_size_inches(fig_width, fig_height)\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "plt.savefig(fig_save_path)\n",
        "print(\"figure saved at : \", fig_save_path)\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "plt.show()\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YaWPRW9EGxgH"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "\n",
        "################################ End of Part IV ################################\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G8uG43MtHNGC"
      },
      "source": [
        "################################################################################\n",
        "> # **Part - V**\n",
        "\n",
        "*   install virtual display libraries for rendering on colab / remote server ^\n",
        "*   load preTrained networks and save images for gif\n",
        "*   generate and save gif from previously saved images\n",
        "\n",
        "*   ^ If running locally; do not install xvbf and pyvirtualdisplay. Just comment out the virtual display code and render it normally. \n",
        "*   ^ You will still require to use ipythondisplay, if you want to render it in the Jupyter Notebook.\n",
        "\n",
        "################################################################################"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VL3tpKf3HLAq"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "#### to render on colab / server / headless machine install virtual display libraries\n",
        "\n",
        "!apt-get install -y xvfb python-opengl > /dev/null 2>&1\n",
        "\n",
        "!pip install gym pyvirtualdisplay > /dev/null 2>&1\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j5Rx_IFKHK-D"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "############################# save images for gif ##############################\n",
        "\n",
        "\n",
        "import os\n",
        "import glob\n",
        "\n",
        "import gym\n",
        "import roboschool\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from PIL import Image\n",
        "\n",
        "\n",
        "from IPython import display as ipythondisplay\n",
        "\n",
        "from pyvirtualdisplay import Display\n",
        "\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "One frame corresponding to each timestep is saved in a folder :\n",
        "\n",
        "PPO_gif_images/env_name/000001.jpg\n",
        "PPO_gif_images/env_name/000002.jpg\n",
        "PPO_gif_images/env_name/000003.jpg\n",
        "...\n",
        "...\n",
        "...\n",
        "\n",
        "\n",
        "if this section is run multiple times or for multiple episodes for the same env_name; \n",
        "then the saved images will be overwritten.\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "#### beginning of virtual display code section\n",
        "\n",
        "display = Display(visible=0, size=(400, 300))\n",
        "display.start()\n",
        "\n",
        "#### end of virtual display code section\n",
        "\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "################## hyperparameters ##################\n",
        "\n",
        "env_name = \"CartPole-v1\"\n",
        "has_continuous_action_space = False\n",
        "max_ep_len = 400\n",
        "action_std = None\n",
        "\n",
        "\n",
        "# env_name = \"LunarLander-v2\"\n",
        "# has_continuous_action_space = False\n",
        "# max_ep_len = 300\n",
        "# action_std = None\n",
        "\n",
        "# env_name = \"BipedalWalker-v2\"\n",
        "# has_continuous_action_space = True\n",
        "# max_ep_len = 1500           # max timesteps in one episode\n",
        "# action_std = 0.1            # set same std for action distribution which was used while saving\n",
        "\n",
        "# env_name = \"RoboschoolWalker2d-v1\"\n",
        "# has_continuous_action_space = True\n",
        "# max_ep_len = 1000           # max timesteps in one episode\n",
        "# action_std = 0.1            # set same std for action distribution which was used while saving\n",
        "\n",
        "\n",
        "total_test_episodes = 1     # save gif for only one episode\n",
        "\n",
        "render_ipython = False      # plot the images using matplotlib and ipythondisplay before saving (slow)\n",
        "\n",
        "K_epochs = 80               # update policy for K epochs\n",
        "eps_clip = 0.2              # clip parameter for PPO\n",
        "gamma = 0.99                # discount factor\n",
        "\n",
        "lr_actor = 0.0003         # learning rate for actor\n",
        "lr_critic = 0.001         # learning rate for critic\n",
        "\n",
        "#####################################################\n",
        "\n",
        "\n",
        "env = gym.make(env_name)\n",
        "\n",
        "# state space dimension\n",
        "state_dim = env.observation_space.shape[0]\n",
        "\n",
        "# action space dimension\n",
        "if has_continuous_action_space:\n",
        "    action_dim = env.action_space.shape[0]\n",
        "else:\n",
        "    action_dim = env.action_space.n\n",
        "\n",
        "\n",
        "\n",
        "# make directory for saving gif images\n",
        "gif_images_dir = \"PPO_gif_images\" + '/'\n",
        "if not os.path.exists(gif_images_dir):\n",
        "    os.makedirs(gif_images_dir)\n",
        "\n",
        "# make environment directory for saving gif images\n",
        "gif_images_dir = gif_images_dir + '/' + env_name + '/'\n",
        "if not os.path.exists(gif_images_dir):\n",
        "    os.makedirs(gif_images_dir)\n",
        "\n",
        "# make directory for gif\n",
        "gif_dir = \"PPO_gifs\" + '/'\n",
        "if not os.path.exists(gif_dir):\n",
        "    os.makedirs(gif_dir)\n",
        "\n",
        "# make environment directory for gif\n",
        "gif_dir = gif_dir + '/' + env_name  + '/'\n",
        "if not os.path.exists(gif_dir):\n",
        "    os.makedirs(gif_dir)\n",
        "\n",
        "\n",
        "\n",
        "ppo_agent = PPO(state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, has_continuous_action_space, action_std)\n",
        "\n",
        "\n",
        "# preTrained weights directory\n",
        "\n",
        "random_seed = 0             #### set this to load a particular checkpoint trained on random seed\n",
        "run_num_pretrained = 0      #### set this to load a particular checkpoint num\n",
        "\n",
        "\n",
        "directory = \"PPO_preTrained\" + '/' + env_name + '/'\n",
        "checkpoint_path = directory + \"PPO_{}_{}_{}.pth\".format(env_name, random_seed, run_num_pretrained)\n",
        "print(\"loading network from : \" + checkpoint_path)\n",
        "\n",
        "ppo_agent.load(checkpoint_path)\n",
        "\n",
        "print(\"--------------------------------------------------------------------------------------------\")\n",
        "\n",
        "\n",
        "\n",
        "test_running_reward = 0\n",
        "\n",
        "for ep in range(1, total_test_episodes+1):\n",
        "    \n",
        "    ep_reward = 0\n",
        "    state = env.reset()\n",
        "\n",
        "    for t in range(1, max_ep_len+1):\n",
        "        action = ppo_agent.select_action(state)\n",
        "        state, reward, done, _ = env.step(action)\n",
        "        ep_reward += reward\n",
        "\n",
        "        img = env.render(mode = 'rgb_array')\n",
        "\n",
        "\n",
        "        #### beginning of ipythondisplay code section 1\n",
        "\n",
        "        if render_ipython:\n",
        "            plt.imshow(img)\n",
        "            ipythondisplay.clear_output(wait=True)\n",
        "            ipythondisplay.display(plt.gcf())\n",
        "\n",
        "        #### end of ipythondisplay code section 1\n",
        "\n",
        "\n",
        "        img = Image.fromarray(img)\n",
        "        img.save(gif_images_dir + '/' + str(t).zfill(6) + '.jpg')\n",
        "        \n",
        "        if done:\n",
        "            break\n",
        "    \n",
        "    # clear buffer    \n",
        "    ppo_agent.buffer.clear()\n",
        "    \n",
        "    test_running_reward +=  ep_reward\n",
        "    print('Episode: {} \\t\\t Reward: {}'.format(ep, round(ep_reward, 2)))\n",
        "    ep_reward = 0\n",
        "\n",
        "\n",
        "\n",
        "env.close()\n",
        "\n",
        "\n",
        "#### beginning of ipythondisplay code section 2\n",
        "\n",
        "if render_ipython:\n",
        "    ipythondisplay.clear_output(wait=True)\n",
        "\n",
        "#### end of ipythondisplay code section 2\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "print(\"total number of frames / timesteps / images saved : \", t)\n",
        "\n",
        "avg_test_reward = test_running_reward / total_test_episodes\n",
        "avg_test_reward = round(avg_test_reward, 2)\n",
        "print(\"average test reward : \" + str(avg_test_reward))\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BoVshl_ZHK7s"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "\n",
        "######################## generate gif from saved images ########################\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "env_name = 'CartPole-v1'\n",
        "# env_name = 'LunarLander-v2'\n",
        "# env_name = 'BipedalWalker-v2'\n",
        "# env_name = 'RoboschoolWalker2d-v1'\n",
        "\n",
        "\n",
        "gif_num = 0     #### change this to prevent overwriting gifs in same env_name folder\n",
        "\n",
        "# adjust following parameters to get desired duration, size (bytes) and smoothness of gif\n",
        "total_timesteps = 300\n",
        "step = 10\n",
        "frame_duration = 150\n",
        "\n",
        "\n",
        "# input images\n",
        "gif_images_dir = \"PPO_gif_images/\" + env_name + '/*.jpg'\n",
        "\n",
        "\n",
        "# ouput gif path\n",
        "gif_dir = \"PPO_gifs\"\n",
        "if not os.path.exists(gif_dir):\n",
        "    os.makedirs(gif_dir)\n",
        "\n",
        "gif_dir = gif_dir + '/' + env_name\n",
        "if not os.path.exists(gif_dir):\n",
        "    os.makedirs(gif_dir)\n",
        "\n",
        "gif_path = gif_dir + '/PPO_' + env_name + '_gif_' + str(gif_num) + '.gif'\n",
        "\n",
        "\n",
        "\n",
        "img_paths = sorted(glob.glob(gif_images_dir))\n",
        "img_paths = img_paths[:total_timesteps]\n",
        "img_paths = img_paths[::step]\n",
        "\n",
        "\n",
        "print(\"total frames in gif : \", len(img_paths))\n",
        "print(\"total duration of gif : \" + str(round(len(img_paths) * frame_duration / 1000, 2)) + \" seconds\")\n",
        "\n",
        "\n",
        "\n",
        "# save gif\n",
        "img, *imgs = [Image.open(f) for f in img_paths]\n",
        "img.save(fp=gif_path, format='GIF', append_images=imgs, save_all=True, optimize=True, duration=frame_duration, loop=0)\n",
        "\n",
        "print(\"saved gif at : \", gif_path)\n",
        "\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "20d1bR8xHK5j"
      },
      "outputs": [],
      "source": [
        "\n",
        "############################# check gif byte size ##############################\n",
        "\n",
        "\n",
        "import os\n",
        "import glob\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "env_name = 'CartPole-v1'\n",
        "# env_name = 'LunarLander-v2'\n",
        "# env_name = 'BipedalWalker-v2'\n",
        "# env_name = 'RoboschoolWalker2d-v1'\n",
        "\n",
        "\n",
        "gif_dir = \"PPO_gifs/\" + env_name + '/*.gif'\n",
        "\n",
        "gif_paths = sorted(glob.glob(gif_dir))\n",
        "\n",
        "for gif_path in gif_paths:\n",
        "    file_size = os.path.getsize(gif_path)\n",
        "    print(gif_path + '\\t\\t' + str(round(file_size / (1024 * 1024), 2)) + \" MB\")\n",
        "\n",
        "\n",
        "print(\"============================================================================================\")\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rM5UIAkcGxeA"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "\n",
        "################################# End of Part V ################################\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7YUzQOu1HYHR"
      },
      "source": [
        "################################################################################\n",
        "\n",
        "---------------------------------------------------------------------------- That's all folks ! ----------------------------------------------------------------------------\n",
        "\n",
        "\n",
        "################################################################################"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "e7JowRQEGGKQ",
        "Z4VJcUT2GlJz"
      ],
      "toc_visible": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}