cart-pole-drl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-learning for Cart-Pole\n",
    "\n",
    "This notebook uses OpenAI Gym and Deep Q-learning to creating a playing agent for Cart-Pole. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import dependencies and create a Cart-Pole playing environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env = gym.make('CartPole-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the OpenAI Gym environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get a list of the possible actions for this game"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Discrete(2)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two possible actions, moving the cart left and right--coded as 0 or 1 in the environment\n",
    "\n",
    "---\n",
    "\n",
    "Let's run a random simulation to see how the game it played"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.reset()\n",
    "rewards = []\n",
    "for move in range(100):\n",
    "    env.render()\n",
    "    state, reward, done, info = env.step(env.action_space.sample())\n",
    "    rewards.append(reward)\n",
    "    if done:\n",
    "        rewards = []\n",
    "        env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "print(rewards)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object of the game is to move the cart left or right to keep the pole from falling. The longer the pole stays up, the more reward we receive. For this game, we get a reward of 1 for each step that the pole is still standing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the Q-Network\n",
    "\n",
    "In reinforcement learning we usually keep a matrix of all state-action pairs and update the values to help the agent learn. For some games, such as cart-pole, the number of state-action paris is simply too large for this to be feasible. Even for a simple game like cart-pole, there are four real-valued numbers that make up each possible state--position and velocity of the cart, and position and velocity of the pole. This creates a nearly infinite number of states.\n",
    "\n",
    "In deep Q-learning, we use a neural network to approximate the Q-table. Our A-network takes a state as input and outputs q-values for each possible action. \n",
    "\n",
    "Our targets for training are $\\hat{Q}(s,a) = r + \\gamma \\max{Q(s', a')}$, thus we want to minimize $(\\hat{Q}(s,a) - Q(s,a))^2$. This can be thought of as a measurement of how much reward can be expected in the next time step if we take a given action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class QNetwork():\n",
    "    def __init__(self, learning_rate=0.01, state_size=4, action_size=2, hidden_size=10, name='QNetwork'):\n",
    "        with tf.variable_scope(name):\n",
    "            self.inputs_ = tf.placeholder(tf.float32, [None, state_size], name='inputs')\n",
    "            self.actions_ = tf.placeholder(tf.int32, [None], name='actions')\n",
    "            one_hot_actions = tf.one_hot(self.actions_, action_size)\n",
    "\n",
    "            # Target placeholder for training\n",
    "            self.targetQs_ = tf.placeholder(tf.float32, [None], name='target')\n",
    "            \n",
    "            # Hidden layers\n",
    "            self.fc1 = tf.contrib.layers.fully_connected(self.inputs_, hidden_size)\n",
    "            self.fc2 = tf.contrib.layers.fully_connected(self.fc1, hidden_size)\n",
    "            self.fc3 = tf.contrib.layers.fully_connected(self.fc2, hidden_size)\n",
    "            \n",
    "            # Output layer\n",
    "            self.output = tf.contrib.layers.fully_connected(self.fc2, action_size, activation_fn=None)\n",
    "            \n",
    "            # Trian on (targetQ - Q)^2\n",
    "            self.Q = tf.reduce_sum(tf.multiply(self.output, one_hot_actions), axis=1)\n",
    "            \n",
    "            self.loss = tf.reduce_mean(tf.square(self.targetQs_ - self.Q))\n",
    "            self.optimizer = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### State Memory \n",
    "Reinforcement learning algorithms can have stability issues due to correlations between states. Thus, it's usually not a good idea to train in sequential states as the agent plays the game. Instead, we will let the agent play the game, store the experiences in memory, and then train the network on a random sample of past experiences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "class Memory():\n",
    "    def __init__(self, max_size=1000):\n",
    "        self.buffer = deque(maxlen=max_size)\n",
    "        \n",
    "    def add(self, experience):\n",
    "        self.buffer.append(experience)\n",
    "        \n",
    "    def sample(self, batch_size):\n",
    "        index_list = np.random.choice(np.arange(len(self.buffer)), size=batch_size, replace=False)\n",
    "        return [self.buffer[index] for index in index_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploration vs. Exploitation\n",
    "In order for the agent to learn, it needs to explore its environmnet by taking random actions. As the agent learns, we want to take advantage of its exploration early on and choose what it thinks is the best action (exploit). At each step in the game we will decide whether the agent will explore or exploit. At the start of the game exploration will be more likely, but as the game progresses we will push the agent to exploit more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Algorithm\n",
    "The network will be trained in *episodes*, which is the same as one simulation of the game. For Cart-Pole, the goal of an episode is to keep the pole upright for 195 frames. We start a new episode when meeting that goal or if the game ends because the pole tilts too far or the cart tries to move off the screen. This is how we'll train the agent.\n",
    "\n",
    "* Initialize the memory $D$\n",
    "* Initialize the action-value network $Q$ with random weights\n",
    "* **For** episode = 1, $M$ **do**\n",
    "  * **For** $t$, $T$ **do**\n",
    "     * With probability $\\epsilon$ select a random action $a_t$, otherwise select $a_t = \\mathrm{argmax}_a Q(s,a)$\n",
    "     * Execute action $a_t$ in simulator and observe reward $r_{t+1}$ and new state $s_{t+1}$\n",
    "     * Store transition $<s_t, a_t, r_{t+1}, s_{t+1}>$ in memory $D$\n",
    "     * Sample random mini-batch from $D$: $<s_j, a_j, r_j, s'_j>$\n",
    "     * Set $\\hat{Q}_j = r_j$ if the episode ends at $j+1$, otherwise set $\\hat{Q}_j = r_j + \\gamma \\max_{a'}{Q(s'_j, a')}$\n",
    "     * Make a gradient descent step with loss $(\\hat{Q}_j - Q(s_j, a_j))^2$\n",
    "  * **endfor**\n",
    "* **endfor**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_episodes = 1500          # max number of episodes to learn from\n",
    "max_steps = 200                # max steps in an episode\n",
    "gamma = 0.99                   # future reward discount\n",
    "\n",
    "# Exploration parameters\n",
    "explore_start = 1.0            # exploration probability at start\n",
    "explore_stop = 0.01            # minimum exploration probability \n",
    "decay_rate = 0.0001            # exponential decay rate for exploration prob\n",
    "\n",
    "# Network parameters\n",
    "hidden_size = 128               # number of units in each Q-network hidden layer\n",
    "learning_rate = 0.0001         # Q-network learning rate\n",
    "\n",
    "# Memory parameters\n",
    "memory_size = 10000            # memory capacity\n",
    "batch_size = 20                # experience mini-batch size\n",
    "pretrain_length = batch_size   # number experiences to pretrain the memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "mainQN = QNetwork(name='main', hidden_size=hidden_size, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Memory\n",
    "Reset the simulation and populate the memory with a set of transitions to train on. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize the simulation\n",
    "env.reset()\n",
    "# Take one random step to get the pole and cart moving\n",
    "state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "memory = Memory(max_size=memory_size)\n",
    "\n",
    "for index in range(pretrain_length):\n",
    "    # Uncomment to watch simulation\n",
    "    # env.render()\n",
    "    \n",
    "    action = env.action_space.sample()\n",
    "    next_state, reward, done, _ = env.step(action)\n",
    "    \n",
    "    if done:\n",
    "        next_state = np.zeros(state.shape)\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        env.reset()\n",
    "        state, reward, done, _ = env.step(env.action_space.sample())\n",
    "    else:\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1 Total reward: 4.0 Training loss: 1.1756 Explore P: 0.9996\n",
      "Episode: 2 Total reward: 13.0 Training loss: 1.1753 Explore P: 0.9983\n",
      "Episode: 3 Total reward: 14.0 Training loss: 1.0523 Explore P: 0.9969\n",
      "Episode: 4 Total reward: 20.0 Training loss: 1.0462 Explore P: 0.9950\n",
      "Episode: 5 Total reward: 16.0 Training loss: 1.1859 Explore P: 0.9934\n",
      "Episode: 6 Total reward: 14.0 Training loss: 1.1397 Explore P: 0.9920\n",
      "Episode: 7 Total reward: 47.0 Training loss: 1.0715 Explore P: 0.9874\n",
      "Episode: 8 Total reward: 26.0 Training loss: 1.3286 Explore P: 0.9849\n",
      "Episode: 9 Total reward: 11.0 Training loss: 1.2614 Explore P: 0.9838\n",
      "Episode: 10 Total reward: 14.0 Training loss: 1.3090 Explore P: 0.9824\n",
      "Episode: 11 Total reward: 39.0 Training loss: 1.3295 Explore P: 0.9787\n",
      "Episode: 12 Total reward: 23.0 Training loss: 1.5444 Explore P: 0.9764\n",
      "Episode: 13 Total reward: 19.0 Training loss: 1.3815 Explore P: 0.9746\n",
      "Episode: 14 Total reward: 32.0 Training loss: 1.5441 Explore P: 0.9715\n",
      "Episode: 15 Total reward: 19.0 Training loss: 1.9377 Explore P: 0.9697\n",
      "Episode: 16 Total reward: 47.0 Training loss: 3.5237 Explore P: 0.9652\n",
      "Episode: 17 Total reward: 15.0 Training loss: 4.4482 Explore P: 0.9638\n",
      "Episode: 18 Total reward: 11.0 Training loss: 2.5958 Explore P: 0.9627\n",
      "Episode: 19 Total reward: 25.0 Training loss: 3.0245 Explore P: 0.9603\n",
      "Episode: 20 Total reward: 29.0 Training loss: 3.0580 Explore P: 0.9576\n",
      "Episode: 21 Total reward: 10.0 Training loss: 3.3248 Explore P: 0.9566\n",
      "Episode: 22 Total reward: 37.0 Training loss: 15.4116 Explore P: 0.9531\n",
      "Episode: 23 Total reward: 18.0 Training loss: 8.2506 Explore P: 0.9514\n",
      "Episode: 24 Total reward: 31.0 Training loss: 7.7061 Explore P: 0.9485\n",
      "Episode: 25 Total reward: 9.0 Training loss: 10.8701 Explore P: 0.9477\n",
      "Episode: 26 Total reward: 40.0 Training loss: 38.1418 Explore P: 0.9439\n",
      "Episode: 27 Total reward: 19.0 Training loss: 16.2673 Explore P: 0.9422\n",
      "Episode: 28 Total reward: 34.0 Training loss: 23.5307 Explore P: 0.9390\n",
      "Episode: 29 Total reward: 10.0 Training loss: 41.6069 Explore P: 0.9381\n",
      "Episode: 30 Total reward: 15.0 Training loss: 16.6162 Explore P: 0.9367\n",
      "Episode: 31 Total reward: 38.0 Training loss: 7.1066 Explore P: 0.9332\n",
      "Episode: 32 Total reward: 11.0 Training loss: 17.7573 Explore P: 0.9321\n",
      "Episode: 33 Total reward: 12.0 Training loss: 8.3796 Explore P: 0.9310\n",
      "Episode: 34 Total reward: 15.0 Training loss: 46.8782 Explore P: 0.9297\n",
      "Episode: 35 Total reward: 13.0 Training loss: 25.8022 Explore P: 0.9285\n",
      "Episode: 36 Total reward: 16.0 Training loss: 26.0559 Explore P: 0.9270\n",
      "Episode: 37 Total reward: 18.0 Training loss: 33.8866 Explore P: 0.9253\n",
      "Episode: 38 Total reward: 31.0 Training loss: 71.4338 Explore P: 0.9225\n",
      "Episode: 39 Total reward: 39.0 Training loss: 11.4439 Explore P: 0.9190\n",
      "Episode: 40 Total reward: 16.0 Training loss: 45.7908 Explore P: 0.9175\n",
      "Episode: 41 Total reward: 17.0 Training loss: 9.8843 Explore P: 0.9160\n",
      "Episode: 42 Total reward: 16.0 Training loss: 6.1753 Explore P: 0.9145\n",
      "Episode: 43 Total reward: 15.0 Training loss: 56.3963 Explore P: 0.9132\n",
      "Episode: 44 Total reward: 10.0 Training loss: 10.8131 Explore P: 0.9123\n",
      "Episode: 45 Total reward: 10.0 Training loss: 11.5590 Explore P: 0.9114\n",
      "Episode: 46 Total reward: 32.0 Training loss: 117.7647 Explore P: 0.9085\n",
      "Episode: 47 Total reward: 20.0 Training loss: 13.7307 Explore P: 0.9067\n",
      "Episode: 48 Total reward: 16.0 Training loss: 354.0401 Explore P: 0.9053\n",
      "Episode: 49 Total reward: 26.0 Training loss: 15.8618 Explore P: 0.9029\n",
      "Episode: 50 Total reward: 14.0 Training loss: 152.5880 Explore P: 0.9017\n",
      "Episode: 51 Total reward: 18.0 Training loss: 12.2115 Explore P: 0.9001\n",
      "Episode: 52 Total reward: 44.0 Training loss: 156.2911 Explore P: 0.8962\n",
      "Episode: 53 Total reward: 12.0 Training loss: 119.7370 Explore P: 0.8951\n",
      "Episode: 54 Total reward: 32.0 Training loss: 106.4296 Explore P: 0.8923\n",
      "Episode: 55 Total reward: 23.0 Training loss: 323.9529 Explore P: 0.8902\n",
      "Episode: 56 Total reward: 8.0 Training loss: 44.7567 Explore P: 0.8895\n",
      "Episode: 57 Total reward: 12.0 Training loss: 14.6540 Explore P: 0.8885\n",
      "Episode: 58 Total reward: 23.0 Training loss: 91.5568 Explore P: 0.8865\n",
      "Episode: 59 Total reward: 14.0 Training loss: 271.1984 Explore P: 0.8852\n",
      "Episode: 60 Total reward: 19.0 Training loss: 204.7924 Explore P: 0.8836\n",
      "Episode: 61 Total reward: 12.0 Training loss: 8.8831 Explore P: 0.8825\n",
      "Episode: 62 Total reward: 22.0 Training loss: 98.7600 Explore P: 0.8806\n",
      "Episode: 63 Total reward: 27.0 Training loss: 58.3895 Explore P: 0.8783\n",
      "Episode: 64 Total reward: 8.0 Training loss: 12.6082 Explore P: 0.8776\n",
      "Episode: 65 Total reward: 35.0 Training loss: 135.2939 Explore P: 0.8745\n",
      "Episode: 66 Total reward: 13.0 Training loss: 155.6817 Explore P: 0.8734\n",
      "Episode: 67 Total reward: 9.0 Training loss: 13.2349 Explore P: 0.8726\n",
      "Episode: 68 Total reward: 21.0 Training loss: 14.0729 Explore P: 0.8708\n",
      "Episode: 69 Total reward: 16.0 Training loss: 264.4030 Explore P: 0.8695\n",
      "Episode: 70 Total reward: 12.0 Training loss: 120.5529 Explore P: 0.8684\n",
      "Episode: 71 Total reward: 26.0 Training loss: 10.4504 Explore P: 0.8662\n",
      "Episode: 72 Total reward: 14.0 Training loss: 11.2306 Explore P: 0.8650\n",
      "Episode: 73 Total reward: 10.0 Training loss: 10.1919 Explore P: 0.8641\n",
      "Episode: 74 Total reward: 34.0 Training loss: 12.1061 Explore P: 0.8612\n",
      "Episode: 75 Total reward: 10.0 Training loss: 12.0729 Explore P: 0.8604\n",
      "Episode: 76 Total reward: 21.0 Training loss: 183.3339 Explore P: 0.8586\n",
      "Episode: 77 Total reward: 13.0 Training loss: 78.0409 Explore P: 0.8575\n",
      "Episode: 78 Total reward: 22.0 Training loss: 9.5751 Explore P: 0.8556\n",
      "Episode: 79 Total reward: 18.0 Training loss: 9.4819 Explore P: 0.8541\n",
      "Episode: 80 Total reward: 32.0 Training loss: 7.5449 Explore P: 0.8514\n",
      "Episode: 81 Total reward: 22.0 Training loss: 80.0893 Explore P: 0.8496\n",
      "Episode: 82 Total reward: 18.0 Training loss: 83.2184 Explore P: 0.8481\n",
      "Episode: 83 Total reward: 12.0 Training loss: 538.6312 Explore P: 0.8471\n",
      "Episode: 84 Total reward: 27.0 Training loss: 70.5857 Explore P: 0.8448\n",
      "Episode: 85 Total reward: 24.0 Training loss: 75.8883 Explore P: 0.8428\n",
      "Episode: 86 Total reward: 18.0 Training loss: 8.4785 Explore P: 0.8413\n",
      "Episode: 87 Total reward: 20.0 Training loss: 154.0514 Explore P: 0.8397\n",
      "Episode: 88 Total reward: 10.0 Training loss: 86.2317 Explore P: 0.8388\n",
      "Episode: 89 Total reward: 28.0 Training loss: 366.9674 Explore P: 0.8365\n",
      "Episode: 90 Total reward: 18.0 Training loss: 421.3952 Explore P: 0.8350\n",
      "Episode: 91 Total reward: 15.0 Training loss: 7.8214 Explore P: 0.8338\n",
      "Episode: 92 Total reward: 24.0 Training loss: 6.3736 Explore P: 0.8318\n",
      "Episode: 93 Total reward: 7.0 Training loss: 204.4367 Explore P: 0.8312\n",
      "Episode: 94 Total reward: 17.0 Training loss: 159.3862 Explore P: 0.8298\n",
      "Episode: 95 Total reward: 29.0 Training loss: 7.0987 Explore P: 0.8275\n",
      "Episode: 96 Total reward: 21.0 Training loss: 85.1140 Explore P: 0.8257\n",
      "Episode: 97 Total reward: 16.0 Training loss: 153.7276 Explore P: 0.8244\n",
      "Episode: 98 Total reward: 71.0 Training loss: 157.2853 Explore P: 0.8187\n",
      "Episode: 99 Total reward: 26.0 Training loss: 336.9306 Explore P: 0.8166\n",
      "Episode: 100 Total reward: 21.0 Training loss: 80.6328 Explore P: 0.8149\n",
      "Episode: 101 Total reward: 19.0 Training loss: 5.9293 Explore P: 0.8134\n",
      "Episode: 102 Total reward: 9.0 Training loss: 99.6739 Explore P: 0.8126\n",
      "Episode: 103 Total reward: 14.0 Training loss: 4.1712 Explore P: 0.8115\n",
      "Episode: 104 Total reward: 12.0 Training loss: 84.1356 Explore P: 0.8106\n",
      "Episode: 105 Total reward: 28.0 Training loss: 255.8639 Explore P: 0.8083\n",
      "Episode: 106 Total reward: 15.0 Training loss: 5.2189 Explore P: 0.8071\n",
      "Episode: 107 Total reward: 15.0 Training loss: 178.7370 Explore P: 0.8059\n",
      "Episode: 108 Total reward: 17.0 Training loss: 5.8840 Explore P: 0.8046\n",
      "Episode: 109 Total reward: 7.0 Training loss: 5.4816 Explore P: 0.8040\n",
      "Episode: 110 Total reward: 12.0 Training loss: 4.2377 Explore P: 0.8031\n",
      "Episode: 111 Total reward: 31.0 Training loss: 4.4330 Explore P: 0.8006\n",
      "Episode: 112 Total reward: 14.0 Training loss: 399.0104 Explore P: 0.7995\n",
      "Episode: 113 Total reward: 31.0 Training loss: 5.8449 Explore P: 0.7971\n",
      "Episode: 114 Total reward: 13.0 Training loss: 76.9266 Explore P: 0.7960\n",
      "Episode: 115 Total reward: 45.0 Training loss: 5.5173 Explore P: 0.7925\n",
      "Episode: 116 Total reward: 13.0 Training loss: 304.6474 Explore P: 0.7915\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 117 Total reward: 24.0 Training loss: 119.8353 Explore P: 0.7896\n",
      "Episode: 118 Total reward: 26.0 Training loss: 76.8118 Explore P: 0.7876\n",
      "Episode: 119 Total reward: 17.0 Training loss: 79.7945 Explore P: 0.7863\n",
      "Episode: 120 Total reward: 11.0 Training loss: 123.1683 Explore P: 0.7854\n",
      "Episode: 121 Total reward: 21.0 Training loss: 178.7022 Explore P: 0.7838\n",
      "Episode: 122 Total reward: 23.0 Training loss: 81.9697 Explore P: 0.7820\n",
      "Episode: 123 Total reward: 11.0 Training loss: 127.7104 Explore P: 0.7812\n",
      "Episode: 124 Total reward: 10.0 Training loss: 3.5623 Explore P: 0.7804\n",
      "Episode: 125 Total reward: 20.0 Training loss: 3.0939 Explore P: 0.7789\n",
      "Episode: 126 Total reward: 7.0 Training loss: 101.6192 Explore P: 0.7783\n",
      "Episode: 127 Total reward: 15.0 Training loss: 276.9601 Explore P: 0.7772\n",
      "Episode: 128 Total reward: 11.0 Training loss: 2.9213 Explore P: 0.7763\n",
      "Episode: 129 Total reward: 20.0 Training loss: 70.7553 Explore P: 0.7748\n",
      "Episode: 130 Total reward: 14.0 Training loss: 151.4538 Explore P: 0.7737\n",
      "Episode: 131 Total reward: 14.0 Training loss: 116.8299 Explore P: 0.7727\n",
      "Episode: 132 Total reward: 28.0 Training loss: 100.9106 Explore P: 0.7705\n",
      "Episode: 133 Total reward: 15.0 Training loss: 89.9882 Explore P: 0.7694\n",
      "Episode: 134 Total reward: 15.0 Training loss: 96.3701 Explore P: 0.7682\n",
      "Episode: 135 Total reward: 17.0 Training loss: 355.0692 Explore P: 0.7670\n",
      "Episode: 136 Total reward: 11.0 Training loss: 2.4984 Explore P: 0.7661\n",
      "Episode: 137 Total reward: 13.0 Training loss: 2.3700 Explore P: 0.7651\n",
      "Episode: 138 Total reward: 14.0 Training loss: 66.3114 Explore P: 0.7641\n",
      "Episode: 139 Total reward: 25.0 Training loss: 2.5515 Explore P: 0.7622\n",
      "Episode: 140 Total reward: 23.0 Training loss: 1.9765 Explore P: 0.7605\n",
      "Episode: 141 Total reward: 13.0 Training loss: 219.6210 Explore P: 0.7595\n",
      "Episode: 142 Total reward: 15.0 Training loss: 2.0553 Explore P: 0.7584\n",
      "Episode: 143 Total reward: 14.0 Training loss: 72.2972 Explore P: 0.7573\n",
      "Episode: 144 Total reward: 19.0 Training loss: 66.4142 Explore P: 0.7559\n",
      "Episode: 145 Total reward: 13.0 Training loss: 179.7241 Explore P: 0.7549\n",
      "Episode: 146 Total reward: 19.0 Training loss: 2.1647 Explore P: 0.7535\n",
      "Episode: 147 Total reward: 19.0 Training loss: 61.3489 Explore P: 0.7521\n",
      "Episode: 148 Total reward: 10.0 Training loss: 181.4843 Explore P: 0.7514\n",
      "Episode: 149 Total reward: 22.0 Training loss: 2.4541 Explore P: 0.7497\n",
      "Episode: 150 Total reward: 22.0 Training loss: 2.0248 Explore P: 0.7481\n",
      "Episode: 151 Total reward: 17.0 Training loss: 2.6169 Explore P: 0.7469\n",
      "Episode: 152 Total reward: 20.0 Training loss: 62.3719 Explore P: 0.7454\n",
      "Episode: 153 Total reward: 21.0 Training loss: 183.4442 Explore P: 0.7439\n",
      "Episode: 154 Total reward: 12.0 Training loss: 1.9759 Explore P: 0.7430\n",
      "Episode: 155 Total reward: 16.0 Training loss: 120.7015 Explore P: 0.7418\n",
      "Episode: 156 Total reward: 16.0 Training loss: 52.9175 Explore P: 0.7406\n",
      "Episode: 157 Total reward: 40.0 Training loss: 119.3727 Explore P: 0.7377\n",
      "Episode: 158 Total reward: 14.0 Training loss: 2.1258 Explore P: 0.7367\n",
      "Episode: 159 Total reward: 27.0 Training loss: 1.4107 Explore P: 0.7347\n",
      "Episode: 160 Total reward: 37.0 Training loss: 49.5152 Explore P: 0.7321\n",
      "Episode: 161 Total reward: 36.0 Training loss: 1.3071 Explore P: 0.7295\n",
      "Episode: 162 Total reward: 15.0 Training loss: 1.1182 Explore P: 0.7284\n",
      "Episode: 163 Total reward: 34.0 Training loss: 0.8964 Explore P: 0.7259\n",
      "Episode: 164 Total reward: 21.0 Training loss: 44.8059 Explore P: 0.7244\n",
      "Episode: 165 Total reward: 16.0 Training loss: 107.4909 Explore P: 0.7233\n",
      "Episode: 166 Total reward: 27.0 Training loss: 47.6570 Explore P: 0.7214\n",
      "Episode: 167 Total reward: 11.0 Training loss: 1.3468 Explore P: 0.7206\n",
      "Episode: 168 Total reward: 18.0 Training loss: 1.4070 Explore P: 0.7193\n",
      "Episode: 169 Total reward: 9.0 Training loss: 1.1480 Explore P: 0.7187\n",
      "Episode: 170 Total reward: 24.0 Training loss: 1.4720 Explore P: 0.7170\n",
      "Episode: 171 Total reward: 11.0 Training loss: 75.9982 Explore P: 0.7162\n",
      "Episode: 172 Total reward: 9.0 Training loss: 52.6668 Explore P: 0.7156\n",
      "Episode: 173 Total reward: 25.0 Training loss: 36.2106 Explore P: 0.7138\n",
      "Episode: 174 Total reward: 13.0 Training loss: 91.3215 Explore P: 0.7129\n",
      "Episode: 175 Total reward: 10.0 Training loss: 120.4119 Explore P: 0.7122\n",
      "Episode: 176 Total reward: 22.0 Training loss: 33.6921 Explore P: 0.7106\n",
      "Episode: 177 Total reward: 11.0 Training loss: 1.3823 Explore P: 0.7099\n",
      "Episode: 178 Total reward: 18.0 Training loss: 33.0225 Explore P: 0.7086\n",
      "Episode: 179 Total reward: 12.0 Training loss: 1.1151 Explore P: 0.7078\n",
      "Episode: 180 Total reward: 22.0 Training loss: 0.9773 Explore P: 0.7062\n",
      "Episode: 181 Total reward: 24.0 Training loss: 50.0517 Explore P: 0.7046\n",
      "Episode: 182 Total reward: 8.0 Training loss: 95.6576 Explore P: 0.7040\n",
      "Episode: 183 Total reward: 17.0 Training loss: 89.3774 Explore P: 0.7028\n",
      "Episode: 184 Total reward: 11.0 Training loss: 41.7725 Explore P: 0.7021\n",
      "Episode: 185 Total reward: 18.0 Training loss: 1.2328 Explore P: 0.7008\n",
      "Episode: 186 Total reward: 14.0 Training loss: 27.1881 Explore P: 0.6999\n",
      "Episode: 187 Total reward: 13.0 Training loss: 1.5340 Explore P: 0.6990\n",
      "Episode: 188 Total reward: 18.0 Training loss: 60.5353 Explore P: 0.6977\n",
      "Episode: 189 Total reward: 18.0 Training loss: 2.3637 Explore P: 0.6965\n",
      "Episode: 190 Total reward: 12.0 Training loss: 24.2581 Explore P: 0.6957\n",
      "Episode: 191 Total reward: 13.0 Training loss: 1.7350 Explore P: 0.6948\n",
      "Episode: 192 Total reward: 17.0 Training loss: 1.3911 Explore P: 0.6936\n",
      "Episode: 193 Total reward: 21.0 Training loss: 44.7350 Explore P: 0.6922\n",
      "Episode: 194 Total reward: 12.0 Training loss: 26.6191 Explore P: 0.6914\n",
      "Episode: 195 Total reward: 21.0 Training loss: 99.3264 Explore P: 0.6899\n",
      "Episode: 196 Total reward: 18.0 Training loss: 1.8306 Explore P: 0.6887\n",
      "Episode: 197 Total reward: 37.0 Training loss: 34.2061 Explore P: 0.6862\n",
      "Episode: 198 Total reward: 14.0 Training loss: 2.2357 Explore P: 0.6853\n",
      "Episode: 199 Total reward: 17.0 Training loss: 39.7538 Explore P: 0.6841\n",
      "Episode: 200 Total reward: 19.0 Training loss: 2.3696 Explore P: 0.6828\n",
      "Episode: 201 Total reward: 10.0 Training loss: 1.6859 Explore P: 0.6822\n",
      "Episode: 202 Total reward: 10.0 Training loss: 2.6554 Explore P: 0.6815\n",
      "Episode: 203 Total reward: 32.0 Training loss: 35.6828 Explore P: 0.6793\n",
      "Episode: 204 Total reward: 23.0 Training loss: 80.4351 Explore P: 0.6778\n",
      "Episode: 205 Total reward: 35.0 Training loss: 107.0616 Explore P: 0.6755\n",
      "Episode: 206 Total reward: 46.0 Training loss: 73.0960 Explore P: 0.6724\n",
      "Episode: 207 Total reward: 12.0 Training loss: 21.8300 Explore P: 0.6716\n",
      "Episode: 208 Total reward: 14.0 Training loss: 1.7546 Explore P: 0.6707\n",
      "Episode: 209 Total reward: 23.0 Training loss: 104.7136 Explore P: 0.6692\n",
      "Episode: 210 Total reward: 11.0 Training loss: 80.2424 Explore P: 0.6685\n",
      "Episode: 211 Total reward: 62.0 Training loss: 60.5992 Explore P: 0.6644\n",
      "Episode: 212 Total reward: 14.0 Training loss: 23.9182 Explore P: 0.6635\n",
      "Episode: 213 Total reward: 29.0 Training loss: 37.9362 Explore P: 0.6616\n",
      "Episode: 214 Total reward: 37.0 Training loss: 1.3622 Explore P: 0.6592\n",
      "Episode: 215 Total reward: 11.0 Training loss: 20.3886 Explore P: 0.6585\n",
      "Episode: 216 Total reward: 12.0 Training loss: 1.0421 Explore P: 0.6577\n",
      "Episode: 217 Total reward: 25.0 Training loss: 67.0008 Explore P: 0.6561\n",
      "Episode: 218 Total reward: 14.0 Training loss: 56.9089 Explore P: 0.6552\n",
      "Episode: 219 Total reward: 23.0 Training loss: 0.9590 Explore P: 0.6537\n",
      "Episode: 220 Total reward: 39.0 Training loss: 56.2006 Explore P: 0.6512\n",
      "Episode: 221 Total reward: 25.0 Training loss: 41.1856 Explore P: 0.6496\n",
      "Episode: 222 Total reward: 11.0 Training loss: 32.5857 Explore P: 0.6489\n",
      "Episode: 223 Total reward: 49.0 Training loss: 22.5871 Explore P: 0.6457\n",
      "Episode: 224 Total reward: 25.0 Training loss: 16.5371 Explore P: 0.6442\n",
      "Episode: 225 Total reward: 15.0 Training loss: 31.2203 Explore P: 0.6432\n",
      "Episode: 226 Total reward: 31.0 Training loss: 43.6499 Explore P: 0.6413\n",
      "Episode: 227 Total reward: 20.0 Training loss: 34.4391 Explore P: 0.6400\n",
      "Episode: 228 Total reward: 25.0 Training loss: 2.1587 Explore P: 0.6384\n",
      "Episode: 229 Total reward: 53.0 Training loss: 1.5587 Explore P: 0.6351\n",
      "Episode: 230 Total reward: 22.0 Training loss: 1.4342 Explore P: 0.6337\n",
      "Episode: 231 Total reward: 12.0 Training loss: 18.8004 Explore P: 0.6330\n",
      "Episode: 232 Total reward: 32.0 Training loss: 33.5512 Explore P: 0.6310\n",
      "Episode: 233 Total reward: 14.0 Training loss: 38.5225 Explore P: 0.6301\n",
      "Episode: 234 Total reward: 14.0 Training loss: 1.3732 Explore P: 0.6292\n",
      "Episode: 235 Total reward: 20.0 Training loss: 34.8787 Explore P: 0.6280\n",
      "Episode: 236 Total reward: 12.0 Training loss: 17.1002 Explore P: 0.6273\n",
      "Episode: 237 Total reward: 22.0 Training loss: 41.2450 Explore P: 0.6259\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 238 Total reward: 13.0 Training loss: 0.9780 Explore P: 0.6251\n",
      "Episode: 239 Total reward: 45.0 Training loss: 19.1864 Explore P: 0.6224\n",
      "Episode: 240 Total reward: 23.0 Training loss: 20.5513 Explore P: 0.6209\n",
      "Episode: 241 Total reward: 17.0 Training loss: 21.7457 Explore P: 0.6199\n",
      "Episode: 242 Total reward: 17.0 Training loss: 16.3265 Explore P: 0.6189\n",
      "Episode: 243 Total reward: 18.0 Training loss: 30.8824 Explore P: 0.6178\n",
      "Episode: 244 Total reward: 18.0 Training loss: 14.7986 Explore P: 0.6167\n",
      "Episode: 245 Total reward: 16.0 Training loss: 18.0239 Explore P: 0.6157\n",
      "Episode: 246 Total reward: 42.0 Training loss: 1.4199 Explore P: 0.6132\n",
      "Episode: 247 Total reward: 21.0 Training loss: 0.9784 Explore P: 0.6119\n",
      "Episode: 248 Total reward: 63.0 Training loss: 1.2571 Explore P: 0.6081\n",
      "Episode: 249 Total reward: 26.0 Training loss: 18.1063 Explore P: 0.6066\n",
      "Episode: 250 Total reward: 14.0 Training loss: 24.2054 Explore P: 0.6057\n",
      "Episode: 251 Total reward: 24.0 Training loss: 34.1749 Explore P: 0.6043\n",
      "Episode: 252 Total reward: 64.0 Training loss: 1.1932 Explore P: 0.6005\n",
      "Episode: 253 Total reward: 17.0 Training loss: 1.3697 Explore P: 0.5995\n",
      "Episode: 254 Total reward: 102.0 Training loss: 2.8539 Explore P: 0.5935\n",
      "Episode: 255 Total reward: 123.0 Training loss: 28.7468 Explore P: 0.5864\n",
      "Episode: 256 Total reward: 30.0 Training loss: 35.0689 Explore P: 0.5847\n",
      "Episode: 257 Total reward: 28.0 Training loss: 2.2217 Explore P: 0.5831\n",
      "Episode: 258 Total reward: 23.0 Training loss: 0.8825 Explore P: 0.5818\n",
      "Episode: 259 Total reward: 24.0 Training loss: 35.6616 Explore P: 0.5804\n",
      "Episode: 260 Total reward: 16.0 Training loss: 19.3981 Explore P: 0.5795\n",
      "Episode: 261 Total reward: 21.0 Training loss: 31.3186 Explore P: 0.5783\n",
      "Episode: 262 Total reward: 19.0 Training loss: 72.2449 Explore P: 0.5772\n",
      "Episode: 263 Total reward: 25.0 Training loss: 20.6708 Explore P: 0.5758\n",
      "Episode: 264 Total reward: 22.0 Training loss: 12.3732 Explore P: 0.5745\n",
      "Episode: 265 Total reward: 40.0 Training loss: 53.1285 Explore P: 0.5723\n",
      "Episode: 266 Total reward: 26.0 Training loss: 0.9722 Explore P: 0.5708\n",
      "Episode: 267 Total reward: 21.0 Training loss: 14.0547 Explore P: 0.5696\n",
      "Episode: 268 Total reward: 42.0 Training loss: 21.1848 Explore P: 0.5673\n",
      "Episode: 269 Total reward: 17.0 Training loss: 1.1384 Explore P: 0.5664\n",
      "Episode: 270 Total reward: 16.0 Training loss: 1.4366 Explore P: 0.5655\n",
      "Episode: 271 Total reward: 64.0 Training loss: 18.4542 Explore P: 0.5619\n",
      "Episode: 272 Total reward: 29.0 Training loss: 35.8770 Explore P: 0.5603\n",
      "Episode: 273 Total reward: 42.0 Training loss: 18.5974 Explore P: 0.5580\n",
      "Episode: 274 Total reward: 22.0 Training loss: 66.6930 Explore P: 0.5568\n",
      "Episode: 275 Total reward: 27.0 Training loss: 18.1106 Explore P: 0.5553\n",
      "Episode: 276 Total reward: 15.0 Training loss: 2.1875 Explore P: 0.5545\n",
      "Episode: 277 Total reward: 14.0 Training loss: 14.3765 Explore P: 0.5538\n",
      "Episode: 278 Total reward: 22.0 Training loss: 1.5512 Explore P: 0.5526\n",
      "Episode: 279 Total reward: 66.0 Training loss: 20.2266 Explore P: 0.5490\n",
      "Episode: 280 Total reward: 95.0 Training loss: 47.3960 Explore P: 0.5439\n",
      "Episode: 281 Total reward: 33.0 Training loss: 2.1813 Explore P: 0.5421\n",
      "Episode: 282 Total reward: 43.0 Training loss: 1.2044 Explore P: 0.5399\n",
      "Episode: 283 Total reward: 15.0 Training loss: 15.3530 Explore P: 0.5391\n",
      "Episode: 284 Total reward: 56.0 Training loss: 2.0650 Explore P: 0.5361\n",
      "Episode: 285 Total reward: 25.0 Training loss: 29.9267 Explore P: 0.5348\n",
      "Episode: 286 Total reward: 13.0 Training loss: 1.9375 Explore P: 0.5341\n",
      "Episode: 287 Total reward: 31.0 Training loss: 22.7180 Explore P: 0.5325\n",
      "Episode: 288 Total reward: 28.0 Training loss: 1.8125 Explore P: 0.5310\n",
      "Episode: 289 Total reward: 23.0 Training loss: 26.1822 Explore P: 0.5298\n",
      "Episode: 290 Total reward: 18.0 Training loss: 1.6541 Explore P: 0.5289\n",
      "Episode: 291 Total reward: 39.0 Training loss: 36.6711 Explore P: 0.5269\n",
      "Episode: 292 Total reward: 113.0 Training loss: 87.3130 Explore P: 0.5211\n",
      "Episode: 293 Total reward: 27.0 Training loss: 2.4138 Explore P: 0.5197\n",
      "Episode: 294 Total reward: 49.0 Training loss: 26.9535 Explore P: 0.5172\n",
      "Episode: 295 Total reward: 29.0 Training loss: 1.5142 Explore P: 0.5157\n",
      "Episode: 296 Total reward: 29.0 Training loss: 62.9564 Explore P: 0.5143\n",
      "Episode: 297 Total reward: 26.0 Training loss: 26.0655 Explore P: 0.5130\n",
      "Episode: 298 Total reward: 17.0 Training loss: 47.9946 Explore P: 0.5121\n",
      "Episode: 299 Total reward: 29.0 Training loss: 16.6444 Explore P: 0.5106\n",
      "Episode: 300 Total reward: 41.0 Training loss: 2.4643 Explore P: 0.5086\n",
      "Episode: 301 Total reward: 22.0 Training loss: 123.1417 Explore P: 0.5075\n",
      "Episode: 302 Total reward: 20.0 Training loss: 21.2678 Explore P: 0.5065\n",
      "Episode: 303 Total reward: 14.0 Training loss: 18.7001 Explore P: 0.5058\n",
      "Episode: 304 Total reward: 33.0 Training loss: 131.2409 Explore P: 0.5042\n",
      "Episode: 305 Total reward: 101.0 Training loss: 2.1386 Explore P: 0.4992\n",
      "Episode: 306 Total reward: 72.0 Training loss: 21.2177 Explore P: 0.4957\n",
      "Episode: 307 Total reward: 120.0 Training loss: 1.3964 Explore P: 0.4899\n",
      "Episode: 308 Total reward: 38.0 Training loss: 0.8197 Explore P: 0.4881\n",
      "Episode: 309 Total reward: 27.0 Training loss: 1.5250 Explore P: 0.4868\n",
      "Episode: 310 Total reward: 57.0 Training loss: 57.8280 Explore P: 0.4841\n",
      "Episode: 311 Total reward: 47.0 Training loss: 2.6075 Explore P: 0.4819\n",
      "Episode: 312 Total reward: 73.0 Training loss: 22.4005 Explore P: 0.4784\n",
      "Episode: 313 Total reward: 48.0 Training loss: 1.5562 Explore P: 0.4762\n",
      "Episode: 314 Total reward: 52.0 Training loss: 26.6713 Explore P: 0.4738\n",
      "Episode: 315 Total reward: 48.0 Training loss: 2.9655 Explore P: 0.4716\n",
      "Episode: 316 Total reward: 52.0 Training loss: 60.8128 Explore P: 0.4692\n",
      "Episode: 317 Total reward: 92.0 Training loss: 1.5835 Explore P: 0.4650\n",
      "Episode: 318 Total reward: 63.0 Training loss: 2.2054 Explore P: 0.4621\n",
      "Episode: 319 Total reward: 41.0 Training loss: 28.5544 Explore P: 0.4603\n",
      "Episode: 320 Total reward: 41.0 Training loss: 46.1782 Explore P: 0.4584\n",
      "Episode: 321 Total reward: 48.0 Training loss: 53.3134 Explore P: 0.4563\n",
      "Episode: 322 Total reward: 139.0 Training loss: 2.0955 Explore P: 0.4501\n",
      "Episode: 323 Total reward: 68.0 Training loss: 0.6024 Explore P: 0.4471\n",
      "Episode: 324 Total reward: 30.0 Training loss: 76.6272 Explore P: 0.4458\n",
      "Episode: 325 Total reward: 64.0 Training loss: 1.6955 Explore P: 0.4430\n",
      "Episode: 326 Total reward: 76.0 Training loss: 41.4120 Explore P: 0.4398\n",
      "Episode: 327 Total reward: 46.0 Training loss: 1.7411 Explore P: 0.4378\n",
      "Episode: 328 Total reward: 60.0 Training loss: 54.4557 Explore P: 0.4352\n",
      "Episode: 329 Total reward: 51.0 Training loss: 1.3942 Explore P: 0.4331\n",
      "Episode: 330 Total reward: 96.0 Training loss: 3.4435 Explore P: 0.4290\n",
      "Episode: 331 Total reward: 36.0 Training loss: 48.3144 Explore P: 0.4275\n",
      "Episode: 332 Total reward: 55.0 Training loss: 21.6347 Explore P: 0.4252\n",
      "Episode: 333 Total reward: 18.0 Training loss: 47.6124 Explore P: 0.4245\n",
      "Episode: 334 Total reward: 43.0 Training loss: 152.7848 Explore P: 0.4227\n",
      "Episode: 335 Total reward: 30.0 Training loss: 33.6328 Explore P: 0.4215\n",
      "Episode: 336 Total reward: 67.0 Training loss: 2.9801 Explore P: 0.4187\n",
      "Episode: 337 Total reward: 121.0 Training loss: 3.1729 Explore P: 0.4138\n",
      "Episode: 338 Total reward: 42.0 Training loss: 45.4624 Explore P: 0.4121\n",
      "Episode: 339 Total reward: 37.0 Training loss: 55.9991 Explore P: 0.4106\n",
      "Episode: 340 Total reward: 57.0 Training loss: 2.2636 Explore P: 0.4083\n",
      "Episode: 341 Total reward: 90.0 Training loss: 1.2559 Explore P: 0.4048\n",
      "Episode: 342 Total reward: 117.0 Training loss: 48.2137 Explore P: 0.4002\n",
      "Episode: 343 Total reward: 68.0 Training loss: 34.2201 Explore P: 0.3975\n",
      "Episode: 344 Total reward: 77.0 Training loss: 30.8821 Explore P: 0.3946\n",
      "Episode: 345 Total reward: 46.0 Training loss: 53.2726 Explore P: 0.3928\n",
      "Episode: 346 Total reward: 171.0 Training loss: 53.7899 Explore P: 0.3863\n",
      "Episode: 347 Total reward: 77.0 Training loss: 32.5015 Explore P: 0.3834\n",
      "Episode: 348 Total reward: 54.0 Training loss: 25.4479 Explore P: 0.3814\n",
      "Episode: 349 Total reward: 73.0 Training loss: 84.2978 Explore P: 0.3787\n",
      "Episode: 350 Total reward: 81.0 Training loss: 29.8416 Explore P: 0.3757\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 351 Total reward: 105.0 Training loss: 2.1272 Explore P: 0.3719\n",
      "Episode: 352 Total reward: 79.0 Training loss: 75.6897 Explore P: 0.3691\n",
      "Episode: 353 Total reward: 93.0 Training loss: 84.0930 Explore P: 0.3657\n",
      "Episode: 354 Total reward: 112.0 Training loss: 42.4233 Explore P: 0.3618\n",
      "Episode: 355 Total reward: 138.0 Training loss: 38.0009 Explore P: 0.3570\n",
      "Episode: 356 Total reward: 199.0 Training loss: 42.0065 Explore P: 0.3501\n",
      "Episode: 357 Total reward: 163.0 Training loss: 1.7738 Explore P: 0.3446\n",
      "Episode: 358 Total reward: 81.0 Training loss: 140.9275 Explore P: 0.3419\n",
      "Episode: 359 Total reward: 157.0 Training loss: 2.7810 Explore P: 0.3368\n",
      "Episode: 360 Total reward: 73.0 Training loss: 6.3305 Explore P: 0.3344\n",
      "Episode: 361 Total reward: 60.0 Training loss: 72.9790 Explore P: 0.3324\n",
      "Episode: 362 Total reward: 118.0 Training loss: 2.2914 Explore P: 0.3287\n",
      "Episode: 363 Total reward: 199.0 Training loss: 4.6469 Explore P: 0.3224\n",
      "Episode: 364 Total reward: 199.0 Training loss: 46.7155 Explore P: 0.3162\n",
      "Episode: 365 Total reward: 151.0 Training loss: 87.9453 Explore P: 0.3116\n",
      "Episode: 366 Total reward: 199.0 Training loss: 17.5587 Explore P: 0.3057\n",
      "Episode: 367 Total reward: 199.0 Training loss: 2.9894 Explore P: 0.2999\n",
      "Episode: 368 Total reward: 199.0 Training loss: 39.6621 Explore P: 0.2942\n",
      "Episode: 369 Total reward: 198.0 Training loss: 2.5999 Explore P: 0.2886\n",
      "Episode: 370 Total reward: 199.0 Training loss: 91.5248 Explore P: 0.2831\n",
      "Episode: 371 Total reward: 184.0 Training loss: 2.9430 Explore P: 0.2781\n",
      "Episode: 372 Total reward: 199.0 Training loss: 4.8893 Explore P: 0.2728\n",
      "Episode: 373 Total reward: 199.0 Training loss: 6.9474 Explore P: 0.2677\n",
      "Episode: 374 Total reward: 199.0 Training loss: 52.8340 Explore P: 0.2626\n",
      "Episode: 375 Total reward: 199.0 Training loss: 4.5890 Explore P: 0.2576\n",
      "Episode: 376 Total reward: 199.0 Training loss: 2.0561 Explore P: 0.2527\n",
      "Episode: 377 Total reward: 179.0 Training loss: 1.1039 Explore P: 0.2484\n",
      "Episode: 378 Total reward: 152.0 Training loss: 0.9807 Explore P: 0.2448\n",
      "Episode: 379 Total reward: 157.0 Training loss: 2.5560 Explore P: 0.2412\n",
      "Episode: 380 Total reward: 154.0 Training loss: 0.8035 Explore P: 0.2376\n",
      "Episode: 381 Total reward: 92.0 Training loss: 168.1537 Explore P: 0.2355\n",
      "Episode: 382 Total reward: 117.0 Training loss: 3.8320 Explore P: 0.2329\n",
      "Episode: 383 Total reward: 35.0 Training loss: 1.9793 Explore P: 0.2321\n",
      "Episode: 384 Total reward: 62.0 Training loss: 73.9128 Explore P: 0.2308\n",
      "Episode: 385 Total reward: 81.0 Training loss: 3.3801 Explore P: 0.2290\n",
      "Episode: 386 Total reward: 47.0 Training loss: 0.9748 Explore P: 0.2280\n",
      "Episode: 387 Total reward: 82.0 Training loss: 2.2721 Explore P: 0.2262\n",
      "Episode: 388 Total reward: 86.0 Training loss: 437.4698 Explore P: 0.2243\n",
      "Episode: 389 Total reward: 91.0 Training loss: 2.6066 Explore P: 0.2224\n",
      "Episode: 390 Total reward: 115.0 Training loss: 1.9224 Explore P: 0.2200\n",
      "Episode: 391 Total reward: 91.0 Training loss: 1.7991 Explore P: 0.2181\n",
      "Episode: 392 Total reward: 91.0 Training loss: 0.8765 Explore P: 0.2162\n",
      "Episode: 393 Total reward: 46.0 Training loss: 1.0425 Explore P: 0.2152\n",
      "Episode: 394 Total reward: 41.0 Training loss: 2.9720 Explore P: 0.2144\n",
      "Episode: 395 Total reward: 77.0 Training loss: 291.0005 Explore P: 0.2128\n",
      "Episode: 396 Total reward: 65.0 Training loss: 58.6118 Explore P: 0.2115\n",
      "Episode: 397 Total reward: 51.0 Training loss: 427.4689 Explore P: 0.2105\n",
      "Episode: 398 Total reward: 89.0 Training loss: 86.2140 Explore P: 0.2087\n",
      "Episode: 399 Total reward: 66.0 Training loss: 3.7241 Explore P: 0.2074\n",
      "Episode: 400 Total reward: 35.0 Training loss: 4.7065 Explore P: 0.2067\n",
      "Episode: 401 Total reward: 42.0 Training loss: 2.5742 Explore P: 0.2059\n",
      "Episode: 402 Total reward: 58.0 Training loss: 144.4817 Explore P: 0.2047\n",
      "Episode: 403 Total reward: 130.0 Training loss: 2.9369 Explore P: 0.2022\n",
      "Episode: 404 Total reward: 199.0 Training loss: 336.2489 Explore P: 0.1984\n",
      "Episode: 405 Total reward: 191.0 Training loss: 125.1122 Explore P: 0.1949\n",
      "Episode: 406 Total reward: 199.0 Training loss: 3.0739 Explore P: 0.1912\n",
      "Episode: 407 Total reward: 199.0 Training loss: 2.9937 Explore P: 0.1877\n",
      "Episode: 408 Total reward: 199.0 Training loss: 2.5133 Explore P: 0.1842\n",
      "Episode: 409 Total reward: 199.0 Training loss: 1.6691 Explore P: 0.1807\n",
      "Episode: 410 Total reward: 199.0 Training loss: 52.9792 Explore P: 0.1774\n",
      "Episode: 411 Total reward: 199.0 Training loss: 1.0437 Explore P: 0.1741\n",
      "Episode: 412 Total reward: 199.0 Training loss: 2.6758 Explore P: 0.1708\n",
      "Episode: 413 Total reward: 199.0 Training loss: 1.7746 Explore P: 0.1677\n",
      "Episode: 414 Total reward: 199.0 Training loss: 2.8530 Explore P: 0.1646\n",
      "Episode: 415 Total reward: 199.0 Training loss: 1.3817 Explore P: 0.1615\n",
      "Episode: 416 Total reward: 199.0 Training loss: 0.6273 Explore P: 0.1585\n",
      "Episode: 417 Total reward: 199.0 Training loss: 1.6934 Explore P: 0.1556\n",
      "Episode: 418 Total reward: 199.0 Training loss: 2.5464 Explore P: 0.1527\n",
      "Episode: 419 Total reward: 199.0 Training loss: 1.3971 Explore P: 0.1499\n",
      "Episode: 420 Total reward: 199.0 Training loss: 1.1567 Explore P: 0.1472\n",
      "Episode: 421 Total reward: 199.0 Training loss: 0.4862 Explore P: 0.1445\n",
      "Episode: 422 Total reward: 199.0 Training loss: 0.5851 Explore P: 0.1418\n",
      "Episode: 423 Total reward: 199.0 Training loss: 267.4133 Explore P: 0.1392\n",
      "Episode: 424 Total reward: 199.0 Training loss: 45.6843 Explore P: 0.1367\n",
      "Episode: 425 Total reward: 199.0 Training loss: 0.7413 Explore P: 0.1342\n",
      "Episode: 426 Total reward: 199.0 Training loss: 1.2148 Explore P: 0.1317\n",
      "Episode: 427 Total reward: 199.0 Training loss: 1.0496 Explore P: 0.1293\n",
      "Episode: 428 Total reward: 199.0 Training loss: 0.4405 Explore P: 0.1270\n",
      "Episode: 429 Total reward: 199.0 Training loss: 0.7047 Explore P: 0.1247\n",
      "Episode: 430 Total reward: 199.0 Training loss: 0.4062 Explore P: 0.1224\n",
      "Episode: 431 Total reward: 199.0 Training loss: 0.2243 Explore P: 0.1202\n",
      "Episode: 432 Total reward: 199.0 Training loss: 46.2425 Explore P: 0.1180\n",
      "Episode: 433 Total reward: 199.0 Training loss: 0.2318 Explore P: 0.1159\n",
      "Episode: 434 Total reward: 199.0 Training loss: 0.9312 Explore P: 0.1138\n",
      "Episode: 435 Total reward: 199.0 Training loss: 0.3175 Explore P: 0.1118\n",
      "Episode: 436 Total reward: 199.0 Training loss: 0.1731 Explore P: 0.1098\n",
      "Episode: 437 Total reward: 199.0 Training loss: 0.2117 Explore P: 0.1078\n",
      "Episode: 438 Total reward: 199.0 Training loss: 0.2600 Explore P: 0.1059\n",
      "Episode: 439 Total reward: 199.0 Training loss: 32.6381 Explore P: 0.1040\n",
      "Episode: 440 Total reward: 199.0 Training loss: 0.3713 Explore P: 0.1021\n",
      "Episode: 441 Total reward: 199.0 Training loss: 0.1710 Explore P: 0.1003\n",
      "Episode: 442 Total reward: 199.0 Training loss: 324.6094 Explore P: 0.0985\n",
      "Episode: 443 Total reward: 199.0 Training loss: 0.4218 Explore P: 0.0968\n",
      "Episode: 444 Total reward: 199.0 Training loss: 0.4085 Explore P: 0.0951\n",
      "Episode: 445 Total reward: 199.0 Training loss: 0.4542 Explore P: 0.0934\n",
      "Episode: 446 Total reward: 199.0 Training loss: 0.2896 Explore P: 0.0918\n",
      "Episode: 447 Total reward: 199.0 Training loss: 0.1602 Explore P: 0.0902\n",
      "Episode: 448 Total reward: 199.0 Training loss: 0.2052 Explore P: 0.0886\n",
      "Episode: 449 Total reward: 199.0 Training loss: 353.0534 Explore P: 0.0870\n",
      "Episode: 450 Total reward: 199.0 Training loss: 0.1692 Explore P: 0.0855\n",
      "Episode: 451 Total reward: 199.0 Training loss: 0.3250 Explore P: 0.0840\n",
      "Episode: 452 Total reward: 199.0 Training loss: 0.2329 Explore P: 0.0826\n",
      "Episode: 453 Total reward: 199.0 Training loss: 0.2505 Explore P: 0.0811\n",
      "Episode: 454 Total reward: 199.0 Training loss: 0.3985 Explore P: 0.0797\n",
      "Episode: 455 Total reward: 199.0 Training loss: 0.2700 Explore P: 0.0784\n",
      "Episode: 456 Total reward: 199.0 Training loss: 0.4076 Explore P: 0.0770\n",
      "Episode: 457 Total reward: 199.0 Training loss: 0.2375 Explore P: 0.0757\n",
      "Episode: 458 Total reward: 199.0 Training loss: 0.1582 Explore P: 0.0744\n",
      "Episode: 459 Total reward: 199.0 Training loss: 0.2196 Explore P: 0.0731\n",
      "Episode: 460 Total reward: 199.0 Training loss: 0.1402 Explore P: 0.0719\n",
      "Episode: 461 Total reward: 199.0 Training loss: 0.3282 Explore P: 0.0707\n",
      "Episode: 462 Total reward: 199.0 Training loss: 0.2690 Explore P: 0.0695\n",
      "Episode: 463 Total reward: 199.0 Training loss: 0.2229 Explore P: 0.0683\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 464 Total reward: 199.0 Training loss: 0.3676 Explore P: 0.0671\n",
      "Episode: 465 Total reward: 199.0 Training loss: 0.3111 Explore P: 0.0660\n",
      "Episode: 466 Total reward: 199.0 Training loss: 315.3241 Explore P: 0.0649\n",
      "Episode: 467 Total reward: 199.0 Training loss: 0.5417 Explore P: 0.0638\n",
      "Episode: 468 Total reward: 199.0 Training loss: 0.2517 Explore P: 0.0628\n",
      "Episode: 469 Total reward: 199.0 Training loss: 0.4428 Explore P: 0.0617\n",
      "Episode: 470 Total reward: 199.0 Training loss: 0.6560 Explore P: 0.0607\n",
      "Episode: 471 Total reward: 199.0 Training loss: 0.1914 Explore P: 0.0597\n",
      "Episode: 472 Total reward: 199.0 Training loss: 266.7272 Explore P: 0.0587\n",
      "Episode: 473 Total reward: 199.0 Training loss: 0.2310 Explore P: 0.0578\n",
      "Episode: 474 Total reward: 199.0 Training loss: 229.7998 Explore P: 0.0568\n",
      "Episode: 475 Total reward: 199.0 Training loss: 285.0441 Explore P: 0.0559\n",
      "Episode: 476 Total reward: 199.0 Training loss: 0.2414 Explore P: 0.0550\n",
      "Episode: 477 Total reward: 199.0 Training loss: 258.3245 Explore P: 0.0541\n",
      "Episode: 478 Total reward: 199.0 Training loss: 0.3251 Explore P: 0.0533\n",
      "Episode: 479 Total reward: 199.0 Training loss: 0.2240 Explore P: 0.0524\n",
      "Episode: 480 Total reward: 199.0 Training loss: 0.2032 Explore P: 0.0516\n",
      "Episode: 481 Total reward: 199.0 Training loss: 0.1418 Explore P: 0.0507\n",
      "Episode: 482 Total reward: 199.0 Training loss: 0.2596 Explore P: 0.0499\n",
      "Episode: 483 Total reward: 199.0 Training loss: 0.2563 Explore P: 0.0492\n",
      "Episode: 484 Total reward: 199.0 Training loss: 0.2949 Explore P: 0.0484\n",
      "Episode: 485 Total reward: 199.0 Training loss: 0.3744 Explore P: 0.0476\n",
      "Episode: 486 Total reward: 199.0 Training loss: 0.1250 Explore P: 0.0469\n",
      "Episode: 487 Total reward: 199.0 Training loss: 0.5174 Explore P: 0.0462\n",
      "Episode: 488 Total reward: 199.0 Training loss: 0.1291 Explore P: 0.0454\n",
      "Episode: 489 Total reward: 199.0 Training loss: 0.1595 Explore P: 0.0447\n",
      "Episode: 490 Total reward: 199.0 Training loss: 0.3262 Explore P: 0.0441\n",
      "Episode: 491 Total reward: 199.0 Training loss: 0.4580 Explore P: 0.0434\n",
      "Episode: 492 Total reward: 199.0 Training loss: 0.1858 Explore P: 0.0427\n",
      "Episode: 493 Total reward: 199.0 Training loss: 0.2579 Explore P: 0.0421\n",
      "Episode: 494 Total reward: 199.0 Training loss: 0.1718 Explore P: 0.0415\n",
      "Episode: 495 Total reward: 199.0 Training loss: 0.2422 Explore P: 0.0408\n",
      "Episode: 496 Total reward: 199.0 Training loss: 0.2393 Explore P: 0.0402\n",
      "Episode: 497 Total reward: 199.0 Training loss: 0.2923 Explore P: 0.0396\n",
      "Episode: 498 Total reward: 199.0 Training loss: 0.1560 Explore P: 0.0390\n",
      "Episode: 499 Total reward: 199.0 Training loss: 0.2836 Explore P: 0.0385\n",
      "Episode: 500 Total reward: 199.0 Training loss: 0.2856 Explore P: 0.0379\n",
      "Episode: 501 Total reward: 199.0 Training loss: 0.1609 Explore P: 0.0374\n",
      "Episode: 502 Total reward: 199.0 Training loss: 0.1673 Explore P: 0.0368\n",
      "Episode: 503 Total reward: 199.0 Training loss: 0.2997 Explore P: 0.0363\n",
      "Episode: 504 Total reward: 199.0 Training loss: 138.8440 Explore P: 0.0358\n",
      "Episode: 505 Total reward: 199.0 Training loss: 0.3624 Explore P: 0.0353\n",
      "Episode: 506 Total reward: 199.0 Training loss: 0.1804 Explore P: 0.0348\n",
      "Episode: 507 Total reward: 199.0 Training loss: 0.2234 Explore P: 0.0343\n",
      "Episode: 508 Total reward: 199.0 Training loss: 0.1588 Explore P: 0.0338\n",
      "Episode: 509 Total reward: 199.0 Training loss: 0.4527 Explore P: 0.0333\n",
      "Episode: 510 Total reward: 199.0 Training loss: 216.9014 Explore P: 0.0329\n",
      "Episode: 511 Total reward: 199.0 Training loss: 0.3706 Explore P: 0.0324\n",
      "Episode: 512 Total reward: 199.0 Training loss: 0.4239 Explore P: 0.0320\n",
      "Episode: 513 Total reward: 199.0 Training loss: 0.1884 Explore P: 0.0316\n",
      "Episode: 514 Total reward: 199.0 Training loss: 0.2130 Explore P: 0.0311\n",
      "Episode: 515 Total reward: 199.0 Training loss: 126.4870 Explore P: 0.0307\n",
      "Episode: 516 Total reward: 199.0 Training loss: 0.1493 Explore P: 0.0303\n",
      "Episode: 517 Total reward: 199.0 Training loss: 0.2435 Explore P: 0.0299\n",
      "Episode: 518 Total reward: 199.0 Training loss: 0.1257 Explore P: 0.0295\n",
      "Episode: 519 Total reward: 199.0 Training loss: 0.1419 Explore P: 0.0291\n",
      "Episode: 520 Total reward: 199.0 Training loss: 0.1724 Explore P: 0.0288\n",
      "Episode: 521 Total reward: 199.0 Training loss: 0.2624 Explore P: 0.0284\n",
      "Episode: 522 Total reward: 199.0 Training loss: 0.3318 Explore P: 0.0280\n",
      "Episode: 523 Total reward: 199.0 Training loss: 0.1169 Explore P: 0.0277\n",
      "Episode: 524 Total reward: 199.0 Training loss: 0.2295 Explore P: 0.0273\n",
      "Episode: 525 Total reward: 199.0 Training loss: 0.2648 Explore P: 0.0270\n",
      "Episode: 526 Total reward: 199.0 Training loss: 0.1279 Explore P: 0.0266\n",
      "Episode: 527 Total reward: 199.0 Training loss: 110.3417 Explore P: 0.0263\n",
      "Episode: 528 Total reward: 199.0 Training loss: 0.2806 Explore P: 0.0260\n",
      "Episode: 529 Total reward: 199.0 Training loss: 0.1521 Explore P: 0.0257\n",
      "Episode: 530 Total reward: 199.0 Training loss: 0.1319 Explore P: 0.0254\n",
      "Episode: 531 Total reward: 199.0 Training loss: 0.3769 Explore P: 0.0251\n",
      "Episode: 532 Total reward: 199.0 Training loss: 0.1795 Explore P: 0.0248\n",
      "Episode: 533 Total reward: 103.0 Training loss: 0.1401 Explore P: 0.0246\n",
      "Episode: 534 Total reward: 199.0 Training loss: 0.2761 Explore P: 0.0243\n",
      "Episode: 535 Total reward: 119.0 Training loss: 0.1163 Explore P: 0.0242\n",
      "Episode: 536 Total reward: 199.0 Training loss: 0.1288 Explore P: 0.0239\n",
      "Episode: 537 Total reward: 166.0 Training loss: 0.3153 Explore P: 0.0237\n",
      "Episode: 538 Total reward: 199.0 Training loss: 0.7775 Explore P: 0.0234\n",
      "Episode: 539 Total reward: 166.0 Training loss: 0.1311 Explore P: 0.0232\n",
      "Episode: 540 Total reward: 199.0 Training loss: 0.3983 Explore P: 0.0229\n",
      "Episode: 541 Total reward: 199.0 Training loss: 0.3361 Explore P: 0.0226\n",
      "Episode: 542 Total reward: 199.0 Training loss: 0.2217 Explore P: 0.0224\n",
      "Episode: 543 Total reward: 123.0 Training loss: 0.1656 Explore P: 0.0222\n",
      "Episode: 544 Total reward: 149.0 Training loss: 0.2754 Explore P: 0.0221\n",
      "Episode: 545 Total reward: 174.0 Training loss: 0.3914 Explore P: 0.0219\n",
      "Episode: 546 Total reward: 153.0 Training loss: 0.2412 Explore P: 0.0217\n",
      "Episode: 547 Total reward: 199.0 Training loss: 0.3066 Explore P: 0.0214\n",
      "Episode: 548 Total reward: 199.0 Training loss: 0.2616 Explore P: 0.0212\n",
      "Episode: 549 Total reward: 100.0 Training loss: 0.1317 Explore P: 0.0211\n",
      "Episode: 550 Total reward: 165.0 Training loss: 0.2129 Explore P: 0.0209\n",
      "Episode: 551 Total reward: 199.0 Training loss: 0.1069 Explore P: 0.0207\n",
      "Episode: 552 Total reward: 199.0 Training loss: 0.2774 Explore P: 0.0205\n",
      "Episode: 553 Total reward: 78.0 Training loss: 0.2059 Explore P: 0.0204\n",
      "Episode: 554 Total reward: 199.0 Training loss: 0.1320 Explore P: 0.0202\n",
      "Episode: 555 Total reward: 87.0 Training loss: 0.3499 Explore P: 0.0201\n",
      "Episode: 556 Total reward: 199.0 Training loss: 0.2720 Explore P: 0.0199\n",
      "Episode: 557 Total reward: 199.0 Training loss: 0.0952 Explore P: 0.0197\n",
      "Episode: 558 Total reward: 144.0 Training loss: 0.2044 Explore P: 0.0196\n",
      "Episode: 559 Total reward: 171.0 Training loss: 0.1669 Explore P: 0.0194\n",
      "Episode: 560 Total reward: 199.0 Training loss: 0.1670 Explore P: 0.0192\n",
      "Episode: 561 Total reward: 199.0 Training loss: 0.3387 Explore P: 0.0191\n",
      "Episode: 562 Total reward: 199.0 Training loss: 0.2244 Explore P: 0.0189\n",
      "Episode: 563 Total reward: 199.0 Training loss: 0.5043 Explore P: 0.0187\n",
      "Episode: 564 Total reward: 199.0 Training loss: 0.0803 Explore P: 0.0185\n",
      "Episode: 565 Total reward: 132.0 Training loss: 0.2384 Explore P: 0.0184\n",
      "Episode: 566 Total reward: 199.0 Training loss: 0.6607 Explore P: 0.0183\n",
      "Episode: 567 Total reward: 122.0 Training loss: 0.2858 Explore P: 0.0182\n",
      "Episode: 568 Total reward: 128.0 Training loss: 0.3465 Explore P: 0.0181\n",
      "Episode: 569 Total reward: 137.0 Training loss: 0.1632 Explore P: 0.0179\n",
      "Episode: 570 Total reward: 73.0 Training loss: 0.2270 Explore P: 0.0179\n",
      "Episode: 571 Total reward: 103.0 Training loss: 0.0889 Explore P: 0.0178\n",
      "Episode: 572 Total reward: 70.0 Training loss: 0.1899 Explore P: 0.0178\n",
      "Episode: 573 Total reward: 153.0 Training loss: 0.1248 Explore P: 0.0176\n",
      "Episode: 574 Total reward: 156.0 Training loss: 0.3964 Explore P: 0.0175\n",
      "Episode: 575 Total reward: 109.0 Training loss: 0.2420 Explore P: 0.0174\n",
      "Episode: 576 Total reward: 142.0 Training loss: 0.2142 Explore P: 0.0173\n",
      "Episode: 577 Total reward: 121.0 Training loss: 0.2657 Explore P: 0.0172\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 578 Total reward: 72.0 Training loss: 0.6169 Explore P: 0.0172\n",
      "Episode: 579 Total reward: 199.0 Training loss: 0.3492 Explore P: 0.0170\n",
      "Episode: 580 Total reward: 84.0 Training loss: 0.3777 Explore P: 0.0170\n",
      "Episode: 581 Total reward: 76.0 Training loss: 84.2530 Explore P: 0.0169\n",
      "Episode: 582 Total reward: 95.0 Training loss: 0.5778 Explore P: 0.0169\n",
      "Episode: 583 Total reward: 199.0 Training loss: 0.2978 Explore P: 0.0167\n",
      "Episode: 584 Total reward: 63.0 Training loss: 0.1074 Explore P: 0.0167\n",
      "Episode: 585 Total reward: 88.0 Training loss: 0.3942 Explore P: 0.0166\n",
      "Episode: 586 Total reward: 199.0 Training loss: 0.2634 Explore P: 0.0165\n",
      "Episode: 587 Total reward: 125.0 Training loss: 0.2356 Explore P: 0.0164\n",
      "Episode: 588 Total reward: 199.0 Training loss: 0.1060 Explore P: 0.0163\n",
      "Episode: 589 Total reward: 54.0 Training loss: 0.2123 Explore P: 0.0163\n",
      "Episode: 590 Total reward: 61.0 Training loss: 0.1699 Explore P: 0.0162\n",
      "Episode: 591 Total reward: 199.0 Training loss: 0.4668 Explore P: 0.0161\n",
      "Episode: 592 Total reward: 142.0 Training loss: 0.2722 Explore P: 0.0160\n",
      "Episode: 593 Total reward: 199.0 Training loss: 0.2001 Explore P: 0.0159\n",
      "Episode: 594 Total reward: 199.0 Training loss: 0.1574 Explore P: 0.0158\n",
      "Episode: 595 Total reward: 199.0 Training loss: 0.4673 Explore P: 0.0157\n",
      "Episode: 596 Total reward: 100.0 Training loss: 0.2242 Explore P: 0.0156\n",
      "Episode: 597 Total reward: 199.0 Training loss: 0.3222 Explore P: 0.0155\n",
      "Episode: 598 Total reward: 199.0 Training loss: 4.0033 Explore P: 0.0154\n",
      "Episode: 599 Total reward: 199.0 Training loss: 0.2613 Explore P: 0.0153\n",
      "Episode: 600 Total reward: 199.0 Training loss: 0.1150 Explore P: 0.0152\n",
      "Episode: 601 Total reward: 199.0 Training loss: 0.1762 Explore P: 0.0151\n",
      "Episode: 602 Total reward: 115.0 Training loss: 0.1108 Explore P: 0.0150\n",
      "Episode: 603 Total reward: 89.0 Training loss: 0.3132 Explore P: 0.0150\n",
      "Episode: 604 Total reward: 199.0 Training loss: 0.2228 Explore P: 0.0149\n",
      "Episode: 605 Total reward: 199.0 Training loss: 0.3473 Explore P: 0.0148\n",
      "Episode: 606 Total reward: 199.0 Training loss: 250.8660 Explore P: 0.0147\n",
      "Episode: 607 Total reward: 148.0 Training loss: 0.2939 Explore P: 0.0146\n",
      "Episode: 608 Total reward: 139.0 Training loss: 0.4761 Explore P: 0.0146\n",
      "Episode: 609 Total reward: 199.0 Training loss: 0.3007 Explore P: 0.0145\n",
      "Episode: 610 Total reward: 199.0 Training loss: 0.1928 Explore P: 0.0144\n",
      "Episode: 611 Total reward: 199.0 Training loss: 0.1506 Explore P: 0.0143\n",
      "Episode: 612 Total reward: 199.0 Training loss: 0.1687 Explore P: 0.0142\n",
      "Episode: 613 Total reward: 199.0 Training loss: 0.2781 Explore P: 0.0141\n",
      "Episode: 614 Total reward: 199.0 Training loss: 0.2610 Explore P: 0.0140\n",
      "Episode: 615 Total reward: 199.0 Training loss: 0.2177 Explore P: 0.0140\n",
      "Episode: 616 Total reward: 118.0 Training loss: 0.2153 Explore P: 0.0139\n",
      "Episode: 617 Total reward: 131.0 Training loss: 0.2916 Explore P: 0.0139\n",
      "Episode: 618 Total reward: 199.0 Training loss: 0.5013 Explore P: 0.0138\n",
      "Episode: 619 Total reward: 199.0 Training loss: 159.8071 Explore P: 0.0137\n",
      "Episode: 620 Total reward: 199.0 Training loss: 2.8122 Explore P: 0.0136\n",
      "Episode: 621 Total reward: 199.0 Training loss: 0.8162 Explore P: 0.0136\n",
      "Episode: 622 Total reward: 199.0 Training loss: 0.3258 Explore P: 0.0135\n",
      "Episode: 623 Total reward: 199.0 Training loss: 0.1336 Explore P: 0.0134\n",
      "Episode: 624 Total reward: 199.0 Training loss: 0.1858 Explore P: 0.0134\n",
      "Episode: 625 Total reward: 199.0 Training loss: 0.3802 Explore P: 0.0133\n",
      "Episode: 626 Total reward: 199.0 Training loss: 0.2334 Explore P: 0.0132\n",
      "Episode: 627 Total reward: 199.0 Training loss: 0.3573 Explore P: 0.0132\n",
      "Episode: 628 Total reward: 199.0 Training loss: 0.1295 Explore P: 0.0131\n",
      "Episode: 629 Total reward: 199.0 Training loss: 0.3515 Explore P: 0.0130\n",
      "Episode: 630 Total reward: 199.0 Training loss: 0.2244 Explore P: 0.0130\n",
      "Episode: 631 Total reward: 199.0 Training loss: 0.2216 Explore P: 0.0129\n",
      "Episode: 632 Total reward: 199.0 Training loss: 0.2608 Explore P: 0.0129\n",
      "Episode: 633 Total reward: 199.0 Training loss: 0.1095 Explore P: 0.0128\n",
      "Episode: 634 Total reward: 199.0 Training loss: 0.2628 Explore P: 0.0128\n",
      "Episode: 635 Total reward: 199.0 Training loss: 0.5734 Explore P: 0.0127\n",
      "Episode: 636 Total reward: 199.0 Training loss: 0.1622 Explore P: 0.0126\n",
      "Episode: 637 Total reward: 199.0 Training loss: 0.2036 Explore P: 0.0126\n",
      "Episode: 638 Total reward: 199.0 Training loss: 0.4365 Explore P: 0.0125\n",
      "Episode: 639 Total reward: 199.0 Training loss: 0.2580 Explore P: 0.0125\n",
      "Episode: 640 Total reward: 199.0 Training loss: 0.3614 Explore P: 0.0124\n",
      "Episode: 641 Total reward: 199.0 Training loss: 0.1117 Explore P: 0.0124\n",
      "Episode: 642 Total reward: 199.0 Training loss: 0.2955 Explore P: 0.0124\n",
      "Episode: 643 Total reward: 199.0 Training loss: 0.3942 Explore P: 0.0123\n",
      "Episode: 644 Total reward: 199.0 Training loss: 0.2117 Explore P: 0.0123\n",
      "Episode: 645 Total reward: 199.0 Training loss: 271.9186 Explore P: 0.0122\n",
      "Episode: 646 Total reward: 199.0 Training loss: 0.3243 Explore P: 0.0122\n",
      "Episode: 647 Total reward: 199.0 Training loss: 3.7811 Explore P: 0.0121\n",
      "Episode: 648 Total reward: 199.0 Training loss: 0.2246 Explore P: 0.0121\n",
      "Episode: 649 Total reward: 199.0 Training loss: 0.2025 Explore P: 0.0120\n",
      "Episode: 650 Total reward: 199.0 Training loss: 0.3298 Explore P: 0.0120\n",
      "Episode: 651 Total reward: 199.0 Training loss: 0.3632 Explore P: 0.0120\n",
      "Episode: 652 Total reward: 199.0 Training loss: 0.2831 Explore P: 0.0119\n",
      "Episode: 653 Total reward: 199.0 Training loss: 0.6059 Explore P: 0.0119\n",
      "Episode: 654 Total reward: 199.0 Training loss: 0.3132 Explore P: 0.0119\n",
      "Episode: 655 Total reward: 199.0 Training loss: 0.2726 Explore P: 0.0118\n",
      "Episode: 656 Total reward: 199.0 Training loss: 0.2659 Explore P: 0.0118\n",
      "Episode: 657 Total reward: 199.0 Training loss: 13.8581 Explore P: 0.0117\n",
      "Episode: 658 Total reward: 199.0 Training loss: 0.2583 Explore P: 0.0117\n",
      "Episode: 659 Total reward: 199.0 Training loss: 0.2022 Explore P: 0.0117\n",
      "Episode: 660 Total reward: 199.0 Training loss: 0.3043 Explore P: 0.0116\n",
      "Episode: 661 Total reward: 199.0 Training loss: 0.4852 Explore P: 0.0116\n",
      "Episode: 662 Total reward: 199.0 Training loss: 0.4260 Explore P: 0.0116\n",
      "Episode: 663 Total reward: 199.0 Training loss: 0.2368 Explore P: 0.0115\n",
      "Episode: 664 Total reward: 199.0 Training loss: 0.1965 Explore P: 0.0115\n",
      "Episode: 665 Total reward: 199.0 Training loss: 0.3622 Explore P: 0.0115\n",
      "Episode: 666 Total reward: 199.0 Training loss: 0.2421 Explore P: 0.0115\n",
      "Episode: 667 Total reward: 199.0 Training loss: 0.1476 Explore P: 0.0114\n",
      "Episode: 668 Total reward: 199.0 Training loss: 0.2042 Explore P: 0.0114\n",
      "Episode: 669 Total reward: 199.0 Training loss: 0.3781 Explore P: 0.0114\n",
      "Episode: 670 Total reward: 199.0 Training loss: 0.2926 Explore P: 0.0113\n",
      "Episode: 671 Total reward: 199.0 Training loss: 232.6172 Explore P: 0.0113\n",
      "Episode: 672 Total reward: 199.0 Training loss: 0.2247 Explore P: 0.0113\n",
      "Episode: 673 Total reward: 199.0 Training loss: 0.3271 Explore P: 0.0113\n",
      "Episode: 674 Total reward: 199.0 Training loss: 0.2424 Explore P: 0.0112\n",
      "Episode: 675 Total reward: 199.0 Training loss: 0.2772 Explore P: 0.0112\n",
      "Episode: 676 Total reward: 199.0 Training loss: 212.1298 Explore P: 0.0112\n",
      "Episode: 677 Total reward: 199.0 Training loss: 0.3303 Explore P: 0.0112\n",
      "Episode: 678 Total reward: 199.0 Training loss: 0.4014 Explore P: 0.0111\n",
      "Episode: 679 Total reward: 199.0 Training loss: 0.2041 Explore P: 0.0111\n",
      "Episode: 680 Total reward: 199.0 Training loss: 0.2996 Explore P: 0.0111\n",
      "Episode: 681 Total reward: 199.0 Training loss: 0.0574 Explore P: 0.0111\n",
      "Episode: 682 Total reward: 199.0 Training loss: 251.5338 Explore P: 0.0111\n",
      "Episode: 683 Total reward: 199.0 Training loss: 0.2080 Explore P: 0.0110\n",
      "Episode: 684 Total reward: 199.0 Training loss: 0.2399 Explore P: 0.0110\n",
      "Episode: 685 Total reward: 199.0 Training loss: 0.1960 Explore P: 0.0110\n",
      "Episode: 686 Total reward: 199.0 Training loss: 0.1967 Explore P: 0.0110\n",
      "Episode: 687 Total reward: 199.0 Training loss: 0.1693 Explore P: 0.0110\n",
      "Episode: 688 Total reward: 199.0 Training loss: 0.2816 Explore P: 0.0109\n",
      "Episode: 689 Total reward: 199.0 Training loss: 0.6358 Explore P: 0.0109\n",
      "Episode: 690 Total reward: 199.0 Training loss: 0.3084 Explore P: 0.0109\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 691 Total reward: 199.0 Training loss: 0.2153 Explore P: 0.0109\n",
      "Episode: 692 Total reward: 199.0 Training loss: 0.5129 Explore P: 0.0109\n",
      "Episode: 693 Total reward: 199.0 Training loss: 0.2925 Explore P: 0.0109\n",
      "Episode: 694 Total reward: 199.0 Training loss: 0.4260 Explore P: 0.0108\n",
      "Episode: 695 Total reward: 199.0 Training loss: 0.3335 Explore P: 0.0108\n",
      "Episode: 696 Total reward: 199.0 Training loss: 0.4102 Explore P: 0.0108\n",
      "Episode: 697 Total reward: 199.0 Training loss: 0.2342 Explore P: 0.0108\n",
      "Episode: 698 Total reward: 199.0 Training loss: 0.2981 Explore P: 0.0108\n",
      "Episode: 699 Total reward: 199.0 Training loss: 0.3775 Explore P: 0.0108\n",
      "Episode: 700 Total reward: 199.0 Training loss: 0.1577 Explore P: 0.0107\n",
      "Episode: 701 Total reward: 199.0 Training loss: 0.4164 Explore P: 0.0107\n",
      "Episode: 702 Total reward: 199.0 Training loss: 0.1895 Explore P: 0.0107\n",
      "Episode: 703 Total reward: 199.0 Training loss: 0.3284 Explore P: 0.0107\n",
      "Episode: 704 Total reward: 199.0 Training loss: 0.1521 Explore P: 0.0107\n",
      "Episode: 705 Total reward: 199.0 Training loss: 0.5423 Explore P: 0.0107\n",
      "Episode: 706 Total reward: 199.0 Training loss: 0.1817 Explore P: 0.0107\n",
      "Episode: 707 Total reward: 199.0 Training loss: 0.4591 Explore P: 0.0106\n",
      "Episode: 708 Total reward: 199.0 Training loss: 0.6386 Explore P: 0.0106\n",
      "Episode: 709 Total reward: 199.0 Training loss: 0.2571 Explore P: 0.0106\n",
      "Episode: 710 Total reward: 199.0 Training loss: 0.1635 Explore P: 0.0106\n",
      "Episode: 711 Total reward: 199.0 Training loss: 0.1379 Explore P: 0.0106\n",
      "Episode: 712 Total reward: 199.0 Training loss: 0.5101 Explore P: 0.0106\n",
      "Episode: 713 Total reward: 199.0 Training loss: 0.3035 Explore P: 0.0106\n",
      "Episode: 714 Total reward: 199.0 Training loss: 0.1619 Explore P: 0.0106\n",
      "Episode: 715 Total reward: 199.0 Training loss: 0.4874 Explore P: 0.0105\n",
      "Episode: 716 Total reward: 199.0 Training loss: 134.6547 Explore P: 0.0105\n",
      "Episode: 717 Total reward: 199.0 Training loss: 0.3355 Explore P: 0.0105\n",
      "Episode: 718 Total reward: 199.0 Training loss: 0.3444 Explore P: 0.0105\n",
      "Episode: 719 Total reward: 199.0 Training loss: 0.2052 Explore P: 0.0105\n",
      "Episode: 720 Total reward: 199.0 Training loss: 0.2845 Explore P: 0.0105\n",
      "Episode: 721 Total reward: 199.0 Training loss: 0.4237 Explore P: 0.0105\n",
      "Episode: 722 Total reward: 199.0 Training loss: 0.9381 Explore P: 0.0105\n",
      "Episode: 723 Total reward: 199.0 Training loss: 0.8202 Explore P: 0.0105\n",
      "Episode: 724 Total reward: 199.0 Training loss: 0.8193 Explore P: 0.0105\n",
      "Episode: 725 Total reward: 199.0 Training loss: 0.4997 Explore P: 0.0105\n",
      "Episode: 726 Total reward: 199.0 Training loss: 0.5080 Explore P: 0.0104\n",
      "Episode: 727 Total reward: 199.0 Training loss: 0.8178 Explore P: 0.0104\n",
      "Episode: 728 Total reward: 199.0 Training loss: 0.5532 Explore P: 0.0104\n",
      "Episode: 729 Total reward: 199.0 Training loss: 577.7544 Explore P: 0.0104\n",
      "Episode: 730 Total reward: 199.0 Training loss: 0.8602 Explore P: 0.0104\n",
      "Episode: 731 Total reward: 19.0 Training loss: 0.3950 Explore P: 0.0104\n",
      "Episode: 732 Total reward: 11.0 Training loss: 0.9893 Explore P: 0.0104\n",
      "Episode: 733 Total reward: 199.0 Training loss: 0.4568 Explore P: 0.0104\n",
      "Episode: 734 Total reward: 199.0 Training loss: 0.6809 Explore P: 0.0104\n",
      "Episode: 735 Total reward: 199.0 Training loss: 1.5043 Explore P: 0.0104\n",
      "Episode: 736 Total reward: 199.0 Training loss: 0.5399 Explore P: 0.0104\n",
      "Episode: 737 Total reward: 199.0 Training loss: 0.4928 Explore P: 0.0104\n",
      "Episode: 738 Total reward: 199.0 Training loss: 0.3551 Explore P: 0.0104\n",
      "Episode: 739 Total reward: 199.0 Training loss: 0.1797 Explore P: 0.0104\n",
      "Episode: 740 Total reward: 199.0 Training loss: 0.5167 Explore P: 0.0103\n",
      "Episode: 741 Total reward: 199.0 Training loss: 0.5068 Explore P: 0.0103\n",
      "Episode: 742 Total reward: 199.0 Training loss: 0.3953 Explore P: 0.0103\n",
      "Episode: 743 Total reward: 199.0 Training loss: 0.3673 Explore P: 0.0103\n",
      "Episode: 744 Total reward: 199.0 Training loss: 0.3430 Explore P: 0.0103\n",
      "Episode: 745 Total reward: 199.0 Training loss: 0.6551 Explore P: 0.0103\n",
      "Episode: 746 Total reward: 199.0 Training loss: 0.2240 Explore P: 0.0103\n",
      "Episode: 747 Total reward: 199.0 Training loss: 0.2918 Explore P: 0.0103\n",
      "Episode: 748 Total reward: 199.0 Training loss: 0.3925 Explore P: 0.0103\n",
      "Episode: 749 Total reward: 199.0 Training loss: 0.4508 Explore P: 0.0103\n",
      "Episode: 750 Total reward: 199.0 Training loss: 0.4926 Explore P: 0.0103\n",
      "Episode: 751 Total reward: 199.0 Training loss: 0.5516 Explore P: 0.0103\n",
      "Episode: 752 Total reward: 199.0 Training loss: 0.4907 Explore P: 0.0103\n",
      "Episode: 753 Total reward: 199.0 Training loss: 0.2899 Explore P: 0.0103\n",
      "Episode: 754 Total reward: 199.0 Training loss: 0.3123 Explore P: 0.0103\n",
      "Episode: 755 Total reward: 199.0 Training loss: 0.6357 Explore P: 0.0103\n",
      "Episode: 756 Total reward: 199.0 Training loss: 8.9829 Explore P: 0.0103\n",
      "Episode: 757 Total reward: 199.0 Training loss: 0.6363 Explore P: 0.0102\n",
      "Episode: 758 Total reward: 199.0 Training loss: 0.2894 Explore P: 0.0102\n",
      "Episode: 759 Total reward: 199.0 Training loss: 0.4332 Explore P: 0.0102\n",
      "Episode: 760 Total reward: 199.0 Training loss: 0.5926 Explore P: 0.0102\n",
      "Episode: 761 Total reward: 199.0 Training loss: 506.5080 Explore P: 0.0102\n",
      "Episode: 762 Total reward: 199.0 Training loss: 0.5457 Explore P: 0.0102\n",
      "Episode: 763 Total reward: 199.0 Training loss: 0.3512 Explore P: 0.0102\n",
      "Episode: 764 Total reward: 199.0 Training loss: 0.6485 Explore P: 0.0102\n",
      "Episode: 765 Total reward: 199.0 Training loss: 0.4183 Explore P: 0.0102\n",
      "Episode: 766 Total reward: 199.0 Training loss: 1.0808 Explore P: 0.0102\n",
      "Episode: 767 Total reward: 199.0 Training loss: 0.8708 Explore P: 0.0102\n",
      "Episode: 768 Total reward: 199.0 Training loss: 0.6690 Explore P: 0.0102\n",
      "Episode: 769 Total reward: 199.0 Training loss: 0.5592 Explore P: 0.0102\n",
      "Episode: 770 Total reward: 199.0 Training loss: 0.6966 Explore P: 0.0102\n",
      "Episode: 771 Total reward: 199.0 Training loss: 0.4797 Explore P: 0.0102\n",
      "Episode: 772 Total reward: 199.0 Training loss: 0.4591 Explore P: 0.0102\n",
      "Episode: 773 Total reward: 199.0 Training loss: 1048.5974 Explore P: 0.0102\n",
      "Episode: 774 Total reward: 199.0 Training loss: 0.5406 Explore P: 0.0102\n",
      "Episode: 775 Total reward: 199.0 Training loss: 0.6683 Explore P: 0.0102\n",
      "Episode: 776 Total reward: 199.0 Training loss: 0.5354 Explore P: 0.0102\n",
      "Episode: 777 Total reward: 199.0 Training loss: 0.7132 Explore P: 0.0102\n",
      "Episode: 778 Total reward: 199.0 Training loss: 522.4263 Explore P: 0.0102\n",
      "Episode: 779 Total reward: 199.0 Training loss: 0.9972 Explore P: 0.0102\n",
      "Episode: 780 Total reward: 199.0 Training loss: 0.6430 Explore P: 0.0102\n",
      "Episode: 781 Total reward: 199.0 Training loss: 0.4306 Explore P: 0.0102\n",
      "Episode: 782 Total reward: 199.0 Training loss: 0.2736 Explore P: 0.0102\n",
      "Episode: 783 Total reward: 199.0 Training loss: 0.6797 Explore P: 0.0101\n",
      "Episode: 784 Total reward: 199.0 Training loss: 0.6017 Explore P: 0.0101\n",
      "Episode: 785 Total reward: 199.0 Training loss: 0.2942 Explore P: 0.0101\n",
      "Episode: 786 Total reward: 199.0 Training loss: 0.3176 Explore P: 0.0101\n",
      "Episode: 787 Total reward: 199.0 Training loss: 0.2173 Explore P: 0.0101\n",
      "Episode: 788 Total reward: 199.0 Training loss: 0.3894 Explore P: 0.0101\n",
      "Episode: 789 Total reward: 199.0 Training loss: 0.2316 Explore P: 0.0101\n",
      "Episode: 790 Total reward: 199.0 Training loss: 0.3222 Explore P: 0.0101\n",
      "Episode: 791 Total reward: 199.0 Training loss: 0.1653 Explore P: 0.0101\n",
      "Episode: 792 Total reward: 199.0 Training loss: 0.4595 Explore P: 0.0101\n",
      "Episode: 793 Total reward: 199.0 Training loss: 0.1815 Explore P: 0.0101\n",
      "Episode: 794 Total reward: 199.0 Training loss: 0.2181 Explore P: 0.0101\n",
      "Episode: 795 Total reward: 199.0 Training loss: 0.0952 Explore P: 0.0101\n",
      "Episode: 796 Total reward: 199.0 Training loss: 0.6166 Explore P: 0.0101\n",
      "Episode: 797 Total reward: 199.0 Training loss: 0.1695 Explore P: 0.0101\n",
      "Episode: 798 Total reward: 199.0 Training loss: 0.4265 Explore P: 0.0101\n",
      "Episode: 799 Total reward: 199.0 Training loss: 0.1859 Explore P: 0.0101\n",
      "Episode: 800 Total reward: 199.0 Training loss: 0.2769 Explore P: 0.0101\n",
      "Episode: 801 Total reward: 199.0 Training loss: 0.1688 Explore P: 0.0101\n",
      "Episode: 802 Total reward: 199.0 Training loss: 0.1748 Explore P: 0.0101\n",
      "Episode: 803 Total reward: 199.0 Training loss: 0.1359 Explore P: 0.0101\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 804 Total reward: 199.0 Training loss: 0.1456 Explore P: 0.0101\n",
      "Episode: 805 Total reward: 199.0 Training loss: 0.1803 Explore P: 0.0101\n",
      "Episode: 806 Total reward: 199.0 Training loss: 200.8581 Explore P: 0.0101\n",
      "Episode: 807 Total reward: 199.0 Training loss: 0.1792 Explore P: 0.0101\n",
      "Episode: 808 Total reward: 199.0 Training loss: 0.3391 Explore P: 0.0101\n",
      "Episode: 809 Total reward: 199.0 Training loss: 0.4329 Explore P: 0.0101\n",
      "Episode: 810 Total reward: 199.0 Training loss: 0.0870 Explore P: 0.0101\n",
      "Episode: 811 Total reward: 199.0 Training loss: 0.1443 Explore P: 0.0101\n",
      "Episode: 812 Total reward: 199.0 Training loss: 0.1685 Explore P: 0.0101\n",
      "Episode: 813 Total reward: 199.0 Training loss: 0.1582 Explore P: 0.0101\n",
      "Episode: 814 Total reward: 199.0 Training loss: 0.2634 Explore P: 0.0101\n",
      "Episode: 815 Total reward: 199.0 Training loss: 0.1814 Explore P: 0.0101\n",
      "Episode: 816 Total reward: 199.0 Training loss: 0.2449 Explore P: 0.0101\n",
      "Episode: 817 Total reward: 199.0 Training loss: 0.2075 Explore P: 0.0101\n",
      "Episode: 818 Total reward: 199.0 Training loss: 117.4379 Explore P: 0.0101\n",
      "Episode: 819 Total reward: 199.0 Training loss: 115.7762 Explore P: 0.0101\n",
      "Episode: 820 Total reward: 199.0 Training loss: 0.1274 Explore P: 0.0101\n",
      "Episode: 821 Total reward: 199.0 Training loss: 0.1734 Explore P: 0.0101\n",
      "Episode: 822 Total reward: 199.0 Training loss: 0.1714 Explore P: 0.0101\n",
      "Episode: 823 Total reward: 199.0 Training loss: 0.1146 Explore P: 0.0101\n",
      "Episode: 824 Total reward: 199.0 Training loss: 0.3100 Explore P: 0.0101\n",
      "Episode: 825 Total reward: 199.0 Training loss: 0.6842 Explore P: 0.0101\n",
      "Episode: 826 Total reward: 199.0 Training loss: 0.1282 Explore P: 0.0101\n",
      "Episode: 827 Total reward: 199.0 Training loss: 0.1660 Explore P: 0.0101\n",
      "Episode: 828 Total reward: 199.0 Training loss: 0.2183 Explore P: 0.0101\n",
      "Episode: 829 Total reward: 199.0 Training loss: 0.3697 Explore P: 0.0101\n",
      "Episode: 830 Total reward: 199.0 Training loss: 0.5128 Explore P: 0.0101\n",
      "Episode: 831 Total reward: 199.0 Training loss: 0.2246 Explore P: 0.0101\n",
      "Episode: 832 Total reward: 199.0 Training loss: 0.2780 Explore P: 0.0101\n",
      "Episode: 833 Total reward: 199.0 Training loss: 0.1626 Explore P: 0.0101\n",
      "Episode: 834 Total reward: 199.0 Training loss: 0.5266 Explore P: 0.0101\n",
      "Episode: 835 Total reward: 199.0 Training loss: 0.6509 Explore P: 0.0101\n",
      "Episode: 836 Total reward: 199.0 Training loss: 0.2582 Explore P: 0.0101\n",
      "Episode: 837 Total reward: 199.0 Training loss: 0.2055 Explore P: 0.0101\n",
      "Episode: 838 Total reward: 199.0 Training loss: 0.2065 Explore P: 0.0100\n",
      "Episode: 839 Total reward: 199.0 Training loss: 0.1970 Explore P: 0.0100\n",
      "Episode: 840 Total reward: 199.0 Training loss: 0.2114 Explore P: 0.0100\n",
      "Episode: 841 Total reward: 199.0 Training loss: 0.3302 Explore P: 0.0100\n",
      "Episode: 842 Total reward: 199.0 Training loss: 0.6025 Explore P: 0.0100\n",
      "Episode: 843 Total reward: 199.0 Training loss: 0.2095 Explore P: 0.0100\n",
      "Episode: 844 Total reward: 199.0 Training loss: 0.4807 Explore P: 0.0100\n",
      "Episode: 845 Total reward: 199.0 Training loss: 0.4625 Explore P: 0.0100\n",
      "Episode: 846 Total reward: 199.0 Training loss: 0.1726 Explore P: 0.0100\n",
      "Episode: 847 Total reward: 199.0 Training loss: 0.4295 Explore P: 0.0100\n",
      "Episode: 848 Total reward: 139.0 Training loss: 0.5734 Explore P: 0.0100\n",
      "Episode: 849 Total reward: 138.0 Training loss: 0.3473 Explore P: 0.0100\n",
      "Episode: 850 Total reward: 88.0 Training loss: 0.4331 Explore P: 0.0100\n",
      "Episode: 851 Total reward: 81.0 Training loss: 0.5970 Explore P: 0.0100\n",
      "Episode: 852 Total reward: 124.0 Training loss: 5.0296 Explore P: 0.0100\n",
      "Episode: 853 Total reward: 54.0 Training loss: 0.8219 Explore P: 0.0100\n",
      "Episode: 854 Total reward: 54.0 Training loss: 1.0331 Explore P: 0.0100\n",
      "Episode: 855 Total reward: 39.0 Training loss: 0.6330 Explore P: 0.0100\n",
      "Episode: 856 Total reward: 64.0 Training loss: 0.5811 Explore P: 0.0100\n",
      "Episode: 857 Total reward: 47.0 Training loss: 0.8336 Explore P: 0.0100\n",
      "Episode: 858 Total reward: 48.0 Training loss: 0.3980 Explore P: 0.0100\n",
      "Episode: 859 Total reward: 100.0 Training loss: 0.6062 Explore P: 0.0100\n",
      "Episode: 860 Total reward: 199.0 Training loss: 0.4624 Explore P: 0.0100\n",
      "Episode: 861 Total reward: 199.0 Training loss: 0.4191 Explore P: 0.0100\n",
      "Episode: 862 Total reward: 199.0 Training loss: 0.3145 Explore P: 0.0100\n",
      "Episode: 863 Total reward: 199.0 Training loss: 0.3642 Explore P: 0.0100\n",
      "Episode: 864 Total reward: 199.0 Training loss: 0.4405 Explore P: 0.0100\n",
      "Episode: 865 Total reward: 199.0 Training loss: 0.2198 Explore P: 0.0100\n",
      "Episode: 866 Total reward: 199.0 Training loss: 0.4169 Explore P: 0.0100\n",
      "Episode: 867 Total reward: 199.0 Training loss: 0.3036 Explore P: 0.0100\n",
      "Episode: 868 Total reward: 199.0 Training loss: 0.1782 Explore P: 0.0100\n",
      "Episode: 869 Total reward: 199.0 Training loss: 0.2019 Explore P: 0.0100\n",
      "Episode: 870 Total reward: 199.0 Training loss: 0.2873 Explore P: 0.0100\n",
      "Episode: 871 Total reward: 199.0 Training loss: 0.2010 Explore P: 0.0100\n",
      "Episode: 872 Total reward: 199.0 Training loss: 0.3909 Explore P: 0.0100\n",
      "Episode: 873 Total reward: 199.0 Training loss: 0.2608 Explore P: 0.0100\n",
      "Episode: 874 Total reward: 199.0 Training loss: 0.2683 Explore P: 0.0100\n",
      "Episode: 875 Total reward: 199.0 Training loss: 0.2938 Explore P: 0.0100\n",
      "Episode: 876 Total reward: 199.0 Training loss: 0.1724 Explore P: 0.0100\n",
      "Episode: 877 Total reward: 199.0 Training loss: 0.1827 Explore P: 0.0100\n",
      "Episode: 878 Total reward: 199.0 Training loss: 0.1466 Explore P: 0.0100\n",
      "Episode: 879 Total reward: 199.0 Training loss: 0.3540 Explore P: 0.0100\n",
      "Episode: 880 Total reward: 199.0 Training loss: 0.1597 Explore P: 0.0100\n",
      "Episode: 881 Total reward: 199.0 Training loss: 0.3167 Explore P: 0.0100\n",
      "Episode: 882 Total reward: 199.0 Training loss: 94.7706 Explore P: 0.0100\n",
      "Episode: 883 Total reward: 199.0 Training loss: 21.5868 Explore P: 0.0100\n",
      "Episode: 884 Total reward: 199.0 Training loss: 174.0189 Explore P: 0.0100\n",
      "Episode: 885 Total reward: 199.0 Training loss: 0.1111 Explore P: 0.0100\n",
      "Episode: 886 Total reward: 199.0 Training loss: 0.1819 Explore P: 0.0100\n",
      "Episode: 887 Total reward: 199.0 Training loss: 147.4766 Explore P: 0.0100\n",
      "Episode: 888 Total reward: 199.0 Training loss: 0.3537 Explore P: 0.0100\n",
      "Episode: 889 Total reward: 199.0 Training loss: 278.4291 Explore P: 0.0100\n",
      "Episode: 890 Total reward: 199.0 Training loss: 0.2904 Explore P: 0.0100\n",
      "Episode: 891 Total reward: 199.0 Training loss: 0.1773 Explore P: 0.0100\n",
      "Episode: 892 Total reward: 199.0 Training loss: 0.3204 Explore P: 0.0100\n",
      "Episode: 893 Total reward: 199.0 Training loss: 0.4134 Explore P: 0.0100\n",
      "Episode: 894 Total reward: 199.0 Training loss: 0.2690 Explore P: 0.0100\n",
      "Episode: 895 Total reward: 199.0 Training loss: 0.2237 Explore P: 0.0100\n",
      "Episode: 896 Total reward: 199.0 Training loss: 0.3895 Explore P: 0.0100\n",
      "Episode: 897 Total reward: 199.0 Training loss: 0.3408 Explore P: 0.0100\n",
      "Episode: 898 Total reward: 199.0 Training loss: 123.8445 Explore P: 0.0100\n",
      "Episode: 899 Total reward: 199.0 Training loss: 0.4264 Explore P: 0.0100\n",
      "Episode: 900 Total reward: 199.0 Training loss: 0.1750 Explore P: 0.0100\n",
      "Episode: 901 Total reward: 199.0 Training loss: 0.3387 Explore P: 0.0100\n",
      "Episode: 902 Total reward: 199.0 Training loss: 0.2110 Explore P: 0.0100\n",
      "Episode: 903 Total reward: 199.0 Training loss: 0.1618 Explore P: 0.0100\n",
      "Episode: 904 Total reward: 199.0 Training loss: 0.2117 Explore P: 0.0100\n",
      "Episode: 905 Total reward: 199.0 Training loss: 0.0575 Explore P: 0.0100\n",
      "Episode: 906 Total reward: 199.0 Training loss: 0.4635 Explore P: 0.0100\n",
      "Episode: 907 Total reward: 199.0 Training loss: 0.1792 Explore P: 0.0100\n",
      "Episode: 908 Total reward: 199.0 Training loss: 0.1625 Explore P: 0.0100\n",
      "Episode: 909 Total reward: 199.0 Training loss: 292.3007 Explore P: 0.0100\n",
      "Episode: 910 Total reward: 199.0 Training loss: 0.1549 Explore P: 0.0100\n",
      "Episode: 911 Total reward: 199.0 Training loss: 0.0693 Explore P: 0.0100\n",
      "Episode: 912 Total reward: 199.0 Training loss: 0.2767 Explore P: 0.0100\n",
      "Episode: 913 Total reward: 199.0 Training loss: 0.1249 Explore P: 0.0100\n",
      "Episode: 914 Total reward: 199.0 Training loss: 124.8856 Explore P: 0.0100\n",
      "Episode: 915 Total reward: 199.0 Training loss: 0.1369 Explore P: 0.0100\n",
      "Episode: 916 Total reward: 199.0 Training loss: 0.0821 Explore P: 0.0100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 917 Total reward: 199.0 Training loss: 0.1109 Explore P: 0.0100\n",
      "Episode: 918 Total reward: 199.0 Training loss: 0.1198 Explore P: 0.0100\n",
      "Episode: 919 Total reward: 199.0 Training loss: 0.0885 Explore P: 0.0100\n",
      "Episode: 920 Total reward: 199.0 Training loss: 0.1392 Explore P: 0.0100\n",
      "Episode: 921 Total reward: 199.0 Training loss: 0.1546 Explore P: 0.0100\n",
      "Episode: 922 Total reward: 199.0 Training loss: 0.0953 Explore P: 0.0100\n",
      "Episode: 923 Total reward: 199.0 Training loss: 0.1760 Explore P: 0.0100\n",
      "Episode: 924 Total reward: 199.0 Training loss: 0.2316 Explore P: 0.0100\n",
      "Episode: 925 Total reward: 199.0 Training loss: 0.1424 Explore P: 0.0100\n",
      "Episode: 926 Total reward: 199.0 Training loss: 0.0719 Explore P: 0.0100\n",
      "Episode: 927 Total reward: 199.0 Training loss: 0.2688 Explore P: 0.0100\n",
      "Episode: 928 Total reward: 199.0 Training loss: 0.0978 Explore P: 0.0100\n",
      "Episode: 929 Total reward: 199.0 Training loss: 0.1860 Explore P: 0.0100\n",
      "Episode: 930 Total reward: 199.0 Training loss: 0.1680 Explore P: 0.0100\n",
      "Episode: 931 Total reward: 199.0 Training loss: 0.1282 Explore P: 0.0100\n",
      "Episode: 932 Total reward: 199.0 Training loss: 0.0681 Explore P: 0.0100\n",
      "Episode: 933 Total reward: 199.0 Training loss: 0.1820 Explore P: 0.0100\n",
      "Episode: 934 Total reward: 199.0 Training loss: 0.2032 Explore P: 0.0100\n",
      "Episode: 935 Total reward: 199.0 Training loss: 0.2167 Explore P: 0.0100\n",
      "Episode: 936 Total reward: 199.0 Training loss: 0.1116 Explore P: 0.0100\n",
      "Episode: 937 Total reward: 199.0 Training loss: 0.1073 Explore P: 0.0100\n",
      "Episode: 938 Total reward: 199.0 Training loss: 0.2056 Explore P: 0.0100\n",
      "Episode: 939 Total reward: 199.0 Training loss: 0.0944 Explore P: 0.0100\n",
      "Episode: 940 Total reward: 199.0 Training loss: 0.1159 Explore P: 0.0100\n",
      "Episode: 941 Total reward: 199.0 Training loss: 0.5379 Explore P: 0.0100\n",
      "Episode: 942 Total reward: 199.0 Training loss: 0.1134 Explore P: 0.0100\n",
      "Episode: 943 Total reward: 199.0 Training loss: 0.1949 Explore P: 0.0100\n",
      "Episode: 944 Total reward: 199.0 Training loss: 0.1400 Explore P: 0.0100\n",
      "Episode: 945 Total reward: 199.0 Training loss: 0.2943 Explore P: 0.0100\n",
      "Episode: 946 Total reward: 199.0 Training loss: 0.2269 Explore P: 0.0100\n",
      "Episode: 947 Total reward: 199.0 Training loss: 0.2467 Explore P: 0.0100\n",
      "Episode: 948 Total reward: 199.0 Training loss: 0.1440 Explore P: 0.0100\n",
      "Episode: 949 Total reward: 199.0 Training loss: 47.5868 Explore P: 0.0100\n",
      "Episode: 950 Total reward: 199.0 Training loss: 0.2552 Explore P: 0.0100\n",
      "Episode: 951 Total reward: 199.0 Training loss: 367.2444 Explore P: 0.0100\n",
      "Episode: 952 Total reward: 199.0 Training loss: 0.1985 Explore P: 0.0100\n",
      "Episode: 953 Total reward: 199.0 Training loss: 0.1790 Explore P: 0.0100\n",
      "Episode: 954 Total reward: 199.0 Training loss: 159.6455 Explore P: 0.0100\n",
      "Episode: 955 Total reward: 199.0 Training loss: 155.8326 Explore P: 0.0100\n",
      "Episode: 956 Total reward: 199.0 Training loss: 0.3274 Explore P: 0.0100\n",
      "Episode: 957 Total reward: 199.0 Training loss: 0.1528 Explore P: 0.0100\n",
      "Episode: 958 Total reward: 199.0 Training loss: 0.1913 Explore P: 0.0100\n",
      "Episode: 959 Total reward: 199.0 Training loss: 0.5218 Explore P: 0.0100\n",
      "Episode: 960 Total reward: 199.0 Training loss: 0.3506 Explore P: 0.0100\n",
      "Episode: 961 Total reward: 199.0 Training loss: 0.1319 Explore P: 0.0100\n",
      "Episode: 962 Total reward: 199.0 Training loss: 0.2469 Explore P: 0.0100\n",
      "Episode: 963 Total reward: 199.0 Training loss: 0.0878 Explore P: 0.0100\n",
      "Episode: 964 Total reward: 199.0 Training loss: 0.4342 Explore P: 0.0100\n",
      "Episode: 965 Total reward: 199.0 Training loss: 0.3026 Explore P: 0.0100\n",
      "Episode: 966 Total reward: 199.0 Training loss: 0.1864 Explore P: 0.0100\n",
      "Episode: 967 Total reward: 199.0 Training loss: 0.1771 Explore P: 0.0100\n",
      "Episode: 968 Total reward: 199.0 Training loss: 0.4112 Explore P: 0.0100\n",
      "Episode: 969 Total reward: 199.0 Training loss: 0.3849 Explore P: 0.0100\n",
      "Episode: 970 Total reward: 199.0 Training loss: 0.2135 Explore P: 0.0100\n",
      "Episode: 971 Total reward: 199.0 Training loss: 0.1904 Explore P: 0.0100\n",
      "Episode: 972 Total reward: 199.0 Training loss: 0.2689 Explore P: 0.0100\n",
      "Episode: 973 Total reward: 199.0 Training loss: 0.2810 Explore P: 0.0100\n",
      "Episode: 974 Total reward: 199.0 Training loss: 0.2049 Explore P: 0.0100\n",
      "Episode: 975 Total reward: 199.0 Training loss: 0.6716 Explore P: 0.0100\n",
      "Episode: 976 Total reward: 199.0 Training loss: 0.5252 Explore P: 0.0100\n",
      "Episode: 977 Total reward: 199.0 Training loss: 0.5673 Explore P: 0.0100\n",
      "Episode: 978 Total reward: 199.0 Training loss: 0.7470 Explore P: 0.0100\n",
      "Episode: 979 Total reward: 199.0 Training loss: 0.3235 Explore P: 0.0100\n",
      "Episode: 980 Total reward: 199.0 Training loss: 0.3253 Explore P: 0.0100\n",
      "Episode: 981 Total reward: 199.0 Training loss: 0.5227 Explore P: 0.0100\n",
      "Episode: 982 Total reward: 199.0 Training loss: 0.5206 Explore P: 0.0100\n",
      "Episode: 983 Total reward: 199.0 Training loss: 0.2905 Explore P: 0.0100\n",
      "Episode: 984 Total reward: 199.0 Training loss: 0.2194 Explore P: 0.0100\n",
      "Episode: 985 Total reward: 199.0 Training loss: 0.2305 Explore P: 0.0100\n",
      "Episode: 986 Total reward: 199.0 Training loss: 143.9380 Explore P: 0.0100\n",
      "Episode: 987 Total reward: 199.0 Training loss: 0.5476 Explore P: 0.0100\n",
      "Episode: 988 Total reward: 199.0 Training loss: 0.3951 Explore P: 0.0100\n",
      "Episode: 989 Total reward: 199.0 Training loss: 0.4593 Explore P: 0.0100\n",
      "Episode: 990 Total reward: 199.0 Training loss: 0.5431 Explore P: 0.0100\n",
      "Episode: 991 Total reward: 199.0 Training loss: 0.3753 Explore P: 0.0100\n",
      "Episode: 992 Total reward: 199.0 Training loss: 0.2692 Explore P: 0.0100\n",
      "Episode: 993 Total reward: 199.0 Training loss: 0.2461 Explore P: 0.0100\n",
      "Episode: 994 Total reward: 199.0 Training loss: 0.2119 Explore P: 0.0100\n",
      "Episode: 995 Total reward: 199.0 Training loss: 0.8275 Explore P: 0.0100\n",
      "Episode: 996 Total reward: 199.0 Training loss: 0.2215 Explore P: 0.0100\n",
      "Episode: 997 Total reward: 199.0 Training loss: 0.3854 Explore P: 0.0100\n",
      "Episode: 998 Total reward: 199.0 Training loss: 0.3168 Explore P: 0.0100\n",
      "Episode: 999 Total reward: 199.0 Training loss: 403.8680 Explore P: 0.0100\n",
      "Episode: 1000 Total reward: 199.0 Training loss: 0.4723 Explore P: 0.0100\n",
      "Episode: 1001 Total reward: 199.0 Training loss: 0.4692 Explore P: 0.0100\n",
      "Episode: 1002 Total reward: 199.0 Training loss: 0.3138 Explore P: 0.0100\n",
      "Episode: 1003 Total reward: 199.0 Training loss: 0.5019 Explore P: 0.0100\n",
      "Episode: 1004 Total reward: 199.0 Training loss: 0.4632 Explore P: 0.0100\n",
      "Episode: 1005 Total reward: 199.0 Training loss: 0.2997 Explore P: 0.0100\n",
      "Episode: 1006 Total reward: 199.0 Training loss: 0.3924 Explore P: 0.0100\n",
      "Episode: 1007 Total reward: 199.0 Training loss: 0.6803 Explore P: 0.0100\n",
      "Episode: 1008 Total reward: 199.0 Training loss: 0.6696 Explore P: 0.0100\n",
      "Episode: 1009 Total reward: 199.0 Training loss: 0.5557 Explore P: 0.0100\n",
      "Episode: 1010 Total reward: 199.0 Training loss: 358.6624 Explore P: 0.0100\n",
      "Episode: 1011 Total reward: 199.0 Training loss: 1.4973 Explore P: 0.0100\n",
      "Episode: 1012 Total reward: 199.0 Training loss: 0.5423 Explore P: 0.0100\n",
      "Episode: 1013 Total reward: 199.0 Training loss: 0.3721 Explore P: 0.0100\n",
      "Episode: 1014 Total reward: 199.0 Training loss: 0.5902 Explore P: 0.0100\n",
      "Episode: 1015 Total reward: 199.0 Training loss: 0.3356 Explore P: 0.0100\n",
      "Episode: 1016 Total reward: 199.0 Training loss: 0.3392 Explore P: 0.0100\n",
      "Episode: 1017 Total reward: 199.0 Training loss: 0.3330 Explore P: 0.0100\n",
      "Episode: 1018 Total reward: 199.0 Training loss: 0.2638 Explore P: 0.0100\n",
      "Episode: 1019 Total reward: 199.0 Training loss: 0.3002 Explore P: 0.0100\n",
      "Episode: 1020 Total reward: 199.0 Training loss: 0.3758 Explore P: 0.0100\n",
      "Episode: 1021 Total reward: 199.0 Training loss: 0.3022 Explore P: 0.0100\n",
      "Episode: 1022 Total reward: 199.0 Training loss: 348.3476 Explore P: 0.0100\n",
      "Episode: 1023 Total reward: 199.0 Training loss: 0.2881 Explore P: 0.0100\n",
      "Episode: 1024 Total reward: 199.0 Training loss: 0.2197 Explore P: 0.0100\n",
      "Episode: 1025 Total reward: 199.0 Training loss: 0.3746 Explore P: 0.0100\n",
      "Episode: 1026 Total reward: 199.0 Training loss: 0.4006 Explore P: 0.0100\n",
      "Episode: 1027 Total reward: 199.0 Training loss: 0.1361 Explore P: 0.0100\n",
      "Episode: 1028 Total reward: 199.0 Training loss: 0.4367 Explore P: 0.0100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1029 Total reward: 199.0 Training loss: 0.2787 Explore P: 0.0100\n",
      "Episode: 1030 Total reward: 199.0 Training loss: 0.2215 Explore P: 0.0100\n",
      "Episode: 1031 Total reward: 199.0 Training loss: 0.3167 Explore P: 0.0100\n",
      "Episode: 1032 Total reward: 199.0 Training loss: 0.3060 Explore P: 0.0100\n",
      "Episode: 1033 Total reward: 165.0 Training loss: 0.7163 Explore P: 0.0100\n",
      "Episode: 1034 Total reward: 199.0 Training loss: 279.0655 Explore P: 0.0100\n",
      "Episode: 1035 Total reward: 199.0 Training loss: 0.2836 Explore P: 0.0100\n",
      "Episode: 1036 Total reward: 199.0 Training loss: 150.7680 Explore P: 0.0100\n",
      "Episode: 1037 Total reward: 199.0 Training loss: 0.4395 Explore P: 0.0100\n",
      "Episode: 1038 Total reward: 199.0 Training loss: 0.5257 Explore P: 0.0100\n",
      "Episode: 1039 Total reward: 199.0 Training loss: 0.5138 Explore P: 0.0100\n",
      "Episode: 1040 Total reward: 199.0 Training loss: 0.4526 Explore P: 0.0100\n",
      "Episode: 1041 Total reward: 199.0 Training loss: 0.2019 Explore P: 0.0100\n",
      "Episode: 1042 Total reward: 199.0 Training loss: 0.3006 Explore P: 0.0100\n",
      "Episode: 1043 Total reward: 199.0 Training loss: 0.2804 Explore P: 0.0100\n",
      "Episode: 1044 Total reward: 199.0 Training loss: 0.1969 Explore P: 0.0100\n",
      "Episode: 1045 Total reward: 199.0 Training loss: 0.1696 Explore P: 0.0100\n",
      "Episode: 1046 Total reward: 199.0 Training loss: 0.6710 Explore P: 0.0100\n",
      "Episode: 1047 Total reward: 199.0 Training loss: 0.3173 Explore P: 0.0100\n",
      "Episode: 1048 Total reward: 199.0 Training loss: 52.1696 Explore P: 0.0100\n",
      "Episode: 1049 Total reward: 199.0 Training loss: 0.5588 Explore P: 0.0100\n",
      "Episode: 1050 Total reward: 199.0 Training loss: 0.2320 Explore P: 0.0100\n",
      "Episode: 1051 Total reward: 199.0 Training loss: 0.4277 Explore P: 0.0100\n",
      "Episode: 1052 Total reward: 199.0 Training loss: 0.2313 Explore P: 0.0100\n",
      "Episode: 1053 Total reward: 199.0 Training loss: 0.2009 Explore P: 0.0100\n",
      "Episode: 1054 Total reward: 199.0 Training loss: 0.4418 Explore P: 0.0100\n",
      "Episode: 1055 Total reward: 199.0 Training loss: 0.2260 Explore P: 0.0100\n",
      "Episode: 1056 Total reward: 199.0 Training loss: 0.5473 Explore P: 0.0100\n",
      "Episode: 1057 Total reward: 199.0 Training loss: 0.2709 Explore P: 0.0100\n",
      "Episode: 1058 Total reward: 199.0 Training loss: 0.2551 Explore P: 0.0100\n",
      "Episode: 1059 Total reward: 199.0 Training loss: 0.1173 Explore P: 0.0100\n",
      "Episode: 1060 Total reward: 199.0 Training loss: 0.2026 Explore P: 0.0100\n",
      "Episode: 1061 Total reward: 199.0 Training loss: 0.4574 Explore P: 0.0100\n",
      "Episode: 1062 Total reward: 199.0 Training loss: 0.3487 Explore P: 0.0100\n",
      "Episode: 1063 Total reward: 199.0 Training loss: 0.3572 Explore P: 0.0100\n",
      "Episode: 1064 Total reward: 199.0 Training loss: 0.4157 Explore P: 0.0100\n",
      "Episode: 1065 Total reward: 199.0 Training loss: 1.2848 Explore P: 0.0100\n",
      "Episode: 1066 Total reward: 199.0 Training loss: 0.1061 Explore P: 0.0100\n",
      "Episode: 1067 Total reward: 199.0 Training loss: 0.1348 Explore P: 0.0100\n",
      "Episode: 1068 Total reward: 199.0 Training loss: 1.5019 Explore P: 0.0100\n",
      "Episode: 1069 Total reward: 199.0 Training loss: 0.2713 Explore P: 0.0100\n",
      "Episode: 1070 Total reward: 199.0 Training loss: 0.2214 Explore P: 0.0100\n",
      "Episode: 1071 Total reward: 199.0 Training loss: 0.1403 Explore P: 0.0100\n",
      "Episode: 1072 Total reward: 199.0 Training loss: 0.3949 Explore P: 0.0100\n",
      "Episode: 1073 Total reward: 199.0 Training loss: 0.3703 Explore P: 0.0100\n",
      "Episode: 1074 Total reward: 199.0 Training loss: 0.4200 Explore P: 0.0100\n",
      "Episode: 1075 Total reward: 199.0 Training loss: 199.6966 Explore P: 0.0100\n",
      "Episode: 1076 Total reward: 199.0 Training loss: 0.6318 Explore P: 0.0100\n",
      "Episode: 1077 Total reward: 199.0 Training loss: 0.1660 Explore P: 0.0100\n",
      "Episode: 1078 Total reward: 199.0 Training loss: 0.1686 Explore P: 0.0100\n",
      "Episode: 1079 Total reward: 199.0 Training loss: 0.1676 Explore P: 0.0100\n",
      "Episode: 1080 Total reward: 199.0 Training loss: 0.2167 Explore P: 0.0100\n",
      "Episode: 1081 Total reward: 199.0 Training loss: 1.1778 Explore P: 0.0100\n",
      "Episode: 1082 Total reward: 199.0 Training loss: 0.3682 Explore P: 0.0100\n",
      "Episode: 1083 Total reward: 199.0 Training loss: 0.2415 Explore P: 0.0100\n",
      "Episode: 1084 Total reward: 199.0 Training loss: 0.3759 Explore P: 0.0100\n",
      "Episode: 1085 Total reward: 199.0 Training loss: 0.1544 Explore P: 0.0100\n",
      "Episode: 1086 Total reward: 199.0 Training loss: 0.6611 Explore P: 0.0100\n",
      "Episode: 1087 Total reward: 199.0 Training loss: 0.5093 Explore P: 0.0100\n",
      "Episode: 1088 Total reward: 199.0 Training loss: 0.4309 Explore P: 0.0100\n",
      "Episode: 1089 Total reward: 199.0 Training loss: 114.1071 Explore P: 0.0100\n",
      "Episode: 1090 Total reward: 199.0 Training loss: 0.2603 Explore P: 0.0100\n",
      "Episode: 1091 Total reward: 199.0 Training loss: 0.1984 Explore P: 0.0100\n",
      "Episode: 1092 Total reward: 199.0 Training loss: 0.5281 Explore P: 0.0100\n",
      "Episode: 1093 Total reward: 199.0 Training loss: 0.1794 Explore P: 0.0100\n",
      "Episode: 1094 Total reward: 199.0 Training loss: 154.5252 Explore P: 0.0100\n",
      "Episode: 1095 Total reward: 199.0 Training loss: 0.1422 Explore P: 0.0100\n",
      "Episode: 1096 Total reward: 199.0 Training loss: 0.1509 Explore P: 0.0100\n",
      "Episode: 1097 Total reward: 199.0 Training loss: 0.1731 Explore P: 0.0100\n",
      "Episode: 1098 Total reward: 199.0 Training loss: 0.2075 Explore P: 0.0100\n",
      "Episode: 1099 Total reward: 199.0 Training loss: 0.4761 Explore P: 0.0100\n",
      "Episode: 1100 Total reward: 199.0 Training loss: 0.1118 Explore P: 0.0100\n",
      "Episode: 1101 Total reward: 199.0 Training loss: 0.3655 Explore P: 0.0100\n",
      "Episode: 1102 Total reward: 199.0 Training loss: 0.2118 Explore P: 0.0100\n",
      "Episode: 1103 Total reward: 199.0 Training loss: 0.2210 Explore P: 0.0100\n",
      "Episode: 1104 Total reward: 199.0 Training loss: 147.7824 Explore P: 0.0100\n",
      "Episode: 1105 Total reward: 199.0 Training loss: 0.2037 Explore P: 0.0100\n",
      "Episode: 1106 Total reward: 199.0 Training loss: 0.1335 Explore P: 0.0100\n",
      "Episode: 1107 Total reward: 199.0 Training loss: 0.1535 Explore P: 0.0100\n",
      "Episode: 1108 Total reward: 199.0 Training loss: 0.2051 Explore P: 0.0100\n",
      "Episode: 1109 Total reward: 199.0 Training loss: 0.2577 Explore P: 0.0100\n",
      "Episode: 1110 Total reward: 199.0 Training loss: 0.2327 Explore P: 0.0100\n",
      "Episode: 1111 Total reward: 199.0 Training loss: 0.2621 Explore P: 0.0100\n",
      "Episode: 1112 Total reward: 199.0 Training loss: 0.2187 Explore P: 0.0100\n",
      "Episode: 1113 Total reward: 199.0 Training loss: 0.4848 Explore P: 0.0100\n",
      "Episode: 1114 Total reward: 199.0 Training loss: 0.3975 Explore P: 0.0100\n",
      "Episode: 1115 Total reward: 199.0 Training loss: 0.2688 Explore P: 0.0100\n",
      "Episode: 1116 Total reward: 199.0 Training loss: 146.0111 Explore P: 0.0100\n",
      "Episode: 1117 Total reward: 199.0 Training loss: 0.1830 Explore P: 0.0100\n",
      "Episode: 1118 Total reward: 199.0 Training loss: 0.2605 Explore P: 0.0100\n",
      "Episode: 1119 Total reward: 199.0 Training loss: 0.1884 Explore P: 0.0100\n",
      "Episode: 1120 Total reward: 199.0 Training loss: 0.1807 Explore P: 0.0100\n",
      "Episode: 1121 Total reward: 199.0 Training loss: 0.2533 Explore P: 0.0100\n",
      "Episode: 1122 Total reward: 199.0 Training loss: 0.2551 Explore P: 0.0100\n",
      "Episode: 1123 Total reward: 199.0 Training loss: 0.3962 Explore P: 0.0100\n",
      "Episode: 1124 Total reward: 199.0 Training loss: 0.3174 Explore P: 0.0100\n",
      "Episode: 1125 Total reward: 199.0 Training loss: 0.1246 Explore P: 0.0100\n",
      "Episode: 1126 Total reward: 199.0 Training loss: 0.3102 Explore P: 0.0100\n",
      "Episode: 1127 Total reward: 199.0 Training loss: 0.2049 Explore P: 0.0100\n",
      "Episode: 1128 Total reward: 199.0 Training loss: 0.2383 Explore P: 0.0100\n",
      "Episode: 1129 Total reward: 199.0 Training loss: 0.1950 Explore P: 0.0100\n",
      "Episode: 1130 Total reward: 199.0 Training loss: 0.1304 Explore P: 0.0100\n",
      "Episode: 1131 Total reward: 199.0 Training loss: 0.3666 Explore P: 0.0100\n",
      "Episode: 1132 Total reward: 199.0 Training loss: 0.2482 Explore P: 0.0100\n",
      "Episode: 1133 Total reward: 199.0 Training loss: 0.5696 Explore P: 0.0100\n",
      "Episode: 1134 Total reward: 199.0 Training loss: 3.0379 Explore P: 0.0100\n",
      "Episode: 1135 Total reward: 170.0 Training loss: 10.8489 Explore P: 0.0100\n",
      "Episode: 1136 Total reward: 10.0 Training loss: 13.6154 Explore P: 0.0100\n",
      "Episode: 1137 Total reward: 10.0 Training loss: 15.8784 Explore P: 0.0100\n",
      "Episode: 1138 Total reward: 12.0 Training loss: 17.4223 Explore P: 0.0100\n",
      "Episode: 1139 Total reward: 10.0 Training loss: 15.4978 Explore P: 0.0100\n",
      "Episode: 1140 Total reward: 8.0 Training loss: 20.8112 Explore P: 0.0100\n",
      "Episode: 1141 Total reward: 12.0 Training loss: 33.2294 Explore P: 0.0100\n",
      "Episode: 1142 Total reward: 11.0 Training loss: 30.7576 Explore P: 0.0100\n",
      "Episode: 1143 Total reward: 10.0 Training loss: 1520.2538 Explore P: 0.0100\n",
      "Episode: 1144 Total reward: 11.0 Training loss: 33.1774 Explore P: 0.0100\n",
      "Episode: 1145 Total reward: 8.0 Training loss: 21.5935 Explore P: 0.0100\n",
      "Episode: 1146 Total reward: 7.0 Training loss: 23.2464 Explore P: 0.0100\n",
      "Episode: 1147 Total reward: 10.0 Training loss: 26.7547 Explore P: 0.0100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1148 Total reward: 12.0 Training loss: 29.3328 Explore P: 0.0100\n",
      "Episode: 1149 Total reward: 8.0 Training loss: 2511.5962 Explore P: 0.0100\n",
      "Episode: 1150 Total reward: 8.0 Training loss: 35.2401 Explore P: 0.0100\n",
      "Episode: 1151 Total reward: 8.0 Training loss: 3644.5801 Explore P: 0.0100\n",
      "Episode: 1152 Total reward: 10.0 Training loss: 26.5424 Explore P: 0.0100\n",
      "Episode: 1153 Total reward: 8.0 Training loss: 41.1830 Explore P: 0.0100\n",
      "Episode: 1154 Total reward: 9.0 Training loss: 35.6842 Explore P: 0.0100\n",
      "Episode: 1155 Total reward: 8.0 Training loss: 25.6014 Explore P: 0.0100\n",
      "Episode: 1156 Total reward: 11.0 Training loss: 14.1915 Explore P: 0.0100\n",
      "Episode: 1157 Total reward: 9.0 Training loss: 11.1316 Explore P: 0.0100\n",
      "Episode: 1158 Total reward: 10.0 Training loss: 7352.8975 Explore P: 0.0100\n",
      "Episode: 1159 Total reward: 10.0 Training loss: 3767.0461 Explore P: 0.0100\n",
      "Episode: 1160 Total reward: 9.0 Training loss: 5.2918 Explore P: 0.0100\n",
      "Episode: 1161 Total reward: 9.0 Training loss: 3.4745 Explore P: 0.0100\n",
      "Episode: 1162 Total reward: 8.0 Training loss: 2.8340 Explore P: 0.0100\n",
      "Episode: 1163 Total reward: 12.0 Training loss: 5.3057 Explore P: 0.0100\n",
      "Episode: 1164 Total reward: 11.0 Training loss: 1.8219 Explore P: 0.0100\n",
      "Episode: 1165 Total reward: 9.0 Training loss: 4.4264 Explore P: 0.0100\n",
      "Episode: 1166 Total reward: 11.0 Training loss: 5.2429 Explore P: 0.0100\n",
      "Episode: 1167 Total reward: 10.0 Training loss: 9.0768 Explore P: 0.0100\n",
      "Episode: 1168 Total reward: 12.0 Training loss: 3.6564 Explore P: 0.0100\n",
      "Episode: 1169 Total reward: 8.0 Training loss: 2.9943 Explore P: 0.0100\n",
      "Episode: 1170 Total reward: 9.0 Training loss: 1.3777 Explore P: 0.0100\n",
      "Episode: 1171 Total reward: 12.0 Training loss: 3073.1567 Explore P: 0.0100\n",
      "Episode: 1172 Total reward: 10.0 Training loss: 11.2866 Explore P: 0.0100\n",
      "Episode: 1173 Total reward: 13.0 Training loss: 2361.6821 Explore P: 0.0100\n",
      "Episode: 1174 Total reward: 199.0 Training loss: 1.7984 Explore P: 0.0100\n",
      "Episode: 1175 Total reward: 180.0 Training loss: 3.0357 Explore P: 0.0100\n",
      "Episode: 1176 Total reward: 199.0 Training loss: 0.7525 Explore P: 0.0100\n",
      "Episode: 1177 Total reward: 12.0 Training loss: 3.0394 Explore P: 0.0100\n",
      "Episode: 1178 Total reward: 12.0 Training loss: 1.3247 Explore P: 0.0100\n",
      "Episode: 1179 Total reward: 9.0 Training loss: 440.9340 Explore P: 0.0100\n",
      "Episode: 1180 Total reward: 10.0 Training loss: 1.8977 Explore P: 0.0100\n",
      "Episode: 1181 Total reward: 8.0 Training loss: 2.6287 Explore P: 0.0100\n",
      "Episode: 1182 Total reward: 8.0 Training loss: 1.5966 Explore P: 0.0100\n",
      "Episode: 1183 Total reward: 10.0 Training loss: 3.2499 Explore P: 0.0100\n",
      "Episode: 1184 Total reward: 11.0 Training loss: 339.3248 Explore P: 0.0100\n",
      "Episode: 1185 Total reward: 9.0 Training loss: 7.1795 Explore P: 0.0100\n",
      "Episode: 1186 Total reward: 8.0 Training loss: 2.4656 Explore P: 0.0100\n",
      "Episode: 1187 Total reward: 7.0 Training loss: 1113.4392 Explore P: 0.0100\n",
      "Episode: 1188 Total reward: 7.0 Training loss: 3.9041 Explore P: 0.0100\n",
      "Episode: 1189 Total reward: 9.0 Training loss: 2.3961 Explore P: 0.0100\n",
      "Episode: 1190 Total reward: 10.0 Training loss: 476.7811 Explore P: 0.0100\n",
      "Episode: 1191 Total reward: 9.0 Training loss: 2.1412 Explore P: 0.0100\n",
      "Episode: 1192 Total reward: 8.0 Training loss: 4.0800 Explore P: 0.0100\n",
      "Episode: 1193 Total reward: 9.0 Training loss: 668.9236 Explore P: 0.0100\n",
      "Episode: 1194 Total reward: 8.0 Training loss: 1065.2341 Explore P: 0.0100\n",
      "Episode: 1195 Total reward: 11.0 Training loss: 10.5087 Explore P: 0.0100\n",
      "Episode: 1196 Total reward: 7.0 Training loss: 1133.9908 Explore P: 0.0100\n",
      "Episode: 1197 Total reward: 9.0 Training loss: 3.1421 Explore P: 0.0100\n",
      "Episode: 1198 Total reward: 10.0 Training loss: 4.2984 Explore P: 0.0100\n",
      "Episode: 1199 Total reward: 7.0 Training loss: 250.4210 Explore P: 0.0100\n",
      "Episode: 1200 Total reward: 9.0 Training loss: 3.2935 Explore P: 0.0100\n",
      "Episode: 1201 Total reward: 10.0 Training loss: 21.6220 Explore P: 0.0100\n",
      "Episode: 1202 Total reward: 8.0 Training loss: 5.2029 Explore P: 0.0100\n",
      "Episode: 1203 Total reward: 11.0 Training loss: 1039.3136 Explore P: 0.0100\n",
      "Episode: 1204 Total reward: 10.0 Training loss: 4.5258 Explore P: 0.0100\n",
      "Episode: 1205 Total reward: 8.0 Training loss: 1903.9603 Explore P: 0.0100\n",
      "Episode: 1206 Total reward: 8.0 Training loss: 15.2912 Explore P: 0.0100\n",
      "Episode: 1207 Total reward: 10.0 Training loss: 3.1501 Explore P: 0.0100\n",
      "Episode: 1208 Total reward: 9.0 Training loss: 5.2499 Explore P: 0.0100\n",
      "Episode: 1209 Total reward: 11.0 Training loss: 1324.5144 Explore P: 0.0100\n",
      "Episode: 1210 Total reward: 11.0 Training loss: 2.8814 Explore P: 0.0100\n",
      "Episode: 1211 Total reward: 11.0 Training loss: 2.0285 Explore P: 0.0100\n",
      "Episode: 1212 Total reward: 10.0 Training loss: 4.9160 Explore P: 0.0100\n",
      "Episode: 1213 Total reward: 9.0 Training loss: 3.9196 Explore P: 0.0100\n",
      "Episode: 1214 Total reward: 7.0 Training loss: 5.0156 Explore P: 0.0100\n",
      "Episode: 1215 Total reward: 10.0 Training loss: 2.2001 Explore P: 0.0100\n",
      "Episode: 1216 Total reward: 11.0 Training loss: 1665.8470 Explore P: 0.0100\n",
      "Episode: 1217 Total reward: 9.0 Training loss: 2.4831 Explore P: 0.0100\n",
      "Episode: 1218 Total reward: 10.0 Training loss: 1.2328 Explore P: 0.0100\n",
      "Episode: 1219 Total reward: 17.0 Training loss: 4.0630 Explore P: 0.0100\n",
      "Episode: 1220 Total reward: 15.0 Training loss: 1.7013 Explore P: 0.0100\n",
      "Episode: 1221 Total reward: 102.0 Training loss: 420.4478 Explore P: 0.0100\n",
      "Episode: 1222 Total reward: 199.0 Training loss: 410.2014 Explore P: 0.0100\n",
      "Episode: 1223 Total reward: 199.0 Training loss: 451.4930 Explore P: 0.0100\n",
      "Episode: 1224 Total reward: 199.0 Training loss: 291.7295 Explore P: 0.0100\n",
      "Episode: 1225 Total reward: 199.0 Training loss: 4.3408 Explore P: 0.0100\n",
      "Episode: 1226 Total reward: 199.0 Training loss: 1.3838 Explore P: 0.0100\n",
      "Episode: 1227 Total reward: 199.0 Training loss: 2.2423 Explore P: 0.0100\n",
      "Episode: 1228 Total reward: 199.0 Training loss: 339.6191 Explore P: 0.0100\n",
      "Episode: 1229 Total reward: 199.0 Training loss: 4.7481 Explore P: 0.0100\n",
      "Episode: 1230 Total reward: 199.0 Training loss: 3.2813 Explore P: 0.0100\n",
      "Episode: 1231 Total reward: 199.0 Training loss: 5.4852 Explore P: 0.0100\n",
      "Episode: 1232 Total reward: 199.0 Training loss: 3.5574 Explore P: 0.0100\n",
      "Episode: 1233 Total reward: 199.0 Training loss: 5.3705 Explore P: 0.0100\n",
      "Episode: 1234 Total reward: 199.0 Training loss: 3.8859 Explore P: 0.0100\n",
      "Episode: 1235 Total reward: 199.0 Training loss: 1.3864 Explore P: 0.0100\n",
      "Episode: 1236 Total reward: 199.0 Training loss: 5.6403 Explore P: 0.0100\n",
      "Episode: 1237 Total reward: 199.0 Training loss: 3.9043 Explore P: 0.0100\n",
      "Episode: 1238 Total reward: 199.0 Training loss: 6.9502 Explore P: 0.0100\n",
      "Episode: 1239 Total reward: 199.0 Training loss: 2.3348 Explore P: 0.0100\n",
      "Episode: 1240 Total reward: 199.0 Training loss: 4.7422 Explore P: 0.0100\n",
      "Episode: 1241 Total reward: 199.0 Training loss: 98.0308 Explore P: 0.0100\n",
      "Episode: 1242 Total reward: 199.0 Training loss: 13.4103 Explore P: 0.0100\n",
      "Episode: 1243 Total reward: 199.0 Training loss: 4.5538 Explore P: 0.0100\n",
      "Episode: 1244 Total reward: 199.0 Training loss: 3.4000 Explore P: 0.0100\n",
      "Episode: 1245 Total reward: 199.0 Training loss: 5.4241 Explore P: 0.0100\n",
      "Episode: 1246 Total reward: 199.0 Training loss: 5.4735 Explore P: 0.0100\n",
      "Episode: 1247 Total reward: 199.0 Training loss: 1.1170 Explore P: 0.0100\n",
      "Episode: 1248 Total reward: 199.0 Training loss: 3.7595 Explore P: 0.0100\n",
      "Episode: 1249 Total reward: 199.0 Training loss: 0.5495 Explore P: 0.0100\n",
      "Episode: 1250 Total reward: 199.0 Training loss: 11.8954 Explore P: 0.0100\n",
      "Episode: 1251 Total reward: 199.0 Training loss: 3.6780 Explore P: 0.0100\n",
      "Episode: 1252 Total reward: 199.0 Training loss: 712.0688 Explore P: 0.0100\n",
      "Episode: 1253 Total reward: 199.0 Training loss: 1.5359 Explore P: 0.0100\n",
      "Episode: 1254 Total reward: 199.0 Training loss: 5.4210 Explore P: 0.0100\n",
      "Episode: 1255 Total reward: 199.0 Training loss: 5.3889 Explore P: 0.0100\n",
      "Episode: 1256 Total reward: 199.0 Training loss: 30.8565 Explore P: 0.0100\n",
      "Episode: 1257 Total reward: 199.0 Training loss: 1.8835 Explore P: 0.0100\n",
      "Episode: 1258 Total reward: 199.0 Training loss: 419.5950 Explore P: 0.0100\n",
      "Episode: 1259 Total reward: 199.0 Training loss: 6.0646 Explore P: 0.0100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1260 Total reward: 199.0 Training loss: 1.2059 Explore P: 0.0100\n",
      "Episode: 1261 Total reward: 199.0 Training loss: 1.0040 Explore P: 0.0100\n",
      "Episode: 1262 Total reward: 199.0 Training loss: 0.7826 Explore P: 0.0100\n",
      "Episode: 1263 Total reward: 199.0 Training loss: 3.9103 Explore P: 0.0100\n",
      "Episode: 1264 Total reward: 199.0 Training loss: 2.1098 Explore P: 0.0100\n",
      "Episode: 1265 Total reward: 199.0 Training loss: 4.0350 Explore P: 0.0100\n",
      "Episode: 1266 Total reward: 199.0 Training loss: 173.4200 Explore P: 0.0100\n",
      "Episode: 1267 Total reward: 199.0 Training loss: 1.2779 Explore P: 0.0100\n",
      "Episode: 1268 Total reward: 199.0 Training loss: 1.0412 Explore P: 0.0100\n",
      "Episode: 1269 Total reward: 199.0 Training loss: 0.9413 Explore P: 0.0100\n",
      "Episode: 1270 Total reward: 199.0 Training loss: 0.7040 Explore P: 0.0100\n",
      "Episode: 1271 Total reward: 199.0 Training loss: 1.4544 Explore P: 0.0100\n",
      "Episode: 1272 Total reward: 199.0 Training loss: 0.6997 Explore P: 0.0100\n",
      "Episode: 1273 Total reward: 199.0 Training loss: 292.4230 Explore P: 0.0100\n",
      "Episode: 1274 Total reward: 199.0 Training loss: 0.6637 Explore P: 0.0100\n",
      "Episode: 1275 Total reward: 199.0 Training loss: 0.3779 Explore P: 0.0100\n",
      "Episode: 1276 Total reward: 199.0 Training loss: 0.2807 Explore P: 0.0100\n",
      "Episode: 1277 Total reward: 199.0 Training loss: 0.7619 Explore P: 0.0100\n",
      "Episode: 1278 Total reward: 199.0 Training loss: 0.7168 Explore P: 0.0100\n",
      "Episode: 1279 Total reward: 199.0 Training loss: 0.3371 Explore P: 0.0100\n",
      "Episode: 1280 Total reward: 199.0 Training loss: 0.3668 Explore P: 0.0100\n",
      "Episode: 1281 Total reward: 199.0 Training loss: 0.3579 Explore P: 0.0100\n",
      "Episode: 1282 Total reward: 199.0 Training loss: 0.8476 Explore P: 0.0100\n",
      "Episode: 1283 Total reward: 199.0 Training loss: 35.0737 Explore P: 0.0100\n",
      "Episode: 1284 Total reward: 199.0 Training loss: 0.2935 Explore P: 0.0100\n",
      "Episode: 1285 Total reward: 199.0 Training loss: 0.3815 Explore P: 0.0100\n",
      "Episode: 1286 Total reward: 199.0 Training loss: 0.3369 Explore P: 0.0100\n",
      "Episode: 1287 Total reward: 199.0 Training loss: 0.2309 Explore P: 0.0100\n",
      "Episode: 1288 Total reward: 199.0 Training loss: 0.2959 Explore P: 0.0100\n",
      "Episode: 1289 Total reward: 199.0 Training loss: 0.3572 Explore P: 0.0100\n",
      "Episode: 1290 Total reward: 199.0 Training loss: 0.3460 Explore P: 0.0100\n",
      "Episode: 1291 Total reward: 199.0 Training loss: 0.2984 Explore P: 0.0100\n",
      "Episode: 1292 Total reward: 199.0 Training loss: 0.1464 Explore P: 0.0100\n",
      "Episode: 1293 Total reward: 199.0 Training loss: 0.1969 Explore P: 0.0100\n",
      "Episode: 1294 Total reward: 199.0 Training loss: 26.0325 Explore P: 0.0100\n",
      "Episode: 1295 Total reward: 199.0 Training loss: 0.1248 Explore P: 0.0100\n",
      "Episode: 1296 Total reward: 199.0 Training loss: 0.5919 Explore P: 0.0100\n",
      "Episode: 1297 Total reward: 199.0 Training loss: 0.2599 Explore P: 0.0100\n",
      "Episode: 1298 Total reward: 199.0 Training loss: 0.1683 Explore P: 0.0100\n",
      "Episode: 1299 Total reward: 199.0 Training loss: 0.6735 Explore P: 0.0100\n",
      "Episode: 1300 Total reward: 199.0 Training loss: 0.2844 Explore P: 0.0100\n",
      "Episode: 1301 Total reward: 199.0 Training loss: 0.1354 Explore P: 0.0100\n",
      "Episode: 1302 Total reward: 199.0 Training loss: 0.6948 Explore P: 0.0100\n",
      "Episode: 1303 Total reward: 199.0 Training loss: 0.1560 Explore P: 0.0100\n",
      "Episode: 1304 Total reward: 199.0 Training loss: 0.4822 Explore P: 0.0100\n",
      "Episode: 1305 Total reward: 199.0 Training loss: 0.2038 Explore P: 0.0100\n",
      "Episode: 1306 Total reward: 199.0 Training loss: 0.8414 Explore P: 0.0100\n",
      "Episode: 1307 Total reward: 199.0 Training loss: 0.1120 Explore P: 0.0100\n",
      "Episode: 1308 Total reward: 199.0 Training loss: 0.1242 Explore P: 0.0100\n",
      "Episode: 1309 Total reward: 199.0 Training loss: 0.1350 Explore P: 0.0100\n",
      "Episode: 1310 Total reward: 199.0 Training loss: 0.2364 Explore P: 0.0100\n",
      "Episode: 1311 Total reward: 199.0 Training loss: 0.1545 Explore P: 0.0100\n",
      "Episode: 1312 Total reward: 199.0 Training loss: 0.1781 Explore P: 0.0100\n",
      "Episode: 1313 Total reward: 199.0 Training loss: 0.0796 Explore P: 0.0100\n",
      "Episode: 1314 Total reward: 199.0 Training loss: 0.1552 Explore P: 0.0100\n",
      "Episode: 1315 Total reward: 199.0 Training loss: 0.0605 Explore P: 0.0100\n",
      "Episode: 1316 Total reward: 199.0 Training loss: 0.1282 Explore P: 0.0100\n",
      "Episode: 1317 Total reward: 199.0 Training loss: 0.1806 Explore P: 0.0100\n",
      "Episode: 1318 Total reward: 199.0 Training loss: 0.1309 Explore P: 0.0100\n",
      "Episode: 1319 Total reward: 199.0 Training loss: 0.0677 Explore P: 0.0100\n",
      "Episode: 1320 Total reward: 199.0 Training loss: 0.0702 Explore P: 0.0100\n",
      "Episode: 1321 Total reward: 199.0 Training loss: 20.5490 Explore P: 0.0100\n",
      "Episode: 1322 Total reward: 199.0 Training loss: 0.1115 Explore P: 0.0100\n",
      "Episode: 1323 Total reward: 199.0 Training loss: 0.1798 Explore P: 0.0100\n",
      "Episode: 1324 Total reward: 199.0 Training loss: 0.1933 Explore P: 0.0100\n",
      "Episode: 1325 Total reward: 199.0 Training loss: 0.0727 Explore P: 0.0100\n",
      "Episode: 1326 Total reward: 199.0 Training loss: 0.1615 Explore P: 0.0100\n",
      "Episode: 1327 Total reward: 199.0 Training loss: 6.3005 Explore P: 0.0100\n",
      "Episode: 1328 Total reward: 199.0 Training loss: 0.0857 Explore P: 0.0100\n",
      "Episode: 1329 Total reward: 199.0 Training loss: 0.4392 Explore P: 0.0100\n",
      "Episode: 1330 Total reward: 199.0 Training loss: 0.1059 Explore P: 0.0100\n",
      "Episode: 1331 Total reward: 199.0 Training loss: 0.3073 Explore P: 0.0100\n",
      "Episode: 1332 Total reward: 199.0 Training loss: 26.7020 Explore P: 0.0100\n",
      "Episode: 1333 Total reward: 199.0 Training loss: 0.0552 Explore P: 0.0100\n",
      "Episode: 1334 Total reward: 199.0 Training loss: 0.0690 Explore P: 0.0100\n",
      "Episode: 1335 Total reward: 199.0 Training loss: 19.4927 Explore P: 0.0100\n",
      "Episode: 1336 Total reward: 199.0 Training loss: 0.0689 Explore P: 0.0100\n",
      "Episode: 1337 Total reward: 199.0 Training loss: 0.1593 Explore P: 0.0100\n",
      "Episode: 1338 Total reward: 199.0 Training loss: 0.0937 Explore P: 0.0100\n",
      "Episode: 1339 Total reward: 199.0 Training loss: 0.1847 Explore P: 0.0100\n",
      "Episode: 1340 Total reward: 199.0 Training loss: 0.0746 Explore P: 0.0100\n",
      "Episode: 1341 Total reward: 199.0 Training loss: 0.0641 Explore P: 0.0100\n",
      "Episode: 1342 Total reward: 199.0 Training loss: 0.0757 Explore P: 0.0100\n",
      "Episode: 1343 Total reward: 199.0 Training loss: 0.0447 Explore P: 0.0100\n",
      "Episode: 1344 Total reward: 199.0 Training loss: 0.1160 Explore P: 0.0100\n",
      "Episode: 1345 Total reward: 199.0 Training loss: 0.0675 Explore P: 0.0100\n",
      "Episode: 1346 Total reward: 199.0 Training loss: 0.0761 Explore P: 0.0100\n",
      "Episode: 1347 Total reward: 199.0 Training loss: 0.0954 Explore P: 0.0100\n",
      "Episode: 1348 Total reward: 199.0 Training loss: 0.1009 Explore P: 0.0100\n",
      "Episode: 1349 Total reward: 199.0 Training loss: 0.0719 Explore P: 0.0100\n",
      "Episode: 1350 Total reward: 199.0 Training loss: 0.2113 Explore P: 0.0100\n",
      "Episode: 1351 Total reward: 199.0 Training loss: 0.1182 Explore P: 0.0100\n",
      "Episode: 1352 Total reward: 199.0 Training loss: 0.0983 Explore P: 0.0100\n",
      "Episode: 1353 Total reward: 199.0 Training loss: 0.0953 Explore P: 0.0100\n",
      "Episode: 1354 Total reward: 199.0 Training loss: 0.0762 Explore P: 0.0100\n",
      "Episode: 1355 Total reward: 199.0 Training loss: 3.1673 Explore P: 0.0100\n",
      "Episode: 1356 Total reward: 199.0 Training loss: 0.0578 Explore P: 0.0100\n",
      "Episode: 1357 Total reward: 199.0 Training loss: 0.0600 Explore P: 0.0100\n",
      "Episode: 1358 Total reward: 199.0 Training loss: 0.1027 Explore P: 0.0100\n",
      "Episode: 1359 Total reward: 199.0 Training loss: 0.1188 Explore P: 0.0100\n",
      "Episode: 1360 Total reward: 199.0 Training loss: 0.1805 Explore P: 0.0100\n",
      "Episode: 1361 Total reward: 199.0 Training loss: 0.0498 Explore P: 0.0100\n",
      "Episode: 1362 Total reward: 199.0 Training loss: 18.4316 Explore P: 0.0100\n",
      "Episode: 1363 Total reward: 199.0 Training loss: 0.0847 Explore P: 0.0100\n",
      "Episode: 1364 Total reward: 199.0 Training loss: 0.0574 Explore P: 0.0100\n",
      "Episode: 1365 Total reward: 199.0 Training loss: 0.0613 Explore P: 0.0100\n",
      "Episode: 1366 Total reward: 199.0 Training loss: 0.1975 Explore P: 0.0100\n",
      "Episode: 1367 Total reward: 199.0 Training loss: 0.2039 Explore P: 0.0100\n",
      "Episode: 1368 Total reward: 199.0 Training loss: 0.1261 Explore P: 0.0100\n",
      "Episode: 1369 Total reward: 199.0 Training loss: 0.0678 Explore P: 0.0100\n",
      "Episode: 1370 Total reward: 199.0 Training loss: 0.0463 Explore P: 0.0100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1371 Total reward: 199.0 Training loss: 0.0929 Explore P: 0.0100\n",
      "Episode: 1372 Total reward: 199.0 Training loss: 101.8636 Explore P: 0.0100\n",
      "Episode: 1373 Total reward: 199.0 Training loss: 0.1554 Explore P: 0.0100\n",
      "Episode: 1374 Total reward: 199.0 Training loss: 0.0809 Explore P: 0.0100\n",
      "Episode: 1375 Total reward: 199.0 Training loss: 0.0831 Explore P: 0.0100\n",
      "Episode: 1376 Total reward: 199.0 Training loss: 0.1179 Explore P: 0.0100\n",
      "Episode: 1377 Total reward: 199.0 Training loss: 0.1106 Explore P: 0.0100\n",
      "Episode: 1378 Total reward: 199.0 Training loss: 0.1849 Explore P: 0.0100\n",
      "Episode: 1379 Total reward: 199.0 Training loss: 0.0290 Explore P: 0.0100\n",
      "Episode: 1380 Total reward: 199.0 Training loss: 0.0200 Explore P: 0.0100\n",
      "Episode: 1381 Total reward: 199.0 Training loss: 0.0934 Explore P: 0.0100\n",
      "Episode: 1382 Total reward: 199.0 Training loss: 0.0805 Explore P: 0.0100\n",
      "Episode: 1383 Total reward: 199.0 Training loss: 0.1906 Explore P: 0.0100\n",
      "Episode: 1384 Total reward: 199.0 Training loss: 0.1618 Explore P: 0.0100\n",
      "Episode: 1385 Total reward: 199.0 Training loss: 0.0750 Explore P: 0.0100\n",
      "Episode: 1386 Total reward: 199.0 Training loss: 0.0418 Explore P: 0.0100\n",
      "Episode: 1387 Total reward: 199.0 Training loss: 0.1164 Explore P: 0.0100\n",
      "Episode: 1388 Total reward: 199.0 Training loss: 0.1069 Explore P: 0.0100\n",
      "Episode: 1389 Total reward: 199.0 Training loss: 0.0645 Explore P: 0.0100\n",
      "Episode: 1390 Total reward: 199.0 Training loss: 0.1573 Explore P: 0.0100\n",
      "Episode: 1391 Total reward: 199.0 Training loss: 0.0708 Explore P: 0.0100\n",
      "Episode: 1392 Total reward: 199.0 Training loss: 0.1178 Explore P: 0.0100\n",
      "Episode: 1393 Total reward: 199.0 Training loss: 0.0449 Explore P: 0.0100\n",
      "Episode: 1394 Total reward: 199.0 Training loss: 0.2028 Explore P: 0.0100\n",
      "Episode: 1395 Total reward: 199.0 Training loss: 0.2579 Explore P: 0.0100\n",
      "Episode: 1396 Total reward: 199.0 Training loss: 0.3200 Explore P: 0.0100\n",
      "Episode: 1397 Total reward: 199.0 Training loss: 0.1063 Explore P: 0.0100\n",
      "Episode: 1398 Total reward: 199.0 Training loss: 25.0875 Explore P: 0.0100\n",
      "Episode: 1399 Total reward: 199.0 Training loss: 0.0633 Explore P: 0.0100\n",
      "Episode: 1400 Total reward: 199.0 Training loss: 0.0626 Explore P: 0.0100\n",
      "Episode: 1401 Total reward: 199.0 Training loss: 0.0842 Explore P: 0.0100\n",
      "Episode: 1402 Total reward: 199.0 Training loss: 92.8661 Explore P: 0.0100\n",
      "Episode: 1403 Total reward: 199.0 Training loss: 0.0439 Explore P: 0.0100\n",
      "Episode: 1404 Total reward: 199.0 Training loss: 0.1352 Explore P: 0.0100\n",
      "Episode: 1405 Total reward: 199.0 Training loss: 0.1377 Explore P: 0.0100\n",
      "Episode: 1406 Total reward: 199.0 Training loss: 0.0307 Explore P: 0.0100\n",
      "Episode: 1407 Total reward: 199.0 Training loss: 0.0404 Explore P: 0.0100\n",
      "Episode: 1408 Total reward: 199.0 Training loss: 170.2841 Explore P: 0.0100\n",
      "Episode: 1409 Total reward: 199.0 Training loss: 0.0573 Explore P: 0.0100\n",
      "Episode: 1410 Total reward: 199.0 Training loss: 0.0965 Explore P: 0.0100\n",
      "Episode: 1411 Total reward: 199.0 Training loss: 0.0923 Explore P: 0.0100\n",
      "Episode: 1412 Total reward: 199.0 Training loss: 0.1831 Explore P: 0.0100\n",
      "Episode: 1413 Total reward: 199.0 Training loss: 0.1320 Explore P: 0.0100\n",
      "Episode: 1414 Total reward: 199.0 Training loss: 0.1236 Explore P: 0.0100\n",
      "Episode: 1415 Total reward: 199.0 Training loss: 0.0360 Explore P: 0.0100\n",
      "Episode: 1416 Total reward: 199.0 Training loss: 0.1151 Explore P: 0.0100\n",
      "Episode: 1417 Total reward: 199.0 Training loss: 0.0886 Explore P: 0.0100\n",
      "Episode: 1418 Total reward: 199.0 Training loss: 0.2264 Explore P: 0.0100\n",
      "Episode: 1419 Total reward: 199.0 Training loss: 0.2293 Explore P: 0.0100\n",
      "Episode: 1420 Total reward: 199.0 Training loss: 51.9902 Explore P: 0.0100\n",
      "Episode: 1421 Total reward: 199.0 Training loss: 0.0818 Explore P: 0.0100\n",
      "Episode: 1422 Total reward: 199.0 Training loss: 0.3307 Explore P: 0.0100\n",
      "Episode: 1423 Total reward: 199.0 Training loss: 0.0789 Explore P: 0.0100\n",
      "Episode: 1424 Total reward: 199.0 Training loss: 0.3808 Explore P: 0.0100\n",
      "Episode: 1425 Total reward: 199.0 Training loss: 0.0354 Explore P: 0.0100\n",
      "Episode: 1426 Total reward: 199.0 Training loss: 95.5976 Explore P: 0.0100\n",
      "Episode: 1427 Total reward: 199.0 Training loss: 0.4769 Explore P: 0.0100\n",
      "Episode: 1428 Total reward: 199.0 Training loss: 0.1257 Explore P: 0.0100\n",
      "Episode: 1429 Total reward: 199.0 Training loss: 0.2349 Explore P: 0.0100\n",
      "Episode: 1430 Total reward: 199.0 Training loss: 145.2547 Explore P: 0.0100\n",
      "Episode: 1431 Total reward: 199.0 Training loss: 0.0655 Explore P: 0.0100\n",
      "Episode: 1432 Total reward: 199.0 Training loss: 0.1222 Explore P: 0.0100\n",
      "Episode: 1433 Total reward: 199.0 Training loss: 0.1533 Explore P: 0.0100\n",
      "Episode: 1434 Total reward: 199.0 Training loss: 0.3271 Explore P: 0.0100\n",
      "Episode: 1435 Total reward: 199.0 Training loss: 0.1519 Explore P: 0.0100\n",
      "Episode: 1436 Total reward: 199.0 Training loss: 0.1112 Explore P: 0.0100\n",
      "Episode: 1437 Total reward: 199.0 Training loss: 0.0851 Explore P: 0.0100\n",
      "Episode: 1438 Total reward: 199.0 Training loss: 0.2266 Explore P: 0.0100\n",
      "Episode: 1439 Total reward: 199.0 Training loss: 0.1195 Explore P: 0.0100\n",
      "Episode: 1440 Total reward: 199.0 Training loss: 0.0741 Explore P: 0.0100\n",
      "Episode: 1441 Total reward: 199.0 Training loss: 107.2566 Explore P: 0.0100\n",
      "Episode: 1442 Total reward: 199.0 Training loss: 0.1808 Explore P: 0.0100\n",
      "Episode: 1443 Total reward: 199.0 Training loss: 108.1455 Explore P: 0.0100\n",
      "Episode: 1444 Total reward: 199.0 Training loss: 0.0924 Explore P: 0.0100\n",
      "Episode: 1445 Total reward: 199.0 Training loss: 0.1158 Explore P: 0.0100\n",
      "Episode: 1446 Total reward: 199.0 Training loss: 0.0527 Explore P: 0.0100\n",
      "Episode: 1447 Total reward: 199.0 Training loss: 0.1659 Explore P: 0.0100\n",
      "Episode: 1448 Total reward: 199.0 Training loss: 0.1607 Explore P: 0.0100\n",
      "Episode: 1449 Total reward: 199.0 Training loss: 0.0843 Explore P: 0.0100\n",
      "Episode: 1450 Total reward: 199.0 Training loss: 0.3612 Explore P: 0.0100\n",
      "Episode: 1451 Total reward: 199.0 Training loss: 0.1399 Explore P: 0.0100\n",
      "Episode: 1452 Total reward: 199.0 Training loss: 0.6389 Explore P: 0.0100\n",
      "Episode: 1453 Total reward: 199.0 Training loss: 0.0981 Explore P: 0.0100\n",
      "Episode: 1454 Total reward: 199.0 Training loss: 0.1380 Explore P: 0.0100\n",
      "Episode: 1455 Total reward: 199.0 Training loss: 110.8782 Explore P: 0.0100\n",
      "Episode: 1456 Total reward: 199.0 Training loss: 0.1982 Explore P: 0.0100\n",
      "Episode: 1457 Total reward: 199.0 Training loss: 0.2638 Explore P: 0.0100\n",
      "Episode: 1458 Total reward: 199.0 Training loss: 317.2325 Explore P: 0.0100\n",
      "Episode: 1459 Total reward: 199.0 Training loss: 0.1143 Explore P: 0.0100\n",
      "Episode: 1460 Total reward: 199.0 Training loss: 0.1136 Explore P: 0.0100\n",
      "Episode: 1461 Total reward: 199.0 Training loss: 0.2095 Explore P: 0.0100\n",
      "Episode: 1462 Total reward: 199.0 Training loss: 0.2120 Explore P: 0.0100\n",
      "Episode: 1463 Total reward: 199.0 Training loss: 0.2077 Explore P: 0.0100\n",
      "Episode: 1464 Total reward: 199.0 Training loss: 0.0520 Explore P: 0.0100\n",
      "Episode: 1465 Total reward: 199.0 Training loss: 0.1278 Explore P: 0.0100\n",
      "Episode: 1466 Total reward: 199.0 Training loss: 0.0410 Explore P: 0.0100\n",
      "Episode: 1467 Total reward: 199.0 Training loss: 0.1534 Explore P: 0.0100\n",
      "Episode: 1468 Total reward: 199.0 Training loss: 0.0782 Explore P: 0.0100\n",
      "Episode: 1469 Total reward: 199.0 Training loss: 0.0497 Explore P: 0.0100\n",
      "Episode: 1470 Total reward: 199.0 Training loss: 333.4407 Explore P: 0.0100\n",
      "Episode: 1471 Total reward: 199.0 Training loss: 0.1217 Explore P: 0.0100\n",
      "Episode: 1472 Total reward: 199.0 Training loss: 0.2756 Explore P: 0.0100\n",
      "Episode: 1473 Total reward: 199.0 Training loss: 0.2156 Explore P: 0.0100\n",
      "Episode: 1474 Total reward: 199.0 Training loss: 0.1058 Explore P: 0.0100\n",
      "Episode: 1475 Total reward: 199.0 Training loss: 66.3934 Explore P: 0.0100\n",
      "Episode: 1476 Total reward: 199.0 Training loss: 0.1389 Explore P: 0.0100\n",
      "Episode: 1477 Total reward: 199.0 Training loss: 0.0601 Explore P: 0.0100\n",
      "Episode: 1478 Total reward: 199.0 Training loss: 0.0934 Explore P: 0.0100\n",
      "Episode: 1479 Total reward: 199.0 Training loss: 0.2149 Explore P: 0.0100\n",
      "Episode: 1480 Total reward: 199.0 Training loss: 21.1518 Explore P: 0.0100\n",
      "Episode: 1481 Total reward: 199.0 Training loss: 0.3470 Explore P: 0.0100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1482 Total reward: 199.0 Training loss: 0.1756 Explore P: 0.0100\n",
      "Episode: 1483 Total reward: 199.0 Training loss: 0.0995 Explore P: 0.0100\n",
      "Episode: 1484 Total reward: 199.0 Training loss: 0.0378 Explore P: 0.0100\n",
      "Episode: 1485 Total reward: 199.0 Training loss: 0.1935 Explore P: 0.0100\n",
      "Episode: 1486 Total reward: 199.0 Training loss: 0.0845 Explore P: 0.0100\n",
      "Episode: 1487 Total reward: 199.0 Training loss: 0.1346 Explore P: 0.0100\n",
      "Episode: 1488 Total reward: 199.0 Training loss: 90.2376 Explore P: 0.0100\n",
      "Episode: 1489 Total reward: 199.0 Training loss: 0.0648 Explore P: 0.0100\n",
      "Episode: 1490 Total reward: 199.0 Training loss: 0.1960 Explore P: 0.0100\n",
      "Episode: 1491 Total reward: 199.0 Training loss: 0.2953 Explore P: 0.0100\n",
      "Episode: 1492 Total reward: 199.0 Training loss: 0.1383 Explore P: 0.0100\n",
      "Episode: 1493 Total reward: 199.0 Training loss: 0.1005 Explore P: 0.0100\n",
      "Episode: 1494 Total reward: 199.0 Training loss: 0.1874 Explore P: 0.0100\n",
      "Episode: 1495 Total reward: 199.0 Training loss: 0.2152 Explore P: 0.0100\n",
      "Episode: 1496 Total reward: 199.0 Training loss: 0.0448 Explore P: 0.0100\n",
      "Episode: 1497 Total reward: 199.0 Training loss: 0.0821 Explore P: 0.0100\n",
      "Episode: 1498 Total reward: 199.0 Training loss: 28.9242 Explore P: 0.0100\n",
      "Episode: 1499 Total reward: 199.0 Training loss: 0.2589 Explore P: 0.0100\n"
     ]
    }
   ],
   "source": [
    "saver = tf.train.Saver()\n",
    "rewards_list = []\n",
    "loss = 0\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    total_steps = 0\n",
    "    for episode in range(1, train_episodes):\n",
    "        total_reward = 0\n",
    "        current_step = 0\n",
    "        while current_step < max_steps:\n",
    "            total_steps += 1\n",
    "            \n",
    "            # Determine whether to explore or exploit\n",
    "            explore_p = explore_stop + (explore_start - explore_stop)*np.exp(-decay_rate*total_steps)\n",
    "            if explore_p > np.random.rand():\n",
    "                # Explore a random action\n",
    "                action = env.action_space.sample()\n",
    "            else:\n",
    "                # Exploit best action\n",
    "                feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "                Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "                action = np.argmax(Qs)\n",
    "                \n",
    "            # Execute action to get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            total_reward += reward\n",
    "            \n",
    "            if done:\n",
    "                # the episode ends so no next state\n",
    "                next_state = np.zeros(state.shape)\n",
    "                current_step = max_steps\n",
    "                \n",
    "                print('Episode: {}'.format(episode),\n",
    "                      'Total reward: {}'.format(total_reward),\n",
    "                      'Training loss: {:.4f}'.format(loss),\n",
    "                      'Explore P: {:.4f}'.format(explore_p))\n",
    "                rewards_list.append((episode, total_reward))\n",
    "                \n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                \n",
    "                # Start new episode\n",
    "                env.reset()\n",
    "                # Take a random step to get the cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "                \n",
    "            else:\n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                state = next_state\n",
    "                current_step += 1\n",
    "                \n",
    "            # Everything above is just to fill out the memory and let the agent play the game, this is where we actually train\n",
    "            batch = memory.sample(batch_size)\n",
    "            states = np.array([each[0] for each in batch])\n",
    "            actions = np.array([each[1] for each in batch])\n",
    "            rewards = np.array([each[2] for each in batch])\n",
    "            next_states = np.array([each[3] for each in batch])\n",
    "            \n",
    "            target_Qs = sess.run(mainQN.output, feed_dict={mainQN.inputs_: next_states})\n",
    "            # Set target_Qs to 0 for states where episode ends\n",
    "            episode_ends = (next_states == np.zeros(states[0].shape)).all(axis=1)\n",
    "            target_Qs[episode_ends] = (0, 0)\n",
    "            \n",
    "            targets = rewards + gamma * np.max(target_Qs, axis=1)\n",
    "            \n",
    "            loss, _ = sess.run([mainQN.loss, mainQN.optimizer],\n",
    "                              feed_dict={mainQN.inputs_: states,\n",
    "                                        mainQN.targetQs_: targets,\n",
    "                                        mainQN.actions_: actions})\n",
    "            \n",
    "    saver.save(sess, \"checkpoints/cartpole.ckpt\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the training progress\n",
    "Gray line is the total reward for each episode and blue is the running mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def running_mean(x, N):\n",
    "    cumsum = np.cumsum(np.insert(x, 0, 0)) \n",
    "    return (cumsum[N:] - cumsum[:-N]) / N "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1240aab70>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXmUJGd14Pu7kVvtXV29q/cWUktiE1IjhDE2tsyODdiY\n5fgZxssIPPiNGWzPsMyM7fEyZmxgjt/Y2LLBCIbBgMGGYTHIMjaWzdZCC0KiW91Sd6sX9d5dXVtW\nZsR9f0REZlRWZFZWZURmVub9nVOnMiMiI29FRXz3u8t3r6gqhmEYhlGL02kBDMMwjO7EFIRhGIYR\niykIwzAMIxZTEIZhGEYspiAMwzCMWExBGIZhGLGYgjAMwzBiMQVhGIZhxGIKwjAMw4gl22kBWmH9\n+vW6a9euTothGIaxqrj33nvPqeqGpY5b1Qpi165d7N+/v9NiGIZhrCpE5Ggzx5mLyTAMw4jFFIRh\nGIYRiykIwzAMIxZTEIZhGEYsqSkIEdkuIl8VkUdE5Hsi8ivB9gkRuUtEHg1+rw22i4j8kYgcEpEH\nReSmtGQzDMMwliZNC6IM/KqqXg/cCrxVRG4A3gHcrarXAHcH7wFeClwT/NwOfCBF2QzDMIwlSE1B\nqOopVf1O8PoK8AiwFXglcGdw2J3Aq4LXrwQ+oj7fAMZFZEta8hmGYRiNacs6CBHZBTwL+CawSVVP\nga9ERGRjcNhW4InIx44H207VnOt2fAuDHTt2pCp3P3PlyhUymQznzp2jWCyydu1axsfHyWart4yq\ncvLkSfL5PJ978BQXp4qpyTMoZW67doJ16yZQVaanp3Ech4GBARzHYcOGDczNzTE1NcX09DSqiuM4\n5HI5AHK5HPPz8+Tzeb7yvSc5eWk2Ufkyorz8GVdxcabMPz96NtFzA4wM5vnF255OuTTP1NQU33r8\nPA+fnEz8e5aDqvKMLcNs37iGL333SZJoXyyO8JrnXsvO9WOx+69cuUKpVGJiYqLheS5dusTUbJHP\n3HeC+ZLbslzdyJ7N4/zkc65J9TtSVxAiMgJ8Gnibqk6KSN1DY7YtuuNU9Q7gDoB9+/ZZQ+0UKJVK\nnDx5csG28+fPIyKsW7eusu3cuXNMTU1x37GLfPirh/2Ndf+9LRD8l6/bOEDt7TM3NwfAmjVrOHp0\n8dqf2dmFiqDsKh/48gP+m6RkDeQblnkOPHmFbz1+IdnrEJz/udduZUN+nitXrvCXd3+XJyeL6Vzv\nZcj13fXDPG3rGv7vAyeTkUXB85Rff+WzY3eH9+X4+DiOE+8AKZVKnD59mgePX+JT9xzyN3byOqXE\nzU+5anUrCBHJ4SuHj6nqZ4LNp0VkS2A9bAHOBNuPA9sjH98GLByljLZQOxO8PFvi/Xcd5Gde8FRe\nEVEQruvPzI5enOMJb5zv/JcXMjGcT1yeL33ncX7vr/+Vklt/PtDs7DU3MMgT3jjvftn1/Nsf2pOI\nfGcvz/CT7/kbXE+ZHdzAwPoh7nr7DydyboCvPHCU3/7EPcyWymhOKRQKnGSCW29cx3tf+8zEvme5\nvP2OL/LEhRlm8mu5nJ/nwd98ccvnfOlv/RWXZ0sJSAf50XU84Z3jC//+B3nqVWsSOWe/kWYWkwAf\nBB5R1fdFdn0OeFPw+k3AZyPb3xhkM90KXA5dUUZnOXV5juMXZ7nzX4/E7i+5ykDOSUU5AGQy/m3q\nJeDCKLseALlMclPK0KrxPMX1FKe+lbwiCln/fMWIq6Tkeon+DSthMJ9lpuRyYabE6EAukXMOF7JM\nFcuJnCucNCT9/+gn0rQgngf8LPBdEbk/2PYu4PeBT4rILwDHgJ8O9n0ReBlwCJgBfi5F2YxlMF/2\nB6bJmpld6C4sux75THoJcbmM/7uRBdEsJc8/Rz6baflcIY7jXwcPxfWq75OiEMhaLLuA/7rsKdkO\nK4iRQobJ2TJf/O4prtu6NpFzFnIZZuZbixmEisELfHOZhP8f/URqCkJV76G+5++2mOMVeGta8hgr\nZ77sP2iuFz9Al1yPQi65AbeWXGhB1Pn+5VB2QwWRnELLBIrS83wrJ2ldGcpaLFf//pLrka3jg28X\nt123ibGBHLnxTdzylE2JnNMRSeT/DBAYi2ZBtMCqruZqtIe5wIKo5+EpuUo+k56CCAfCegpqOZSC\nuEmSCiIbcYF5qhWFkRQDgfKdL3uVbWVXO+5iWjOU48du2MTu3bvJ55NxL4pIIq5EqLokzYJYOVZq\nw1iSz91/AoB6z5lvQaR3K2WDLy573hJHLk3opkrSJRbqA8VXYg0y9VZEKOtcqeqbL3texbLqJRxp\nPuFgKcLbJWmF3U/03h1mJE44QNWbdfsWRIoxiGxoQbR+rlJwkkIqLqbAgkh4xhpaEKGLSVUpuVqx\nXDpNkgrREUnEUoSqS7LDnrhVjV06Y0nC2PBsTfAwXIQ27+RTnc2mYkEkqSDCILX6FkTSM9Z8TRZT\n+P/I9aDrRKS+K3O5uOZiahlTEMaSuMHAXKxZkRquqp6TgVQfwlD5HDoz1fK5wjTXJBWEiEDgGvE0\n+RlrJYspkD2cGXeLBZEkToIxCMXSXFvFgtTGkoSundmSi+piH7vvVklvsNo4Wgi+Z+XnuDhTYq7k\nUnJ92ZN2iWWCgc3ztOISS4psxiHjSMSCCBRED86MHUfwEkhnhoiLyRTEijEFYSxJOCCpKsWyV/GJ\nV/Z7yfvdo2QzDuuG85XZ/0r4jc8+xMy8yy/92FMBEneJCYGLSZNfKAe+vMXyQguiW1wnScYghOTT\nXLvlOq1Ges9GNVqm9oF3Pa2saIlbxORq+pki2YxQbmHgCOWenvcX+yXpYgIQx89i8jSdGWsuE2NB\ndDjNNQ0cJ8k0V/+3ZTGtHFMQxpJ4njKU962GmfnFZRC8lC0I8NdClBNwPYSKIsksJgAnmPm6KWQx\ngT8Ldj314xw9vADMj0Ekc65Q0VgW08qxS2csibtAQSy2IMqarhkvImQyUgmWR5kuljlyfrrpc917\n9BKQvAURznw9L52BOyNCOSwh0cMxCCG5dRCWxdQ6piCMJSmrMpT3w1VRBVGpeeN5qT+EOUcqdZSi\n3PHPj/E7n39kUZ0ogKPnZ/jQvzwe1DDyefjkJCOFLGMJFZcLcSpZTMmX2gBfAbmBU71SQqIHBz7H\nSTIGYUHqVjEFYSwiOoNTVa54BQbzOQSYiam06aY0a46ScaSyyC3K9074TXOmYlxfn73/BP966DzH\nLlR7QuxcN8Q333Ubg/lkS4NI4BpJo5orQEaoxGDC9M1usSASDVKLVGb+KyW8f7XLgvmrEVMQRkPK\nnuIhDBYWWxAhnqepD1bZjMOjp+uvgyiVFw8q4UAzNVe1LgZyDsOF5JP3Qt+5q5r4zF5EfAvCW1g0\nsRcHvkyCC+VCQ8QsiJVjCsJoSGjuD2TDMtOLZ/HlNriYQhpVlK1lMEjHnZqrWhdpDRahi0lTyuhy\nIkH6SvC1Bwe+JIv1uZWFcomcri8xBWE0JEwcymcdBGU+JpPIXz2c7lN44/ZxwF+sFxJ1hUUrnYaE\nVWDv/PriVqRJ44hvPaS1JiQj1RXtPR2kdpIp6w4ECQPJusD6DVMQRkPCQSns91CKGYjdNriYQmvg\n8kzVXRRdFzEfY0HESTQdE6tIAhFB04xBONV1IL0cpM7gJFruuxfdcO0kzZajHxKRMyLyUGTbJ0Tk\n/uDnSNhpTkR2ichsZN+fpiWXsTzCwShcNxDnyklrUIwyMuDHDe7+/mkOBrGI6EwzTq44DXHL7olU\n5HNEKllMaYxJmUgMotssiESruSa5UK4N92Wvk2apjQ8D/wv4SLhBVV8XvhaR9wKXI8cfVtUbU5TH\nWAFhoDds0VlPQaRdN27vplEAvnbwHF87eI4/f+PNRL1d82Vv0d1cOzS8+qat/PRzdqYinyO+qy2t\nWasj1fhL+Hf3ogWRdDVXsyBaI7XHWlW/BlyI2yf+lOO1wMfT+n4jGSpB6pyDQGwMwlVSLdYHLOqe\n9vePnFkw04wGoivUjA1jhWxqM0oRXzmk0ZMa/OsbJgiE6ZvdYkEkiSMk62IyC6IlOhWDeD5wWlUf\njWzbLSL3icg/icjzOySXUUPo9y5kFlsQYZC4HRZErRvj6Plp3IiyOnl5tvYjUDPOZFIU0u+lnM6g\nJCI1FkSQ5tqDg1+i5b612u3PWBmdUhBvYKH1cArYoarPAt4O/B8RGYv7oIjcLiL7RWT/2bNn2yBq\n/xHNDgotCL+lqMYHqds0U/t3L7i68tpxFi6oujizeCV1bTJMmkrMEUHRIB6T/PkzjlTTXCud0rpj\n9Es6BqFNZDE1U44j7SrD/UDbFYSIZIGfBD4RblPVoqqeD17fCxwGro37vKreoar7VHXfhg0b2iFy\nXxNaEJUYRMzD6xfrS/9Wumnn2srrTM1M88pcnIJYKGtG0pMxzN/3UlgoB77slSB1sK0XXUyJroOw\nGETLdMKC+DHg+6p6PNwgIhtEJBO83gNcAzzWAdn6ntqZWSVjJiPkMk5skLqcsospbobqryyuvp8q\nxpUhr1EQqVoQQU/qFFqOgl+R1FUPDdZa+Nt6b/BL0sWUVuHEfiLNNNePA18H9orIcRH5hWDX61kc\nnP4h4EEReQD4a+Atqhob4DbSo1wuc/DgQS5dulTZVi2Z7JB16qyDSGnWHMcv/+hTAJgveQuqu07H\n1IiqXXCVTdHKcUTwSG/WGl0HESrxXrQgkij3XSkiaRZEy6SW5qqqb6iz/d/EbPs08Om0ZDGao1Ty\n3TSTk5OVbaHfOydCto4F0Y5aTCE3bh/nqvEB5l1vwUwzrpnQIhdTijI6AuoFs9ZUXExSCcpXFsp1\nyew42RhEcllMaXX36ydsJbXRkPBhzWSEbEZi01zLKblV6pHPOhTL7gIXU1yviNq6TWkpCBHxfee0\neaFcL3aUCxYcJkE7Gln1OtaT2mhI2au6mHLOYgtCtf3+8HzGYb7sceJSNbU1rtucp7BnwzDrRwo8\ndm6KjWMDqckkQQwirYwux5HKOohKNdcenB0LfskSVW3ZMnFTbmTVD5iCMBYRl+aadYScs3gldThJ\nb6c/PJ91eOjEJJeDJkFjg7nYKq+hD/r2H9qTukwZ8WMEmkLhQhEJivXVVHPtwcEv/JtcT1u2kHwl\nk4RU/Yu5mIyGuBEFkamxIMLaQ9DewSpMqT09WWTL+AA7JgYp160R1R6ZHJFUZ/ZOJM01/FN7MUgd\n6oRWmwYBqWWU9ROmIIyGuLUxiJrGPF4H3B037RivvB7IZhb0a46i2r5ArjhRd1wKQepMNYupW/pB\n5HLJtm0FKuVRkghDmIupdczFZDQkOivOOosbBoWrD9r5IO5cN1R5nc9K3T7GrqfBCvD0EaqB8rRa\njrq1aa4dDlLv2LGD+fn5RM8ZTjTqNYZaDp5lMbWMKQijIdUYg7PAjVLZ7/kdktupIArZaj/pjOPU\n7VftqbbNRM6IUCx5gUzpnL/WxdRp90k2myWbTXYIqcQgEjAhrNRG65iLyWiIG4xGmUwdBRG6oNoa\ng6h+12Nnp2Pl8mXzs6/awUA+w/lpfzadSs9rp9qQyKN3e1I7gY8pia5y7eh02OuYBWE0JAw5ZDNC\nxoFizYOr4Wy2jQ/i6ED1ttUgUyl2oVwbg5Sve/Z2btk9wabte3hOCk2Jqtk9XmXBXC8qiPDflUTX\nUf//3/p5+hlTEMYiFqa5BhaESOxMvR2lp2vz4XMRH07G8RWXG9unwqNNBgRjAzmeuW2cvdcmX0DS\nT3Otul5CC6IXZ8fRNNeVEi21YTGI1jAXk9GQcGaezTh+oLROMb9ODlZ+M53OZjGlTajoXFd7Os01\n/H8lUW7D9dpXI6xXMQVhNGQuCLwO5jILyj2EaCWI3bkH0ZHF2VXnp4qcnix2PJCbFP7fobjaPWmu\naRDeRolkMdH5QP5qxxSE0ZBiyWUg5zA3N0sGl3K5WlZbVQOLQtruD/+55+0C/FaomRjX1+984RHA\nX5/QC4SLA13PW7C6vdcIZ/xJWBBWi6l1LAZhNGSu7DGYy+K6Lo4IogvLale6m7V5pvYDV6/j0myJ\nm3eu5Z8fPceYTi3YfyXoUd0r40MmMrMOwy29OPhJJYup9XO5lsXUMqYgjIbMl30LAoh1MVUqi7b5\nQRQRXv70LYCvBMRb3DAIoBy/edVRmVl7igYVY5Mss90thPkHyZXaaPk0fU2PGOBGWpQ9JRc8ZY4I\nWruSOuxN0MGZWlak7oAyNtAbc6DQWnC1txeAiUiwKt0WynUDaXaU+5CInBGRhyLbflNETojI/cHP\nyyL73ikih0TkgIi8OC25jOXhelrp5RyX5qqkn+a6FI4joPGN7DetWVjie7XOukMXnut5/gKwVfp3\nLEV4HyXRE8LSXFsnTQviw8BLYra/X1VvDH6+CCAiN+C3In1q8Jk/CXtUG+0jbvD0tFp22ZHFM7tK\nsb4O2vKh9RKX6toLiFSTAMqe35O6FwPUUL0HE3ExmYJomdQUhKp+DWi2r/Qrgb9S1aKqPg4cAm5J\nSzajeUIzfdOmTXViEP7vTloQmQa5873iQ804vuvF84KudT2qIBJNc7We1C3Tiefnl0XkwcAFtTbY\nthV4InLM8WCb0WE8D8rZIQYGBnDiFERoQXRyHUQY2IzJfFmtLqVaolVOPe1dC6IajG/9XFaLqXXa\nrSA+AFwN3AicAt4bbI/7L8ZOIUTkdhHZLyL7z549m46URoV5yZINqqc6LJ6le7S/mmstjSyIHtEP\nC0pQuF5vprhCxIJIqpprb16mttFWBaGqp1XVVVUP+HOqbqTjwPbIoduAk3XOcYeq7lPVfRs2JF/3\nxlhI6O8WkQV9kSHoKNeBYn21VBeRxSmI9OVqx3dE0z972XUiDZR9lGaC2FZqo3XaqiBEZEvk7auB\nMMPpc8DrRaQgIruBa4BvtVM2I55oqqAjixvzuF1Q9iET47ceLvhWz95NI50QKXEcqSpBt4dbaVZq\nMSVSrM9KbbRKakniIvJx4AXAehE5DvwG8AIRuRHffXQEeDOAqn5PRD4JPAyUgbeqao8scVrduJ5S\nyEQWytXM3LRDC+WiZAL5orPOPeuHmZwrs26k0CmxEiW0IMqu5y+U69GZcZyyXym2DqJ1UlMQqvqG\nmM0fbHD87wK/m5Y8xsooL7AgFs/s3G4IUgdfHU1zdXtsrUB4fT31g/G9GqROMs1VtXeSFDpFr2QB\nGikRzZiJXSjXBXWB4twS0fUbvYATyWJy6V0LwqkslGv9XH6spvXz9DN2+YyGlF2tNOhxHMHThQOx\n2wUupqyzOLDpetozhfrChkESLFTUHl4oV01ZTsDFpL0bq2kXpiCMhsyWXEaCekbhzeLWDMTQ6XUQ\n4ey6us0PUPbO7R2uVHdVA+XXmwOfIwIxjalWgmdZTC3TO0+QkQoz82VGC4GCiGkHGb7MddCWrxSy\ni6Tg+oNDpyRKnspaj6AWUy+5z6IkkcUUYhZE6/TQI2QkzeRsiXlXGQkURCZGQYSDcncslKtu67XB\nwb+8/iK5vkhzTSAGYVlMrWMKoo+5cOECxWKx7v6Dp/0mPDvWDfllmH3rv2L+q2rwIEtnW46GKaBR\ny6bH3AvRxYBeDwepQ72XTC0mK7XRKqYg+pizZ89y5MiRuvvPT/vK40eu2whE6gG53RWDCAfP2uB5\ndHAYGVm9C+YymQyOAxkU1/Nw3d4PUifWcrRHLa12YQrCqEs5iPoOhLWYgqd3wXqDSn/kzt1KcfV7\nageHsbGxdouVGI7jkM9myeBR9ujtfhAsdmOuFH8tTMun6WtMQRh1KQcx31ykHwQsnN1VWo52MGga\nGxvptRhEpOVor63xWECTtZgaoaqVFf7mYmoNUxBGXVzPwxGprEYNB9w4C6KTLqZ8kEFVcns3iyl0\nKfV6mmsmZk3LSuiGPiW9QA89QkbSuArZyB0SncWGhC876RPPB0IWy1UF0WslsRe0HKV3S21UGwa1\ndh7PLIhEMAVh1MV1vQWxhdiaR11gQQzk/BhJseTXdzw3VeTybKmnZtmZyGJA1+3d9M2KG7PFGESo\nIHr1OrULUxBGXco1eeTRekAhYa2mThZFKwQWxFxgQbzj098Fkgl0dgthX2qvx/tBOGFrVXMxdQV1\nq7mKyH3U6eoGoKo3pSKR0TW4XjUYGg5QYT2g6DGdHKyGh4eZmppCBObLC/0ST9+6pkNSpUMmuPZu\nLyuIhKq5anArmIupNRqV+35N8PstQAb4aPD+Z4AraQplpE+zHbniXUyRYLB21h++bt06pqenKWQz\nFRdTSD5wPfUCoYIOi/X1kvssiiRUasMjbGTVskh9TV0FoaqHAUTkB1T1eZFd94nIvwC/lbZwRmep\nnalW6+T47zXIqOmG2Wwh6ywIUofbQkKFuJr7A4QKwtXe9a1nYtyYK8FiEMnQTAxiRERuDd+IyHOA\n1bss1Wiasrcw3z4016MWhKse2Q4W6gsH/lBBRC2jfI81A8gGMYhay66XSKoWU8XFtIonBN1AMx3l\nfgH4sIgM4Mck5oCfX+pDIvIh4BXAGVV9WrDtD4AfB+aBw8DPqeolEdkFPAIcCD7+DVV9y/L+FCNp\naovC+a914UK5Dnc3qyiInO9iiqZH5rOy6LjVbEE4jvilNjyvZ9Ncw39Py0FqzIJIgobTEBHJADuD\nAf45wK2q+nRV/XYT5/4w8JKabXcBT1PVZwAHgXdG9h1W1RuDH1MOXUDdNNeaWkzdoCCG8xmm5t0F\nA0sv9YMAX0G76ruYenUldVym3EoIYxiWxdQaDZ8gVXWBtwWvz6vq+WZPrKpfAy7UbPuKqpaDt98A\nti1PXKOduB5kI3HeemmumQ4OVqGCGB3IcmW2xEe/cbSyL+qFaSYo3+34LV87r5TTpNJRLqE0V8ti\nao1mplhfFpG3icgWERkLfxL47p8HvhR5v1tE7hORfxKR59f7kIjcLiL7RWT/2bNnExDDqEfZ8yqV\nUkWk2jAo8vCWvc4W6gsH/pGBLFPzZb5+uDqHiXMvrGYX05DMRxIDess6CkmiJ3W1DD3Wk7pFmolB\nvDn4/auRbQrsWOmXisi7gTLwsWDTKWCHqp4XkZuBvxWRp6rqZO1nVfUO4A6Affv2rf5pYQKUSiXK\n5TKDg4OJntetSWGNW0ntaWf94aGCyGczzM4vTHONBijTjEG0Q+mUSiUyAl655McgetXFBPiNkRo/\n2ktZhFpJc+3N69QullQQqro9yS8UkTfhB69v0+C/rKpFoBi8vldEDgPXAvuT/O5e5bHHHgNg7969\niZ7X9TwKkYEotu9Ch2seVRSEI4tmnb0WoHQcwWOx4u4l4traroTwHjUF0RrNWBCIyHXADcBAuE1V\n/89yv0xEXgL8J+CHVXUmsn0DcEFVXRHZA1wDPLbc8xsrZ3Z2FtetzsBdTzk/Nc/ocDWjWWosCNX2\nl57eu3cvBw4cqLwfGhoCIJtd7EtolwXRLhwRPE8p93DDIJGkSm1YFlMSLKkgROQ/Ay8CrgO+DLwY\nuAdoqCBE5OPAC4D1InIc+A38rKUCcFfwoIbprD8E/DcRKQMu8BZVvRB7YqMlZmZmKBQKleY/IceO\nHVvw/jP3neDKXJnZyOrkuEVMrtfZTJF8Pk+hUIhd89Br/mdHpNLEqVdjEBAG4xMKUq/iCUE30IwF\n8TrgRuA7qvqzIrIF+LOlPqSqb4jZ/ME6x34a+HQTshgt4HkeTzzxBENDQ2zb1jiB7NHTfjWVE5dm\nK9scJ+hJXVusrwtG4nw/WBAOlMIOfj0ag4DAUmp1oZz6gVKzIFqjmSd7Nkh3LYvIKPAksCddsYw0\nCAfJYrG45LHxA66/baEF4XXFQ7hpbGDRtm6QK0kEqaxB6VUXE/iuzORcTElI1L80Y0HcJyLjwIfw\ng8aTwHdSlcpInaWyQMIBKPp8ZRzI4NVYEO1zMTXK0lo3nF+0LSpVGKsYHh5OWqy2kXGk0jWv15Rf\nlGRcTBakToJmspjCNNc/FpEvA2Oqagqix6kM+pHnS0QYk2KMi6nzD2FcDCLqThocHOTaa69d1S4m\nkWpb1VwPT40dJ7kYRC8r0nbQTJD6Q8A/A/+sqofSF8noBsIZ2EjBv0VEpKI0isW5igXidUlrz1xu\n6QFzNSsH8K9z6GLqhmueFiKtr3xXS3NNhGamIX8F7Ab+XEQOicgnROStKctlpMByHrqxwRwA73rZ\n9ZVtYeLM9IXTXLjgJ5m5ql1R7yZfk9Xzvtc9s0OSpIcjQimopNvLMYiw5lQr+MX6xBREizTjYvqK\niPw9cBNwG/BW4Gbgj1OWzUiIYrFIPr/YR98I14MNowU2RoK/1X4QHsVikWw2i9cl/SBqPS5jA7nO\nCJIijkDJ9Qe+bsgcSwMRv32t6y19bCPCivTdcG+uZppxMX0ZWAN8G9/VdKuqnkxbMCMZyuUyR44c\nYc2aNaxfv775z3neokE3XOUadQ+7XRKDEBH+7Q/tZtv4EEOFptZ/rjoyIpTKHpDpcQsiiY5ywbl6\nU4+2jWaepIPAs/BXN58GnhSR80F5DKPLCVdHz83NVbY144ufK7kUsgtbdobvwviEqgYxiO54Cp+z\ne12nRUgVcaSyir2XZ8atprmqqsUgEqIZF9P/CyAia4A34vem3ggkWxnO6Cqmi2WGa2bicXVy/BhE\nW0XrWzKR9M9cD190J4kYhJXaSIRmXExvAZ4PPBu/6upH8F1NRg8zVXTZMLpw8Vm1FHPNOogusSB6\nnehkuJeveVhzqhVsHUQyNONiWgv8CfBtVZ1PWR6jSyi5HoWa1dThZCzSUA6vh5vXdBvR2XAvX/OM\nIwvusZUQzmFMQbTGktMQVf3v+AX0Xg8gIhMisuJeEEZ3sFTKa7lm4A+zS6QmgOiqWteuNuEEqxYV\n6YrEgLRIptSG/9tcTK3RbDXX5wFX47uXBvEruf5guqIZncR147uWZSKBUvAfxF6ezXYT0X9HLw98\nDglkMakGxfoSEalvaebyvQZ4GTANoKongCRajhpdQpw1Ua7TtSzrOF1ZrK8fiLpLOtnmNW2SKbVh\nMYgkaOYuKwad3xRARIbSFcnoBsp1FsBlM0LZq65ishhE+4i68nrbxSTmYuoSmlEQnxGRPwbWiMjP\nAV8B/jIlFTv1AAAgAElEQVRdsYxO4qmidVxH2cjsTlVx1R7CdhG9zL2slDMJ9IMwCyIZmglSvwf4\nPPA54JnA76rq+5s5uYh8SETOiMhDkW0TInKXiDwa/F4bbBcR+aOg3tODInLTyv4ko1XcBoux/JLT\nC6u5pq0gMhl/iV69ct/N9LfoBaKDXS8rZZHWe1JrHywobAdNOTJV9Uuq+h9U9W3A34nI65o8/4eB\nl9Rsewdwt6peA9wdvAd4Kf5q7WuA24EPNPkdRpM0W6yvUdeyXEYWLZRLezabz+fZtWvXskqF9CKZ\nfolBNJHF1OheVtVIqQ1TEK1Q9y4TkRER+XUR+Z8i8qPBDP8twGH8FdVLoqpfA2p7S78SuDN4fSfw\nqsj2j6jPN4DxoL2pkRDnzp0D/GJ7jSiXg54DMYNQ1nEqJac9z+/r2I4010KhsOrLdbdKv8QgkuxJ\n3ee3TMs0SnP9KH7m0tfxK7j+OjAKvFZV97fwnZtU9RSAqp4SkY3B9q3AE5HjjgfbTrXwXUaEyclJ\nYLGCqJ2NTc/79ZuGCtVaTOHg7Ke5+p/3Syr3tj+8m+iXGITjtB6kDu/pbihFv5pppCCeoqpPBxCR\nPwXOATtVdTIlWeL+k4vuEhG5Hd8FxY4dtl4vDWbmywAMFxaXzM5G1kFUYxXtdXds376dJ554Ysnj\nduzYwbFjx9ogUXvopxjEEkbukngWg0iERk92KXyhqi7weELK4XToOgp+nwm2Hwe2R47bBiwqK66q\nd6jqPlXdt2HDhgTEMWqZCSyIwZgubdmMUA6K9bueokjbFyM129tiYGBg6YNWEaGLSentlqPJNQxq\nj/uzl2l0lz1TRC4EPxeBZ4SvRaQ2rrAcPge8KXj9JuCzke1vDGIdtwKXQ1eU0V5CBRAXCM04TtWC\ncKvbjPSJXuVenhknkcVUaRhkLqaWaORiWl4LshhE5OPAC4D1InIc+A3g94FPisgvAMeAnw4O/yL+\niu1DwAzwc61+v9Ecjz766IL3YTevOD93zhGmgofXYhDtZUGQuoevuSNSSVNdKVbuOxnqKojArdQS\nqvqGOrtuizlW8YPhRgKcOXOGixcvsmvXrtj9jdIEwyB0Jq7URsbBdcMYRHCcPYRtYUGpjR52MSXR\nD0JtoVwi9O5d1udcvHhxxZ8Nzfu4WWrGkYpiaLSgzkieqCevly2IRHpSV8p9ty5PP2MKwlhEuUF2\nUjayktoURHvplyymjGgC1VyDc/XwdWoHpiD6lMOHD9fd51UsiOq2cB2EX6wviEGoxSDaycKV1L15\nzUXEr+aagItJpLn+60Z96sYggsyluP+S4IcMJlKTyugopUpsId6CKHse09PTuJ4THGcPYTuIBql7\neeBLppqrWgZTAjTKYurvwjd9jFcni2loaIh8NsO865HL5fCK/oI6UxDtoV/MfYckelKD2H3ZMk1n\nMYnIBBBdebRoEZvRG5Tc+OwkESGXcSiVlbm5OaamZoHudXf02iy7XxZ9OQ4tuZhU1SyIhGim5ejL\ngffjr2w+j18f6SBwXbqiGZ1ituSSy0qsZVAIAhMlVyuZJt2cSrh9+3ay2SVv81VBeJ01tipN7+CI\nJFBqwyzbJGjGav1d/J7UB1R1O/Bi4B/TFMroHCXX4yvfO02pHD+DywcKYt71qkHqLq4sOjQ01HRp\njm6nXxasC0uX+14KRS3FNQGaueXKqnoWcEREVPUuwJr59CiHz0433J8PFmgVy15ktWqfjFwdppst\ntSTJJNSTul+uV5o0Y3tfFpFh4B7gIyJyBmjRADS6lZkg8FyPfMXF5DUsydEOesV11Cz9MuAl1ZPa\nYhCt08zU71XAHPA2fNfSCeAVKcpkJEizXeRCZkuNK6zkAndSseThqV+NqV8Grk7TLy4TJ4mWo6qW\nxZQAzSiId6qqq6olVf2gqr4PeHvaghmd4R++f6bh/kLWbyJUcr1qOmwXxyB6iX7JYsqI0KJ+sCym\nhGhGQdT2lAZ4edKCGJ3nxMVZjp6fAeD3Xv302GMqQeqyV0lFtGyR9tAvsR6/YVDr5b7tvmydRiup\n3wy8BbhWRL4T2TUKtNJy1OhSypGHcuNYYdF+13UrjWouTM8zOphFMF9vu8gGl7nFyXXXU6+a63Lc\npZ72T9ZXmjSK8n0SuBv478A7ItuvqGpjP4SxKskt4Sqam5tjyxp/reTxi7PsLYwANlNrB1dddRWP\nnpnqtBhtIYkYhLmYkqGujlXVi6p6SFV/GhgEXhj8WJ/PHqWZzJHQxfT3j5xmas7PeLIYRPrk8/mu\nXbGeNI4jtJjEFBTrMxOiVZa8giLyVnxrYkfw80kR+Xcr/UIR2Ssi90d+JkXkbSLymyJyIrL9ZSv9\nDqPKcszyaA3+sbGxJY+/8+tHgc6luTp95kPol2wxkdZKbYCfh99nt0cqNJNI/mbgFlWdAhCR3wP+\nFfiTlXyhqh4AbgzOlcFPm/0b/Baj71fVP1zJeY3WCRsB/exzdzI4OMjk5GTscU/bOsZDJ6r72j1w\nZbNZNmzYwOjoaFu/t9Ms5QLsFRxaWyin6veTMBdT6zSjYwUoRd6Xgm1JcBtwWFWPJnQ+owWCPkBs\nGCk0LHT3M8/ZueB9tgNTtYmJCXK5XNu/t5P0SxZTmM7bKJNpKcvYD1KbgmiVRllMWVUtAx8FviEi\nnw52vRq4M6Hvfz3w8cj7XxaRN+JnSf2qqq68b6axbEILwnGkoYLYMLowwymud7WRPD3chnoB4bju\nqeKscC6qqn1zvdKk0SX8FoCq/g/gdmAGmAXekoQbSETywE8Anwo2fQC4Gt/9dAp4b53P3S4i+0Vk\n/9mzZ1sVo6dY7qrpWqKlM5Yqlf2Sp22uvDZTPn1EpCOWWicIXZatxCEsiykZGsUgKldXVb8NfDvh\n734p8B1VPR18x+nKF4v8OfD5uA+p6h3AHQD79u3r9ZTwlllekLraB2IpBXHV+GDltaW5tod+yRYL\n771WSn6biykZGimIDSJSt6RGUHKjFd5AxL0kIltU9VTw9tXAQy2e31gm5SAI4cjSCmIgSHcVtOPp\nl2NjY3UD6r3EQFDmpNcJb6eWLYg+sbjSpJGCyAAjJBeQriAiQ/hrKt4c2fw/RORG/IWiR2r2dTVz\nc3NcuXKFDRu6e4nI+Pg4ly5dqrv/8qyfizA+uHTwNxdx8HZ6pjY0NNQXCiK0IHq/YZD/u5WKrmrl\nvhOhkYI4par/LY0vVdUZYF3Ntp9N47vawdGjfhJWtyuIpdYNXJydx3GEscHqbVHPRRVNuey0BRGy\nZs0aLl++3GkxjBaQiPXaSj0mK7WRDE3FIIzm8FdvdtdlW04MoljyKGSdBQ9pPXLZ6tPX6RjE6Ogo\nMzMzrF+/vucVxK17Jrh+99ZOi5EqlSB1iwqi0/dlL9BIQdzWNimMjjFXcvnOsUs8d88E82Wv0jEu\nVBD1FEw0o6bTFoTjOGzZsqWjMrSLX3z+HtatW7f0gauY8NZqKQbhdd9kbTVSV0Go6oV2CmJ0hs/e\nf5K7Hj7N6ECWeder1FpaiqiLyWZq7aXXB75w7UMrWUyKrYNIAruEfcxUscxdD/vZxacn55gv+y6m\nKPVjENXjen3AMtrLUkHqizPznLo8W/fzfqkNyFixvpaxK9hH1A729x1duFC9WK5aEEvGIGx61lb6\nSQmHrULjYhDfPXGZX//Ug7zmA//K2SvFuufwVDueXdcL2FO+yqkd9F3X5dy5c3X3g//glVyvUo0V\n/IyE+RgFsVQWkz2CRtJUXEwx996x89MAXJ4t838fOFn3HKr908M7TZqp5mp0MWGKbcjp06e5cuVK\nw8+8764DHHhyYfMZT30FMZxvrgCeWRCdo9WSKt1OeGvFWRDRTcVy/SCFh1psLAHsKU+QNB7cYrFY\nN3XT8zyKxaqZrapNyVCrHMBfRT3vuouC1HWzmLq07MPExESnRTASIi7LNbqt0UI69WyhXBKYguhy\njhw5wpNPPpn695Q9L9bFVI9uffi6fbGisTThzD9OAUQnLI3WSbi2DiIRTEH0OEtZFNds8vtKl1xv\nQZC62c8bRtI0WigXDT43siCsmmsymILoc/btXEsh61ByNUhz9QvC5XI58vk8mzZt6rCERr8Rjuv1\nYhCVNNglGgpZFlPrWJC6z8llHHJZhzOTc5RcrQSfRYTdu3cv+XnBLIx20+tWnVPJoFu8zwuK8GUc\nabjS2lPrU5IEZkEkyGp4cGtlzGYcco7wwHE/EF67UM4w2k2jct9++qrgiFQaXMVh/SCSwSyIHmdm\nZqbyWtV3I0XJOL6SCFmOgviBp6xr+JAaydFXC+WWiEGI+PWaGmYxqdo6iAQwBZEAItIR66GZ76zt\n/zBXoyDWDhUW1FXKLUNB/Pzzdi9ZQtxIntVgqbZCvSwmVQ1cTJCRxjEIv2GQaYhWsae7z6i1IK7d\nNMLJS3OV9/llLoDrp5mt0R7COzBOAXhRF9MSMYhuTcVeTZgFscqYmZkhm82Sz+dX9Pm5klt5vXZo\n8arpXJcugFsOIyMjPTfL7idLreJiqrMOQkRwHKlrQfjF+iyLKQk6piBE5AhwBXCBsqruE5EJ4BPA\nLvy2o69V1Yv1ztGPPPHEEwDs3bt3RZ+fLfkWxC27J3j1s/zGM0OFDDNFX3GowlVXXZWApJ1j69be\na6gTVRC9pvxqqaaxLt6n6ruXMktZEFZqIxE6PS35EVW9UVX3Be/fAdytqtcAdwfvVw2dfnCb+f6p\nOb/v9EuetpkNowUAXnh9da2Dp53/O4z+xbcO/NdxCsANg9QisaU4QtQzF1MSdFpB1PJK4M7g9Z3A\nqzooS1eR1KB9pVgGYHSgajyOFCKvBzKJfI9hrBRpUM1VPV+JZJwlgtRgDYMSoJOXUIGviMi9InJ7\nsG2Tqp4CCH5vrP2QiNwuIvtFZP/Zs2fbKG7nmZpaWGQvVBof+fpRfvHO/czMl+sGjcOift8+4jcK\nHBuoxh9uvXodN1w1xmv3beO6zWNpiG4kSK9beOHMPzZIjUbWQTTOYjILonU6GaR+nqqeFJGNwF0i\n8v1mPqSqdwB3AOzbt6+3n5QI8/PznDhxInbf1w76ivJ/f+MYr9w7XPccqsr3T/mlwKP+2cFchre/\n8NoVyWVZTEbSOA0aBnmekhG/Z0S9GISqgmUxJULHLAhVPRn8PgP8DXALcFpEtgAEv8+kKcPs7Cyu\n6zI9Pd31szKvQYPetcO+NTAx1Fxm075daxORyegMva6UK2muMc+kh99xrlEWU6hYLEjdOh1RECIy\nLCKj4WvgRcBDwOeANwWHvQn4bJpyHDt2jEOHDnH8+PEFXdi6hXK53NRxm8YGAJiNpLDGEa6B2D4x\n1JpgRkdZs2ZNp0VIlWq11sX71PO7GGac+kHqULGYgmidTlkQm4B7ROQB4FvAF1T174DfB14oIo8C\nLwzet4VSqdTyOUIrZHZ2tuGMv1lqu8VFKRaLFQWSDR6ELz50quH5QgVRyDhks/W9iwMDAw3Pk8s1\n13XOSIel/j+rnYbVXCsxiPgsJ6imx5qLqXU6EoNQ1ceAZ8ZsPw/c1n6J6jM/P4/jNB5Qo7iuy7Fj\nxxgeHmbbtm0tfXcjC+LIkSOV1+GD8O0jF7g0s5V6eUjzwZOTbZDecc011yy5KKvXXRxGZ6kEqWOz\nmKpB6rouJkILIj0Z+4W+vYTNxhwef/xxDh8+vOzzRluBpk05eFAEmJytbwnNBy6obKa12lFRBWHK\nwkiaRg2DXFUcx3cf1ctiChWHI307vCWGXcEeIDqTmpqdq3tcKTgu10BBNDPg91PZB6P9VFZS1+lJ\nLQQWRD0Xk8UgEsOe9B6g7GnFb3vqwlTd42ZmfOWRc5zELAjDSJrw9op1IQV9HjINVlJXLQi7T1vF\nFESEYrHY0sDZ7Gc9z2N+fn7F31OLG/plUe742mML9v3vbxzlW49fwPWUUxcmAb+LXCuZMKOjo5XX\npiyMpKm4mOLSXINy344T74KKfs6K9bVO31ZzrR3M5+fnOXLkCBMTE6xbt45yubziiqlLcfz4cWZn\nZ5dVcK9RwLrsKVdvGGb+9BQ3bPFXQt9z6Byf2v8E00WXfzxwltuu38jdj/jLSrIZYePGjYt6RUBz\nA/74+DiFQoFjx46ZgjASx2mQxeSqIsEx9VxMGmQxWcvR1ulbBVFLOABPTk5SLBaZnp7m2mtXtrp4\nKWZnZ5f9mZMnT9bd53oeI4UCawoOm8b8Anwf/pcjC44JlQP4WUytDuymGNpPNpttem3MakYaZTEp\nZDKNS224allMSWEKIsB1/QyfcrnctofQdV0ymcVJqaq6rHUZnqdkM0LGcSoZTY1IoudDaIGZomgf\ne/bs6foV/0nQsBaTKo44/krqukHqhecxVo7p2A4SXcsQ5dy5czz++ONNn6fkKRkRshlpSkFkE/TN\nmoJoH34p7N5/ZCsuprgYtRK0HJXYfhFQVSyWxdQ6vX+31aGZmVhcnCJJ6lkqy3VBeZ6SyUAukhv+\njG31g9D1Fsotp9GOWRBGWjgNspi8IAaRceoX63ODIIQpiNbpWwWxXGZmZnj88ce5fPlyXfdPp8x/\nV5WM45BxHEqu/3AM5Re6rm67vlo5PVdHQSw1Oy0UCpXX4cry4eH61WMNYyVUqrnGZjH5+0XqZzGF\nH7PJS+tYDKIB0QE/tB6efPJJAHbv3h2b5bRcJTEzM8PQUGvF88quknWETKZqQZQ8Zcv4AAPZDI+f\nm6aQqw7+uTozq6Vk37FjR+WYfD7Pnj17rC6TkTj1GgapKhr0ecg0iEG4lsWUGGZBNCAMXMcxN1d/\nxfJySMJt5QYxiJzjxyCKZZdiySUTsQgK2apFkc2u7N/uOM6CoLopByMNlg5SL7GS2rMspqSwS9iA\naKC4tjrrqVOnuHjx4qLPNFIqaVH2lEzGn1WVXI93feYhHjoxiW80+A/LYC4ysJtv1uhiqtVcF+/z\nIj2p4/aDX/EVWDBBMlZG317B5bqC4gb+mZmZRdvCEt3NlvtuNW4xW3IrFkQ24zBdLHM5KNiXyziE\nZx+IKIh6WUz9kEJpdD/VLKb4dRCOBKU2rGFQ6vStglgujQbPuH2t9INo9F0HnrzCmclqpdjjF/2M\np1zWIecIJy9VXV/FclWGgYhbKQze1cZQmi1pbhhpISJ+Oq/Uea6CNFfH0fr9IKzURmKYgmiRaAwh\nTimoasvNg77/5CTnp/3v+YMvH+Bdf/PdyiwpLO993ebRRSb18YuzlVzxkcLiwX/Hjh3s2bOn8r7X\nG9EYq4d6K6U9TwMFUt+CqMQgLEjdMm1XECKyXUS+KiKPiMj3RORXgu2/KSInROT+4Odl7ZBntuTy\ne198hJOXGq89iJvNTE1NLYhTHDt2bNExJ06c4NFHH13WeaO4nvKHXz7I733hkQXbv374PK6nfOAf\n/V4V+YxDtmaFtOspr332dvbtWstTNo0sOncmk7FAs9GVOHXWOfgd5WiYxRTqDXMxtU4nLIgy8Kuq\nej1wK/BWEbkh2Pd+Vb0x+PlimkKEA/P3T03y2NlpPn3v8YbHxxW2a4bp6em6++5+5DR/cdcDDT//\nuQf8GkyXZ0v8/cOnK9vLnlKMROkK2cyi2ELGEa7bPMpbfvhqKztgrCrqxRg8j0oWU/2Wo+EizlRF\n7Ava7nRW1VPAqeD1FRF5BGh+CW/CZAO3zAPHL3P0/AyeKrvWDS17kc2VuRKFbIZ8xNcfdS2p6oJz\nup7y8W89AcCvv1brft/MfHW19V99+4mq3BlhvhRRELmFFsQvveBqtq4djD3n7t27m/2zDKMj+BbC\n4u2Kb104ToNSG5iLKSk6GoMQkV3As4BvBpt+WUQeFJEPicjadsgwH5mF//bnH+Z3v/AIX3jw1LLO\ncfDJK/yHTzzAOz7z3QXbowri4MGDTE5OVt6HwWWAAwcOMDs7u8jdNDlb4tHT8Q2ADp25wnwkCD02\nkAWqD8TNO9eyeWxhTOG9r30mv/Pqp6VWxtwwkqLeSumw1IbTYCW1ZTElR8cUhIiMAJ8G3qaqk8AH\ngKuBG/EtjPfW+dztIrJfRPafPXu2ZTkOnr6yaNsDx6vupKlimT/+6iEm5+pXV733mL8eYnK2xMEn\nq+erHfAvX75cef3euw5UXj92bppjx45x8ODBBcf/3pceWaBIotzz6Hlmgx7TP/+DuxERNq/xS2GE\nJb9rWTOYY8uaeKvCMLqJejGGsGFQo1pMnmrF0jBaoyMKQkRy+MrhY6r6GQBVPa2qrqp6wJ8Dt8R9\nVlXvUNV9qrpvw4YNK5YhHLwfPbN4hj5VrK55uPuRM9x37BL/EOmnUEt0pvL5B08xW3J531cO8pGv\nfGvBca7rVqyKmch3nJ+KX0197sri7Tsmhhgq+Gsa9h+9AMDYoB9o/pG9G7nt+o38xxdfV1dWw1gN\nZOpkMWkYg6B+X3XLYkqOTmQxCfBB4BFVfV9k+5bIYa8GHkpblmLZ5fjFWfbtWsurb6qGQdyIaygs\nfneluLjy6rmpIo+fm6ZU9hgZyPKsHeOcnSry4PFLPHxqclHTHtd1KxlNowNZnr3L96Jdmlm63MZA\nLsMztq3h1168l5c9zb9UX3rIrwt13Sa/BWgu4/CGW3awZqh+ZpIthjO6HVUN+j0s3heW2sg49RsG\nWbnv5OjEyqjnAT8LfFdE7g+2vQt4g4jciB+HOgK8OW1BTk8W8Tzl2bsmuHnnWl7+9C389y99n8Nn\npnj7Jx/g1t0TfCXIHPqnA2d57c3bKAQrki/NlHjHp/2Yw+71Q+QywuhAlvmyx2Nnq5lLrqeVGzVc\njV1yPa7Mldk6PsiD2ctcnKm6r4rFYqXO08hAlqm5Mi9+2mZ++uZtlWNeeMMm/vre42EVjUXprYax\n2nGkTi0mfOXRqKOcNQxKjk5kMd1DNJpaJdW01lqefPJJZuf9AXu4UC1DMRRUPZ2cLVWUQ8hn7jvB\nG27ZweRciV/7VDU99fFzM2waK5DLOMyUygvaex4+M8W1m/0Zfjh7f+iEH4vYNjHE2uEcFyMWRFi+\nw/WU2XmXlzxtM6+JKAfwZ0YTwzkuTDffdc4wVhOZOmmsYb9pkXgLAyC0/60UU+v07SWcnZ2tBHkH\nc1U9+ZM3bVt07NO2jgF+POIX79zP2z+xeO3C6cki+axDqezftdcEC9Mu1LiPVJWvH75APuvw9K1r\nGB/Kc3F6fsH+05NF3vzRe3E9Zet4fFD5N378qdywZYxfe1E6fbMNo5P4aaxxFoQ/8Gec+J7VAOXA\nLVyv74nRPH19BUMLIlrpdN3I4gygPRsWr0IG+O1XPZXn7JkA/JhCPnJDvvKZVwFwZc6PXXzn6EUe\nPjnJB+85wneOXWS+7JFxhM1jA5y4NFcxl8+ePcs9h6rZWfUykoYLWd7+omu5bstY03+vYawW6mYx\neUoGwXHqp7mWg16lOTMhWqavq7NtHCvwI3s3MDJQvQxD+Qxvf+G1lFyPtcN5PvyvR7hx+zifu//k\ngs86weD+88/bzfOuXs/1W0b56oHqwL538yhD+QwnLs0wW3L5k6AkRnW/r3Su3zLGPx44y98/cprp\n+TJP3zpeucFfeeNV7F7fese2sbGxBWswatm1a1fL32EYSeKvlF64rbJC2gmqudaxIGauXEKAXNZi\nEK3S1wri6g0jXB1jHdxwVXVW/l9f4VcB+f2fejr/eOAsxy7M8PDJSX7ltmsQETJSPX7DaHW2LyLs\n3TzKPY+e555Hzy84/5t+YCfPv8ZP0b1+yxiOI3xqv1/q44sP+plJ64bz/HhghbTK0NAQ4+PjsbWi\nYGErUcPoBuKC1GFMwsF/vpZaKGcuptbpawWxHNaPFHjNzdsqgea40hgbR/2Vy6FbaGK4umJ5ZCDL\n7nVDvP6WnQvcRkP5DHs3jfDIqYUL9l741E3Lkm/Pnj3Mz8+Tz+c5ffr0ohpQg4O2QM5YPcSlsYYW\ngzgCQZC6toQN+HXKwFxMSWAKYpk0qtG0cTTPC2/YxPOvWQ/AzTsnuPuRM1y3eZRfe/Heup/7t8/f\nw6EzUzz1qjHe83cHOHZhhh/Zu3FJWRzHqSy8y+Vylcqs4+PjDYsEGka3E9dSNFye5CBo8ByqLi7K\n56mgmIspCfpWQUQH16QQEV737O2V99duGuEv3rRvyc+NDea4aae/aO7tL7qW6aLb1CKfbDbL1q1b\nKZUWpruOjIywefNm5ubmuHTpUkWpbd++fUFPacPoVuIURMXF5FBJr3FVcWqy5l2EWc2ZiykB+lJB\nNNPEp1AoUCwWGx7TCiMjI0xNLS7zMVLIxjb3iWPr1q3k8/nY4ntr1qxhbGyMQqHA6Ki/DmNoaKg1\noQ0jZcrlMuVymYnyOfLTOWZnZykUChw+fJipoB6aCOD5r11PydXMeUquhyJ1W+sazdO3CmIpnMB/\nuW7dOs6fP79ofyMFsmbNmgWF+UJCpTA8PMzmzZs5dOhQ7OfXrl3LxYsXY/eF9aey2eySVVlFhPHx\n8YbHGEY3EbpGQwvi2LFjiPh1l8rBpM4RoTwzyZjMcf78ebxSEVWlWCzieR6lUoms09gdbDRHXyuI\nNWvWMDw8TLFYXKAEdu3aVXlfrw3n4OAg8/Pzscomk8mQz+cr7Ui3b9+OiFSUxtq1a8lkMmzbto3j\nxxc2Ktq2bRvDw8PMzMxQLBbZsmULpVKJQqFAqVRifHzcbvwuZOfOnZVSKsbKCSde2YzwvROTfOOx\n8zxz+zifve8Efx9UKFg7lOf8VJE1MsfZc+cYzGVwPeX4xVnef9cBpoouT9kw2uG/pDfoawUxMDDA\n6Ogoo6OjFYUwOjpKoVBgw4YN5PP5WAUxMDDAhg0bWLt27YKWoyEiwq5duyrlu0PXTqFQoFAoVN7H\ntfsMt23ZsoWpqSlGR0dNIawCrJ93MmzevJmjR4/yqhu38v/9wyH+4p+rz9fEcI4XPXUzN+9cy/4j\nvoX9+1/8Pidi2gX/7A9c3TaZe5m+VBAh0YH3mmuu4dKlS6xd6weLc7kc69evXxSr2LRpU8VtE3Xx\n7PDp9ZQAAAmUSURBVNixgzNnzlQK7cUN6o7jVM4ffn5iYgIRYXp6mrm5ucrnQmViGP1EqGj3bh7l\nPT/1dD7y9aPce/QiP/7Mq3jFM7aQcQTHcXjm9jXsXj/M4+eq2XpP2zrGK55xFTsmhrjhOitBkwR9\nqSDi3EKO4zAxMRG7fdOmTRQKBebm5hb59MNsqMHBQUZGRpibm1tWplAYUxgfH+fKlSuxVoVh9BNb\nt27lxIkTDBey/NILrmbjxo14nkculyOfz+M4DsePH+fdL7+emXkXVWW4kF0Qu7NsvWToawXRrOsm\nVApxi8127txZqcC6du1aVLVy/NatWyvB7qXIZrMLrAvD6FdGRkbYuXMnxWKRNWvWxB6za9cuzp07\nx/aREc6dO8e6desYHh5meLj10jRGFVMQLRJNM3Uch/Xr11f2jYzEF/kzDKMxAwMDDeM6juOwcaO/\nmHTHjh2V7aYgkqUvV5IkqSAMwzB6la5TECLyEhE5ICKHROQdaXyH4ziMjo6SzfalAWUYhtEUXaUg\nRCQD/DHwUuAG/DakNyT9Pfl8nquuuspSEw3DMBrQVQoCuAU4pKqPqeo88FfAKzssk2EYRl/SbQpi\nK/BE5P3xYJthGIbRZrpNQcRFjRcsWhCR20Vkv4jsP3v2bMzhhmEYRhJ0m4I4DmyPvN8GLOj1qap3\nqOo+Vd0XLjIzDMMwkqfbFMS3gWtEZLeI5IHXA5/rsEyGYRh9SVfleapqWUR+GfgykAE+pKrf67BY\nhmEYfUlXKQgAVf0i8MVOy2EYhtHvdJuLyTAMw+gSpJnuat2KiJwFjq7w4+uBcwmKkwYmY+t0u3xg\nMiZBt8sH3SXjTlVdMstnVSuIVhCR/aq6r9NyNMJkbJ1ulw9MxiTodvlgdchYi7mYDMMwjFhMQRiG\nYRix9LOCuKPTAjSBydg63S4fmIxJ0O3yweqQcQF9G4MwDMMwGtPPFoRhGIbRgL5UEO1oStSEDNtF\n5Ksi8oiIfE9EfiXYPiEid4nIo8HvtcF2EZE/CmR+UERuaqOsGRG5T0Q+H7zfLSLfDGT8RFAWBREp\nBO8PBft3tUm+cRH5axH5fnA9n9tN11FE/kPwP35IRD4uIgOdvoYi8iEROSMiD0W2LfuaicibguMf\nFZE3tUHGPwj+zw+KyN+IyHhk3zsDGQ+IyIsj21N53uPki+z7NRFREVkfvO/INWwZVe2rH/wSHoeB\nPUAeeAC4oQNybAFuCl6PAgfxmyT9D+AdwfZ3AO8JXr8M+BJ+xdtbgW+2Uda3A/8H+Hzw/pPA64PX\nfwr8UvD63wF/Grx+PfCJNsl3J/CLwes8MN4t1xG/XP3jwGDk2v2bTl9D4IeAm4CHItuWdc2ACeCx\n4Pfa4PXalGV8EZANXr8nIuMNwbNcAHYHz3gmzec9Tr5g+3b8ckFHgfWdvIYt/42dFqDtfzA8F/hy\n5P07gXd2gVyfBV4IHAC2BNu2AAeC138GvCFyfOW4lOXaBtwN/Cjw+eAGPxd5SCvXM3gonhu8zgbH\nScryjQUDsNRs74rrSLXHyURwTT4PvLgbriGwq2bwXdY1A94A/Flk+4Lj0pCxZt+rgY8Frxc8x+F1\nTPt5j5MP+GvgmcARqgqiY9ewlZ9+dDF1XVOiwI3wLOCbwCZVPQUQ/N4YHNYpuf8n8B8BL3i/Drik\nquUYOSoyBvsvB8enyR7gLPCXgRvsL0RkmC65jqp6AvhD4BhwCv+a3Et3XcOQ5V6zTj9LP48/K6eB\nLG2VUUR+Ajihqg/U7OoK+ZZLPyqIJZsStRMRGQE+DbxNVScbHRqzLVW5ReQVwBlVvbdJOTpxbbP4\nZv4HVPVZwDS+e6QebZUx8OO/Et/tcRUwjN9zvZ4MXXV/BtSTqWOyisi7gTLwsXBTHVnaJqOIDAHv\nBv5r3O46cnTj/7tCPyqIJZsStQsRyeErh4+p6meCzadFZEuwfwtwJtjeCbmfB/yEiBzB7w/+o/gW\nxbiIhJWAo3JUZAz2rwEupCzjceC4qn4zeP/X+AqjW67jjwGPq+pZVS0BnwF+gO66hiHLvWYdeZaC\nQO4rgJ/RwC/TJTJejT8ReCB4ZrYB3xGRzV0i37LpRwXRFU2JRESADwKPqOr7Irs+B4SZDG/Cj02E\n298YZEPcClwO3QFpoarvVNVtqroL/zr9g6r+DPBV4DV1ZAxlf01wfKqzIVV9EnhCRPYGm24DHqZ7\nruMx4FYRGQr+56F8XXMNIyz3mn0ZeJGIrA0spRcF21JDRF4C/CfgJ1R1pkb21wdZYLuBa4Bv0cbn\nXVW/q6obVXVX8Mwcx09EeZIuuobLotNBkE784GcUHMTPbnh3h2T4QXxT8kHg/uDnZfj+5ruBR4Pf\nE8HxAvxxIPN3gX1tlvcFVLOY9uA/fIeATwGFYPtA8P5QsH9Pm2S7EdgfXMu/xc8G6ZrrCPwW8H3g\nIeCj+Jk2Hb2GwMfxYyIl/IHsF1ZyzfDjAIeCn59rg4yH8H324TPzp5Hj3x3IeAB4aWR7Ks97nHw1\n+49QDVJ35Bq2+mMrqQ3DMIxY+tHFZBiGYTSBKQjDMAwjFlMQhmEYRiymIAzDMIxYTEEYhmEYsZiC\nMIwIIuKKyP2Rn4bVP0XkLSLyxgS+90hY+dMwugVLczWMCCIypaojHfjeI/i58efa/d2GUQ+zIAyj\nCYIZ/ntE5FvBz1OC7b8pIr8WvP73IvJwUO//r4JtEyLyt8G2b4jIM4Lt60TkK0GBwT8jUpNHRP6f\n4DvuF5E/E5FMB/5kwzAFYRg1DNa4mF4X2TepqrcA/wu/JlUt7wCeparPAN4SbPst4L5g27uAjwTb\nfwO4R/0Cg58DdgCIyPXA64DnqeqNgAv8TLJ/omE0R3bpQwyjr5gNBuY4Ph75/f6Y/Q8CHxORv8Uv\n+QF+SZWfAlDVfwgshzX4zWZ+Mtj+BRG5GBx/G3Az8G2/dBODVIvmGUZbMQVhGM2jdV6HvBx/4P8J\n4L+IyFNpXM457hwC3Kmq72xFUMNIAnMxGUbzvC7y++vRHSLiANtV9av4DZbGgRHgawQuIhF5AXBO\n/b4f0e0vxS8wCH6RvNeIyMZg34SI7EzxbzKMupgFYRgLGRSR+yPv/05Vw1TXgoh8E39i9Yaaz2WA\n/x24jwR4v6peEpHfxO929yAwQ7Wc9m8BHxeR7wD/hF8WHFV9WET+M/CVQOmUgLfi9zc2jLZiaa6G\n0QSWhmr0I+ZiMgzDMGIxC8IwDMOIxSwIwzAMIxZTEIZhGEYspiAMwzCMWExBGIZhGLGYgjAMwzBi\nMQVhGIZhxPL/A1Szmpd8a8diAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1240842e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eps, rews = np.array(rewards_list).T\n",
    "smoothed_rews = running_mean(rews, 10)\n",
    "plt.plot(eps[-len(smoothed_rews):], smoothed_rews)\n",
    "plt.plot(eps, rews, color='grey', alpha=0.3)\n",
    "plt.xlabel('Episode')\n",
    "plt.ylabel('Total Reward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing\n",
    "We can watch our trained model play Cart-Pole!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_episodes = 10\n",
    "test_max_steps = 400\n",
    "env.reset()\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))\n",
    "    \n",
    "    for episode in range(1, test_episodes):\n",
    "        current_step = 0\n",
    "        while current_step < test_max_steps:\n",
    "            env.render()\n",
    "            \n",
    "            feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "            Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "            action = np.argmax(Qs)\n",
    "            \n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            if done:\n",
    "                current_step = test_max_steps\n",
    "                env.reset()\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "            else:\n",
    "                state = next_state\n",
    "                current_step += 1\n",
    "                "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}