6_dqn_cartpole.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ChainerRL Quickstart Guide\n",
    "\n",
    "This is a quickstart guide for users who just want to try ChainerRL for the first time.\n",
    "\n",
    "If you have not yet installed ChainerRL, run the command below to install it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting chainerrl\n",
      "  Downloading chainerrl-0.2.0.tar.gz (56kB)\n",
      "Collecting cached-property (from chainerrl)\n",
      "  Downloading cached_property-1.3.1-py2.py3-none-any.whl\n",
      "Requirement already satisfied: chainer>=2.0.0 in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainerrl)\n",
      "Collecting future (from chainerrl)\n",
      "  Downloading future-0.16.0.tar.gz (824kB)\n",
      "Collecting gym>=0.7.3 (from chainerrl)\n",
      "  Downloading gym-0.9.3.tar.gz (157kB)\n",
      "Requirement already satisfied: numpy>=1.10.4 in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainerrl)\n",
      "Requirement already satisfied: pillow in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainerrl)\n",
      "Requirement already satisfied: scipy in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainerrl)\n",
      "Requirement already satisfied: six>=1.9.0 in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainer>=2.0.0->chainerrl)\n",
      "Requirement already satisfied: protobuf>=2.6.0 in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainer>=2.0.0->chainerrl)\n",
      "Requirement already satisfied: mock in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainer>=2.0.0->chainerrl)\n",
      "Requirement already satisfied: nose in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainer>=2.0.0->chainerrl)\n",
      "Requirement already satisfied: filelock in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from chainer>=2.0.0->chainerrl)\n",
      "Requirement already satisfied: requests>=2.0 in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from gym>=0.7.3->chainerrl)\n",
      "Collecting pyglet>=1.2.0 (from gym>=0.7.3->chainerrl)\n",
      "  Downloading pyglet-1.2.4-py3-none-any.whl (964kB)\n",
      "Requirement already satisfied: olefile in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from pillow->chainerrl)\n",
      "Requirement already satisfied: setuptools in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg (from protobuf>=2.6.0->chainer>=2.0.0->chainerrl)\n",
      "Requirement already satisfied: pbr>=0.11 in /home/shunta/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages (from mock->chainer>=2.0.0->chainerrl)\n",
      "Building wheels for collected packages: chainerrl, future, gym\n",
      "  Running setup.py bdist_wheel for chainerrl: started\n",
      "  Running setup.py bdist_wheel for chainerrl: finished with status 'done'\n",
      "  Stored in directory: /home/shunta/.cache/pip/wheels/50/e1/16/d6879538da7fe0053f5b61c3d1f4e1b009464d3564b99c792c\n",
      "  Running setup.py bdist_wheel for future: started\n",
      "  Running setup.py bdist_wheel for future: finished with status 'done'\n",
      "  Stored in directory: /home/shunta/.cache/pip/wheels/c2/50/7c/0d83b4baac4f63ff7a765bd16390d2ab43c93587fac9d6017a\n",
      "  Running setup.py bdist_wheel for gym: started\n",
      "  Running setup.py bdist_wheel for gym: finished with status 'done'\n",
      "  Stored in directory: /home/shunta/.cache/pip/wheels/2b/16/05/14202d3528fb14912254fe7062bfc8b061ade8de9409f1abd0\n",
      "Successfully built chainerrl future gym\n",
      "Installing collected packages: cached-property, future, pyglet, gym, chainerrl\n",
      "Successfully installed cached-property-1.3.1 chainerrl-0.2.0 future-0.16.0 gym-0.9.3 pyglet-1.2.4\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pip install chainerrl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have already installed ChainerRL, let's begin!\n",
    "\n",
    "First, you need to import necessary modules. The module name of ChainerRL is `chainerrl`. Let's import `gym` and `numpy` as well since they are used later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import chainer\n",
    "import chainer.functions as F\n",
    "import chainer.links as L\n",
    "import chainerrl\n",
    "import gym\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ChainerRL can be used for any problems if they are modeled as \"environments\". [OpenAI Gym](https://github.com/openai/gym) provides various kinds of benchmark environments and defines the common interface among them. ChainerRL uses a subset of the interface. Specifically, an environment must define its observation space and action space and have at least two methods: `reset` and `step`.\n",
    "\n",
    "- `env.reset` will reset the environment to the initial state and return the initial observation.\n",
    "- `env.step` will execute a given action, move to the next state and return four values:\n",
    "  - a next observation\n",
    "  - a scalar reward\n",
    "  - a boolean value indicating whether the current state is terminal or not\n",
    "  - additional information\n",
    "- `env.render` will render the current state.\n",
    "\n",
    "Let's try 'CartPole-v0', which is a classic control problem. You can see below that its observation space consists of four real numbers while its action space consists of two discrete actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-09-23 05:17:39,776] Making new env: CartPole-v0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "observation space: Box(4,)\n",
      "action space: Discrete(2)\n",
      "initial observation: [ 0.0042395  -0.01029501  0.01804505  0.04476617]\n",
      "next observation: [ 0.0040336  -0.205671    0.01894037  0.3430874 ]\n",
      "reward: 1.0\n",
      "done: False\n",
      "info: {}\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "print('observation space:', env.observation_space)\n",
    "print('action space:', env.action_space)\n",
    "\n",
    "obs = env.reset()\n",
    "env.render(close=True)\n",
    "print('initial observation:', obs)\n",
    "\n",
    "action = env.action_space.sample()\n",
    "obs, r, done, info = env.step(action)\n",
    "print('next observation:', obs)\n",
    "print('reward:', r)\n",
    "print('done:', done)\n",
    "print('info:', info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have defined your environment. Next, you need to define an agent, which will learn through interactions with the environment.\n",
    "\n",
    "ChainerRL provides various agents, each of which implements a deep reinforcement learning algorithm.\n",
    "\n",
    "To use [DQN (Deep Q-Network)](http://dx.doi.org/10.1038/nature14236), you need to define a Q-function that receives an observation and returns an expected future return for each action the agent can take. In ChainerRL, you can define your Q-function as `chainer.Link` as below. Note that the outputs are wrapped by `chainerrl.action_value.DiscreteActionValue`, which implements `chainerrl.action_value.ActionValue`. By wrapping the outputs of Q-functions, ChainerRL can treat discrete-action Q-functions like this and [NAFs (Normalized Advantage Functions)](https://arxiv.org/abs/1603.00748) in the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class QFunction(chainer.Chain):\n",
    "\n",
    "    def __init__(self, obs_size, n_actions, n_hidden_channels=50):\n",
    "        super().__init__(\n",
    "            l0=L.Linear(obs_size, n_hidden_channels),\n",
    "            l1=L.Linear(n_hidden_channels, n_hidden_channels),\n",
    "            l2=L.Linear(n_hidden_channels, n_actions))\n",
    "\n",
    "    def __call__(self, x, test=False):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            x (ndarray or chainer.Variable): An observation\n",
    "            test (bool): a flag indicating whether it is in test mode\n",
    "        \"\"\"\n",
    "        h = F.tanh(self.l0(x))\n",
    "        h = F.tanh(self.l1(h))\n",
    "        return chainerrl.action_value.DiscreteActionValue(self.l2(h))\n",
    "\n",
    "obs_size = env.observation_space.shape[0]\n",
    "n_actions = env.action_space.n\n",
    "q_func = QFunction(obs_size, n_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to use CUDA for computation, as usual as in Chainer, call `to_gpu`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Uncomment to use CUDA\n",
    "# q_func.to_gpu(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use ChainerRL's predefined Q-functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "_q_func = chainerrl.q_functions.FCStateQFunctionWithDiscreteAction(\n",
    "    obs_size, n_actions,\n",
    "    n_hidden_layers=2, n_hidden_channels=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in Chainer, `chainer.Optimizer` is used to update models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use Adam to optimize q_func. eps=1e-2 is for stability.\n",
    "optimizer = chainer.optimizers.Adam(eps=1e-2)\n",
    "optimizer.setup(q_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Q-function and its optimizer are used by a DQN agent. To create a DQN agent, you need to specify a bit more parameters and configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the discount factor that discounts future rewards.\n",
    "gamma = 0.95\n",
    "\n",
    "# Use epsilon-greedy for exploration\n",
    "explorer = chainerrl.explorers.ConstantEpsilonGreedy(\n",
    "    epsilon=0.3, random_action_func=env.action_space.sample)\n",
    "\n",
    "# DQN uses Experience Replay.\n",
    "# Specify a replay buffer and its capacity.\n",
    "replay_buffer = chainerrl.replay_buffer.ReplayBuffer(capacity=10 ** 6)\n",
    "\n",
    "# Since observations from CartPole-v0 is numpy.float64 while\n",
    "# Chainer only accepts numpy.float32 by default, specify\n",
    "# a converter as a feature extractor function phi.\n",
    "phi = lambda x: x.astype(np.float32, copy=False)\n",
    "\n",
    "# Now create an agent that will interact with the environment.\n",
    "agent = chainerrl.agents.DoubleDQN(\n",
    "    q_func, optimizer, replay_buffer, gamma, explorer,\n",
    "    replay_start_size=500, update_interval=1,\n",
    "    target_update_interval=100, phi=phi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have an agent and an environment. It's time to start reinforcement learning!\n",
    "\n",
    "In training, use `agent.act_and_train` to select exploratory actions. `agent.stop_episode_and_train` must be called after finishing an episode. You can get training statistics of the agent via `agent.get_statistics`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 10 R: 12.0 statistics: [('average_q', 0.0077787917633448615), ('average_loss', 0)]\n",
      "episode: 20 R: 43.0 statistics: [('average_q', 0.013923729594215806), ('average_loss', 0)]\n",
      "episode: 30 R: 10.0 statistics: [('average_q', 0.04999595856865319), ('average_loss', 0.15626195506060395)]\n",
      "episode: 40 R: 10.0 statistics: [('average_q', 0.18431173820404814), ('average_loss', 0.19973429628136666)]\n",
      "episode: 50 R: 16.0 statistics: [('average_q', 0.4329778858284125), ('average_loss', 0.12129529302886367)]\n",
      "episode: 60 R: 40.0 statistics: [('average_q', 1.5867962687319506), ('average_loss', 0.1231642400453139)]\n",
      "episode: 70 R: 36.0 statistics: [('average_q', 4.5508317081422485), ('average_loss', 0.14574642336842872)]\n",
      "episode: 80 R: 70.0 statistics: [('average_q', 7.293821113338115), ('average_loss', 0.222018443450522)]\n",
      "episode: 90 R: 42.0 statistics: [('average_q', 9.706054559843952), ('average_loss', 0.22261116615911836)]\n",
      "episode: 100 R: 148.0 statistics: [('average_q', 13.271654782141711), ('average_loss', 0.2537233644580171)]\n",
      "episode: 110 R: 185.0 statistics: [('average_q', 17.379473389886567), ('average_loss', 0.23995480935576677)]\n",
      "episode: 120 R: 179.0 statistics: [('average_q', 19.205810990096783), ('average_loss', 0.20982516267359438)]\n",
      "episode: 130 R: 200.0 statistics: [('average_q', 19.86128616157245), ('average_loss', 0.17017104907517325)]\n",
      "episode: 140 R: 160.0 statistics: [('average_q', 20.14523553965665), ('average_loss', 0.17918074812334736)]\n",
      "episode: 150 R: 200.0 statistics: [('average_q', 20.386843352118866), ('average_loss', 0.1511973771788008)]\n",
      "episode: 160 R: 200.0 statistics: [('average_q', 20.524274776492966), ('average_loss', 0.181143022239863)]\n",
      "episode: 170 R: 200.0 statistics: [('average_q', 20.501493065164738), ('average_loss', 0.1426581032476842)]\n",
      "episode: 180 R: 146.0 statistics: [('average_q', 20.37513869566722), ('average_loss', 0.12322326194384814)]\n",
      "episode: 190 R: 55.0 statistics: [('average_q', 20.404746612680285), ('average_loss', 0.13629612704703933)]\n",
      "episode: 200 R: 200.0 statistics: [('average_q', 20.572537269328773), ('average_loss', 0.1488116341248042)]\n",
      "Finished.\n"
     ]
    }
   ],
   "source": [
    "n_episodes = 200\n",
    "max_episode_len = 200\n",
    "for i in range(1, n_episodes + 1):\n",
    "    obs = env.reset()\n",
    "    reward = 0\n",
    "    done = False\n",
    "    R = 0  # return (sum of rewards)\n",
    "    t = 0  # time step\n",
    "    while not done and t < max_episode_len:\n",
    "        # Uncomment to watch the behaviour\n",
    "        # env.render()\n",
    "        action = agent.act_and_train(obs, reward)\n",
    "        obs, reward, done, _ = env.step(action)\n",
    "        R += reward\n",
    "        t += 1\n",
    "    if i % 10 == 0:\n",
    "        print('episode:', i,\n",
    "              'R:', R,\n",
    "              'statistics:', agent.get_statistics())\n",
    "    agent.stop_episode_and_train(obs, reward, done)\n",
    "print('Finished.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you finished training the agent. How good is the agent now? You can test it by using `agent.act` and `agent.stop_episode` instead. Exploration such as epsilon-greedy is not used anymore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test episode: 0 R: 200.0\n",
      "test episode: 1 R: 200.0\n",
      "test episode: 2 R: 200.0\n",
      "test episode: 3 R: 200.0\n",
      "test episode: 4 R: 200.0\n",
      "test episode: 5 R: 200.0\n",
      "test episode: 6 R: 200.0\n",
      "test episode: 7 R: 200.0\n",
      "test episode: 8 R: 200.0\n",
      "test episode: 9 R: 200.0\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    obs = env.reset()\n",
    "    done = False\n",
    "    R = 0\n",
    "    t = 0\n",
    "    while not done and t < 200:\n",
    "        env.render(close=True)\n",
    "        action = agent.act(obs)\n",
    "        obs, r, done, _ = env.step(action)\n",
    "        R += r\n",
    "        t += 1\n",
    "    print('test episode:', i, 'R:', R)\n",
    "    agent.stop_episode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If test scores are good enough, the only remaining task is to save the agent so that you can reuse it. What you need to do is to simply call `agent.save` to save the agent, then `agent.load` to load the saved agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save an agent to the 'agent' directory\n",
    "agent.save('agent')\n",
    "\n",
    "# Uncomment to load an agent from the 'agent' directory\n",
    "# agent.load('agent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RL completed!\n",
    "\n",
    "But writing code like this every time you use RL might be boring. So, ChainerRL has utility functions that do these things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outdir:result step:86 episode:0 R:86.0\n",
      "statistics:[('average_q', 20.728489019516747), ('average_loss', 0.13604925025581077)]\n",
      "outdir:result step:286 episode:1 R:200.0\n",
      "statistics:[('average_q', 20.671014208079793), ('average_loss', 0.14984728771766473)]\n",
      "outdir:result step:396 episode:2 R:110.0\n",
      "statistics:[('average_q', 20.658295082215886), ('average_loss', 0.16141102891913808)]\n",
      "outdir:result step:596 episode:3 R:200.0\n",
      "statistics:[('average_q', 20.65092498811014), ('average_loss', 0.11670109444167831)]\n",
      "outdir:result step:796 episode:4 R:200.0\n",
      "statistics:[('average_q', 20.624282196582172), ('average_loss', 0.15006617026267832)]\n",
      "outdir:result step:996 episode:5 R:200.0\n",
      "statistics:[('average_q', 20.590381701508214), ('average_loss', 0.17453604165516437)]\n",
      "outdir:result step:1196 episode:6 R:200.0\n",
      "statistics:[('average_q', 20.571275081196642), ('average_loss', 0.16252849495287455)]\n",
      "test episode: 0 R: 200.0\n",
      "test episode: 1 R: 200.0\n",
      "test episode: 2 R: 200.0\n",
      "test episode: 3 R: 200.0\n",
      "test episode: 4 R: 200.0\n",
      "test episode: 5 R: 200.0\n",
      "test episode: 6 R: 200.0\n",
      "test episode: 7 R: 200.0\n",
      "test episode: 8 R: 200.0\n",
      "test episode: 9 R: 200.0\n",
      "The best score is updated -3.40282e+38 -> 200.0\n",
      "Saved the agent to result/1196\n",
      "outdir:result step:1244 episode:7 R:48.0\n",
      "statistics:[('average_q', 20.44840300754298), ('average_loss', 0.1455696393507992)]\n",
      "outdir:result step:1444 episode:8 R:200.0\n",
      "statistics:[('average_q', 20.443317168193577), ('average_loss', 0.1385756250812212)]\n",
      "outdir:result step:1644 episode:9 R:200.0\n",
      "statistics:[('average_q', 20.388818403317572), ('average_loss', 0.11136568147911419)]\n",
      "outdir:result step:1844 episode:10 R:200.0\n",
      "statistics:[('average_q', 20.393853468915438), ('average_loss', 0.1388451133452519)]\n",
      "outdir:result step:1951 episode:11 R:107.0\n",
      "statistics:[('average_q', 20.403746200029968), ('average_loss', 0.1201870912602859)]\n",
      "outdir:result step:2000 episode:12 R:49.0\n",
      "statistics:[('average_q', 20.413271961263554), ('average_loss', 0.13582760984249495)]\n",
      "test episode: 0 R: 200.0\n",
      "test episode: 1 R: 200.0\n",
      "test episode: 2 R: 200.0\n",
      "test episode: 3 R: 200.0\n",
      "test episode: 4 R: 200.0\n",
      "test episode: 5 R: 200.0\n",
      "test episode: 6 R: 200.0\n",
      "test episode: 7 R: 200.0\n",
      "test episode: 8 R: 200.0\n",
      "test episode: 9 R: 200.0\n",
      "Saved the agent to result/2000_finish\n"
     ]
    }
   ],
   "source": [
    "# Set up the logger to print info messages for understandability.\n",
    "import logging\n",
    "import sys\n",
    "gym.undo_logger_setup()  # Turn off gym's default logger settings\n",
    "logging.basicConfig(level=logging.INFO, stream=sys.stdout, format='')\n",
    "\n",
    "chainerrl.experiments.train_agent_with_evaluation(\n",
    "    agent, env,\n",
    "    steps=2000,           # Train the agent for 2000 steps\n",
    "    eval_n_runs=10,       # 10 episodes are sampled for each evaluation\n",
    "    max_episode_len=200,  # Maximum length of each episodes\n",
    "    eval_interval=1000,   # Evaluate the agent after every 1000 steps\n",
    "    outdir='result')      # Save everything to 'result' directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all of the ChainerRL quickstart guide. To know more about ChainerRL, please look into the `examples` directory and read and run the examples. Thank you!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}