examples/Tutorial_Using_the_Serpentine_Scenario.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to use a VeinsGym Environment\n",
    "\n",
    "In this tutorial, we will load the *Serpentine* example scenario using `VeinsGym` and run an agent within it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "\n",
    "To run VeinsGym and the examples in this notebook, we need a few dependencies ready to run:\n",
    "\n",
    "- [Sumo](https://www.eclipse.org/sumo/) (VeinsGym is tested with Sumo v1.6.0, other versions *may* work as well)\n",
    "- [OMNeT++](https://omnetpp.org/) v5.6+ (Not yet tested with OMNeT++ v6)\n",
    "- GNU `make` a `C++` compiler (ideally the one that you used to compile OMNeT++, check `opp_run -v`)\n",
    "- [Google Protocol Buffers](https://developers.google.com/protocol-buffers) compiler (`protoc`)\n",
    "\n",
    "If you have fulfilled these dependencies, running `protoc --version`, `sumo --version`, and `opp_run -v` should work fine.\n",
    "Otherwise you may have to adjust your `$PATH` variable to include them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing VeinsGym\n",
    "\n",
    "To run this notbook we first need to install the `veins_gym` package (e.g., to a virtual environment).\n",
    "To build the *Serpentine* scenario in the next step, we will also need *Snamkemake*, so let's install that as well.\n",
    "\n",
    "```bash\n",
    "pip install veins_gym snakemake\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking out the *Serpentine* scenario\n",
    "\n",
    "Next, we need a scenario to run.\n",
    "A scenario contains the code and configuration to run a VANET experiment using Veins and the VeinsGym client code.\n",
    "See https://github.com/tkn-tub/veins-gym/blob/master/doc/getting_started.md to find out how to write your own.\n",
    "\n",
    "In this tutorial we are using the *Serpentine* example scenario.\n",
    "You can clone it from https://github.com/tkn-tub/serpentine-env\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/tkn-tub/serpentine-env\n",
    "```\n",
    "\n",
    "The *Serpentine* scenario brings all the veins, veins-vlc, and scenario-specific code with it.\n",
    "Before you can use it, you have to compile it, though.\n",
    "This is done with *Snakemake*.\n",
    "We use all cores of the machine (`-jall`) to speed up the process.\n",
    "\n",
    "```bash\n",
    "cd serpentine-env\n",
    "snakemake -jall\n",
    "```\n",
    "\n",
    "Now the scenario is ready and we can setup the the gyn environment to expose it to an agent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register your scenario as a Gym Environment\n",
    "\n",
    "We import the general `gym` module and the `veins_gym` connection module to register the scenario as a gym environment.\n",
    "The most important part is to point the `scenario_dir` argument to the directory in which your `omnetpp.ini` files lives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import veins_gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gym.register(\n",
    "    id=\"veins-v1\",\n",
    "    entry_point=\"veins_gym:VeinsEnv\",\n",
    "    kwargs={\n",
    "        \"scenario_dir\": \"../scenario\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create an `env` instance.\n",
    "We can use the `gym.make` call or other supporting wrappers from various RL-toolkits.\n",
    "Just give it the environment name we registered it with above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make(\"veins-v1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this Environment like any other one from OpenAI or such.\n",
    "Veins will automatically be started on `env.reset()` and restarted once an experiment is through."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.reset()\n",
    "done = False\n",
    "while not done:\n",
    "    random_action = env.action_space.sample()\n",
    "    observation, reward, done, info = env.step(random_action)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here on it is up to you.\n",
    "You can extract observations, evaluate exisiting policies, or train agents online.\n",
    "Have fun!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Mean Reward\n",
    "\n",
    "Like before we want to choose random actions.\n",
    "But this time, we are going to track the rewards and compute the mean in the end.\n",
    "In production, your RL-toolkit will probably do this for you"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.reset()\n",
    "done = False\n",
    "rewards = []\n",
    "while not done:\n",
    "    random_action = env.action_space.sample()\n",
    "    observation, reward, done, info = env.step(random_action)\n",
    "    rewards.append(reward)\n",
    "print(\"Number of steps taken:\", len(rewards))\n",
    "print(\"Mean reward:\", sum(rewards) / len(rewards))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}