Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "### Basics\n",
    "For this experiment, you will be providing instructions to a machine learner to learn a reward grid. Every box on the grid is assigned some reward value, and the machine learner can move around the grid, collecting rewards. Some boxes may have <font color = blue>negative reward values (punishment)</font>, so they should be avoioded. It is ideal to guide the learner toward boxes with <font color = red>high reward values</font>. Boxes with <font color = blue>negative rewards</font> are color-coded with <font color = blue>blue</font>, while boxes with <font color = red>high rewards</font> are colored with <font color = red>red</font>. \n",
    "                          \n",
    "<img style=\"float: middle;\" src=\"img/colorbar.png\" width=500 height=500>\n",
    "\n",
    "                          min_reward                       0                         max_reward\n",
    "                              \n",
    "The image below is what a ground truth reward map with one punishment box and one reward box looks like. \n",
    "\n",
    "<img style=\"float: middle;\" src=\"img/image1.png\" width=470 height=470>\n",
    "\n",
    "The learner **does not** know the ground truth rewards, but will do its best to learn them from the instructions you will provide. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment flow\n",
    "The process of teaching a machine learner a reward map consists of multiple timesteps. At each timestep, you will be presented with three maps. On the left, we show the ground truth reward map the machine learner is trying to estimate. In the middle, we see a map of the learner's **current estimation** of the ground truth reward. This starts out random, though we hope to see it improves in the teaching process. On the right, we have the learner's **current** policy, which represents the **most likely action** to be taken at each grid according to his **current** estimation of the rewards.\n",
    "\n",
    "<img style=\"float: middle;\" src=\"img/image2.png\" width=650 height=650>\n",
    "\n",
    "At each timestep, a set of arrows will be displayed on the left and middle maps. By selecting an arrow, you will give the instruction to the learner that it should follow the arrow's direction **if it were in the box the arrow originates from**. Notice that you don't need to think about the destination or the starting point of an agent. **The learner only needs to know what to do in a certain grid to recover the ground truth rewards.** Also, it is possible to use **the same arrow in different iterations**. You can consider that the learner forgets previous instructions after he updated the reward map. Thus, if, in the 3rd iteration, you think that the same arrow you selected at the 1st iteration is the best one again, you can choose that arrow.\n",
    "\n",
    "If an arrow is in a box on an edge of the grid, and its direction is towards the edge, such as the arrow in B1 in the image above, it indicates that the learner should stay in the same grid. When an arrow is selected, the arrow will turn green as in the image below. In this case, the selected arrow is in D4.\n",
    "\n",
    "<img style=\"float: middle;\" src=\"img/green_arrow.png\" width=750 height=750>\n",
    "\n",
    "After you have selected an arrow, you can press 'c' to confirm your choice, and one of the arrows in the batch may turn orange, which represents the selection of a machine teacher. In the image below, the machine teacher's selection is also in D4. The learner's map is still updated according to **the arrow that you selected**. The machine teacher's selection is merely a reference for you. **If your selection is different from the machine teacher's, no need to worry.** The learner is still possible to learn very well.\n",
    "\n",
    "<img style=\"float: middle;\" src=\"img/orange_arrow.png\" width=750 height=750>\n",
    "\n",
    "You can change you selection before the confirmation. After you select an arrow and confirm, click the **\"▶ Run\"** button at the menu bar. Then, the learner's current estimation of the ground truth reward map will be updated base on your selection, and next iteration will start. Your goal is to pick the arrow that is the most helpful to the learner. **<font color = 'red'>Remember that our goal is to teach the learner about the rewards in the entire map, not just the location of the red grid.</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Teaching Heuristics\n",
    "Though you can pick any arrow to instruct the learner as best as possible, we propose two heuristics that may help. \n",
    "\n",
    "1. A local heuristic. \n",
    "\n",
    "    When you select an arrow, you can expect the learner's reward estimation of the destination box to increase. On the other hand, the learner will decrease the reward estimation of the source box and all other neighboring boxes. This is best illustrated below. If the arrow at D3 is selected, we can expect box E3 to increase, while boxes C3, D2, D3, and D4 decrease. \n",
    "\n",
    "<img style=\"float: middle;\" src=\"img/local.jpeg\" width=590 height=590>\n",
    "\n",
    "2. A global heuristic. \n",
    "\n",
    "    A selected arrow can also teach the learner to increase estimation values in that direction, and decrease values in the opposite direction. In the image below, we illustrate this with a rightward arrow that increases the learner's estimated rewards of boxes on the right, but decreases values on the left. Note that boxes close to the selected arrow change more than those far away. \n",
    "\n",
    "\n",
    "<img style=\"float: middle;\" src=\"img/global.jpeg\" width=590 height=590>\n",
    "\n",
    "Notice that for both of these heuristics, the extent to which values increase and decrease may differ between iterations. Also, without knowing the underlying reward, there are usually more than one interpretations of an arrow selection. Thus, it is possible that the learner's reward map update might be different from the expectations. Last but not the least, as the teaching arrows should be selected based on the learner's current reward map, it is **<font color='red'>possible</font>** that **<font color = red>the best example is not where the agent should go to if it already has the ground truth reward</font>**. For instance, if the learner assigns higher reward than needed to a grid, then arrows go towards that grid may no longer be ideal for teaching examples, even if going towards that grid is reasonable given the ground truth reward map."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Warmup round\n",
    "We will begin with a sample warmup round. \n",
    "\n",
    "The warmup round consists of 10 iterations. In each iteration, you can simply click an arrow to select it (on either reward map), and press **'c' on the keyboard** to confirm the selected arrow (in green).\n",
    "At the end of each iteration, after 'c' is pressed, one of the arrows shown on the reward maps might turn yellow. The yellow arrow is the arrow determined to be optimal in the provided set of arrows. \n",
    "\n",
    "After the selected arrow is confirmed, click the \"▶ Run\" button in the menubar to run the next iteration. The number of iterations to be compeleted is shown on top of the reward maps.\n",
    "\n",
    "Begin the warmup round by running the next cell. This may take a few seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "from play import run\n",
    "lfh, _ = run(0, intro=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to start the first iteration. Then follow instructions on the plot and run the cell again until all iterations are finished. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lfh.iteration()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the message \"All iterations are completed\" pops up, the instruction collection is finished and this concludes the warmup round.\n",
    "\n",
    "Next, you will be presented with **5** different ground truth reward maps. For each map, you will have **2** differnt learners to teach. Each learner requires **30** iterations of teaching. The teaching process will be the same as the warmup round. After you finish providing instructions to the machine learner, **remember to run the data saving cell right after to save your instructions**.\n",
    "\n",
    "**<font color = red>Please finish teaching one map for both learners before you close each Jupyter notebook.</font>** Each map can take around 15 minutes. You can leave and resume for different maps.\n",
    "\n",
    "Now you can proceed to the first map by opening the notebook <font color= blue> [test_script1](test_script1.ipynb)</font>."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}