recsim/colab/RecSim_Developing_an_Agent.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SgRmNwtclIiO"
      },
      "source": [
        " Copyright 2019 The RecSim Authors.\n",
        "\n",
        " Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        " you may not use this file except in compliance with the License.\n",
        " You may obtain a copy of the License at\n",
        "\n",
        "     http://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        " Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BtBSsFuoaaNv"
      },
      "source": [
        "# Developing an Agent\n",
        "\n",
        "Having familiarized ourselves with the [overall structure](RecSim_Overview.ipynb) of RecSim and how [environments come together](RecSim_Developing_an_Environment.ipynb), we now turn to the final piece of the puzzle -- agent development.  In this tutorial, we aim to cover the following topics:\n",
        "* basics: what data (and how) does RecSim feed to an agent and what does it expect to receive in return;  \n",
        "* design: what features does RecSim provide for developing agents."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pVQshsAMdYai"
      },
      "source": [
        "# Basics\n",
        "\n",
        "![Detailed view of RecSim](https://github.com/google-research/recsim/blob/master/recsim/colab/figures/recsim_architecture_agent_centered.png?raw=true)\n",
        "To start unpacking the functionality of a RecSim agent, we once again refer to   the structural diagram. Here's what we discern from it on first pass -- an agent is meant to consume:\n",
        "* observations about the user's state,\n",
        "* observations about the user's response to a recommendation,\n",
        "* and a set of available documents $D$, each represented by a vector of features. \n",
        "In return, the agent is expected to produce a $K$-sized slate of elements of $D$ to be presented to the user's choice and transition model.\n",
        "\n",
        "To illustrate RecSim's agent API, we will the develop a simple bandit agent for RecSim's *interest exploration* environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "J1drgXe9CRqH"
      },
      "source": [
        "The *interest exploration* representes a clustered bandit problem: the world consists of some very large number of documents, which cluster into topics (this is a hard clustering -- one topic per document). We further posit that users also cluster into types.\n",
        "\n",
        "A user's affinity towards a document is a sum of the document's production quality plus the user's (user type's) affinity to the topic. This naturally creates a situation where a myopic agent that ranks documents by predicted click rate will favor types with high production value, as they have\n",
        "a high apriori probability of getting clicked across all user types. This leads\n",
        "the agent to ignoring to explore niche interests, producing a suboptimal policy.\n",
        "Hence the need for active exploration.\n",
        "\n",
        "For the purposes of exposition, we will define the agent method by method, which we will then assemble in a class.\n",
        "\n",
        "## Set-Up\n",
        "\n",
        "We now instantiate an environment to illustrate the various data types it produces and consumes, and how they are handled within an agent. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0BHEmh8BUhR_"
      },
      "outputs": [],
      "source": [
        "# @title Install\n",
        "!pip install --upgrade --no-cache-dir recsim"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "OlMUGf8CdX6D"
      },
      "outputs": [],
      "source": [
        "# @title Imports\n",
        "# Generic imports\n",
        "import functools\n",
        "from gym import spaces\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from scipy import stats\n",
        "# RecSim imports\n",
        "from recsim import agent\n",
        "from recsim import document\n",
        "from recsim import user\n",
        "from recsim.choice_model import MultinomialLogitChoiceModel\n",
        "from recsim.simulator import environment\n",
        "from recsim.simulator import recsim_gym\n",
        "from recsim.simulator import runner_lib"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "xJArIasbW27O"
      },
      "outputs": [],
      "source": [
        "from recsim.environments import interest_exploration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "v-Jk5gxeEDkj"
      },
      "source": [
        "Since we're not about to do anything fancy with this environment, we will initialize it with the provided *create_environment* function (further details on this [here](RecSim_Developing_an_Environment.ipynb)). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "f6Z9GmFDEYjD"
      },
      "outputs": [],
      "source": [
        "env_config = {'slate_size': 2,\n",
        "              'seed': 0,\n",
        "              'num_candidates': 15,\n",
        "              'resample_documents': True}\n",
        "ie_environment = interest_exploration.create_environment(env_config)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2ihJbWjbFXFd"
      },
      "source": [
        "At the start of each session, the simulator resets the environment, which triggers a resampling of the user. The *reset* call generates our initial observation. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "weXPwsYAFV6W"
      },
      "outputs": [],
      "source": [
        "initial_observation = ie_environment.reset()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cSDtDdzSF45O"
      },
      "source": [
        "## Observations\n",
        "\n",
        "A RecSim observation is a dictionary with 3 keys: \n",
        "* 'user', which represent the 'User Observable Features' in the structure diagram above,\n",
        "* 'doc', containing the current corpus of recommendable documents and their observable features ('Document Observable Features'),\n",
        "* and 'response', indicating the user's response to the last slate of recommendations ('User Response'). At this stage the 'response' key is vacuous and will be set to *None*, as no recommendation has been made yet.\n",
        "\n",
        "Note that this environment does not implement user observable features, so that field would be empty at all times."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "height": 350
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 18,
          "status": "ok",
          "timestamp": 1568741348688,
          "user": {
            "displayName": "Martin Mladenov",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBGIUwOMhxCVcEq0Q5I9YhNVfoEjuuMOWzhNOPn=s64",
            "userId": "05569865531106976534"
          },
          "user_tz": 420
        },
        "id": "BRJd0MWaGale",
        "outputId": "cc084fac-53e0-415a-8e5a-96b16b26c6e0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "User Observable Features\n",
            "[]\n",
            "User Response\n",
            "None\n",
            "Document Observable Features\n",
            "ID: 15 features: {'quality': 1.2272016322975663, 'cluster_id': 1}\n",
            "ID: 16 features: {'quality': 1.2925848895378007, 'cluster_id': 1}\n",
            "ID: 17 features: {'quality': 1.239770781835802, 'cluster_id': 1}\n",
            "ID: 18 features: {'quality': 1.4604555455549542, 'cluster_id': 1}\n",
            "ID: 19 features: {'quality': 2.1023342470023874, 'cluster_id': 0}\n",
            "ID: 20 features: {'quality': 1.0957290496089296, 'cluster_id': 1}\n",
            "ID: 21 features: {'quality': 2.372569629131807, 'cluster_id': 0}\n",
            "ID: 22 features: {'quality': 1.3492800243147158, 'cluster_id': 1}\n",
            "ID: 23 features: {'quality': 1.0067018798187535, 'cluster_id': 1}\n",
            "ID: 24 features: {'quality': 1.2044856191727935, 'cluster_id': 1}\n",
            "ID: 25 features: {'quality': 2.1835115903440956, 'cluster_id': 0}\n",
            "ID: 26 features: {'quality': 1.1941158468553823, 'cluster_id': 1}\n",
            "ID: 27 features: {'quality': 1.0351464593750552, 'cluster_id': 1}\n",
            "ID: 28 features: {'quality': 2.2959262349993166, 'cluster_id': 0}\n",
            "ID: 29 features: {'quality': 2.059365556961282, 'cluster_id': 0}\n"
          ]
        }
      ],
      "source": [
        "print('User Observable Features')\n",
        "print(initial_observation['user'])\n",
        "print('User Response')\n",
        "print(initial_observation['response'])\n",
        "print('Document Observable Features')\n",
        "for doc_id, doc_features in initial_observation['doc'].items():\n",
        "  print('ID:', doc_id, 'features:', doc_features)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3SC9eGj1KJ9k"
      },
      "source": [
        "We are thus presented with a corpus of 15 documents (*num_candidates*), each represented by their topic and their production quality score. Note, though, that the user's affinity is not an observable quantity.\n",
        "\n",
        "The observation format specification can be accessed as a feature of the environment in the form of an OpenAI gym space. It is also provided to the agent at initialization time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "height": 350
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 71,
          "status": "ok",
          "timestamp": 1568741348810,
          "user": {
            "displayName": "Martin Mladenov",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBGIUwOMhxCVcEq0Q5I9YhNVfoEjuuMOWzhNOPn=s64",
            "userId": "05569865531106976534"
          },
          "user_tz": 420
        },
        "id": "SOvbXmVQF371",
        "outputId": "7bc3f3d6-461a-4cf3-f95e-c0b0a3d7d8cf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Document observation space\n",
            "15 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "16 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "17 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "18 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "19 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "20 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "21 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "22 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "23 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "24 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "25 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "26 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "27 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "28 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "29 : Dict(cluster_id:Discrete(2), quality:Box())\n",
            "Response observation space\n",
            "Tuple(Dict(click:Discrete(2), cluster_id:Discrete(2), quality:Box()), Dict(click:Discrete(2), cluster_id:Discrete(2), quality:Box()))\n",
            "User observation space\n",
            "Box(0,)\n"
          ]
        }
      ],
      "source": [
        "print('Document observation space')\n",
        "for key, space in ie_environment.observation_space['doc'].spaces.items():\n",
        "  print(key, ':', space)\n",
        "print('Response observation space')\n",
        "print(ie_environment.observation_space['response'])\n",
        "print('User observation space')\n",
        "print(ie_environment.observation_space['user'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "St4Kr6vYbmCb"
      },
      "source": [
        "## Slates\n",
        "A RecSim slate is a list of $K$ indices of *obeservation['doc']*. E.g. the slate [0, 1] corresponds to the slate consisting of:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "height": 50
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 26,
          "status": "ok",
          "timestamp": 1568741348855,
          "user": {
            "displayName": "Martin Mladenov",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBGIUwOMhxCVcEq0Q5I9YhNVfoEjuuMOWzhNOPn=s64",
            "userId": "05569865531106976534"
          },
          "user_tz": 420
        },
        "id": "GARdpz7tEN83",
        "outputId": "d6fcaf1f-3731-4e7c-938b-af424b794162"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "('15', {'quality': 1.2272016322975663, 'cluster_id': 1})\n",
            "('16', {'quality': 1.2925848895378007, 'cluster_id': 1})\n"
          ]
        }
      ],
      "source": [
        "slate = [0, 1]\n",
        "for slate_doc in slate:\n",
        "  print(list(initial_observation['doc'].items())[slate_doc])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mIa9Aks1dSPq"
      },
      "source": [
        "The action space gym specification is also provided by the environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "height": 33
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 21,
          "status": "ok",
          "timestamp": 1568741348894,
          "user": {
            "displayName": "Martin Mladenov",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBGIUwOMhxCVcEq0Q5I9YhNVfoEjuuMOWzhNOPn=s64",
            "userId": "05569865531106976534"
          },
          "user_tz": 420
        },
        "id": "g_k6EaxpdOSE",
        "outputId": "39292b56-f2fa-4f50-a0a2-8c0b15261bb2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "MultiDiscrete([15 15])"
            ]
          },
          "execution_count": 60,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ie_environment.action_space"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dFbqcvKCiIdY"
      },
      "source": [
        "When the first slate is available, the simulator will run the environment and generate a new observation, along with a reward for the agent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "K5htT89RiGdO"
      },
      "outputs": [],
      "source": [
        "observation, reward, done, _ = ie_environment.step(slate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Xd5q0q2qdeX5"
      },
      "source": [
        "The main job of the agent is to produce a valid slate for each step of the simulation. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "K011d6qbee6J"
      },
      "source": [
        "## A trivial agent\n",
        "\n",
        "On a most basic level, the main function of the agent can be fulfilled by simply implementing a step-function. Let us implement a very basic agent which just serves the first $K$ documents from the corpus. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mdAGgbcped1J"
      },
      "outputs": [],
      "source": [
        "from recsim.agent import AbstractEpisodicRecommenderAgent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NbSIEw1ngq0L"
      },
      "source": [
        "A RecSim agent inherits from *AbstractEpisodicRecommenderAgent*. Required arguments (which RecSim will pass to the agent at simulation time) for the agent's init are the observation_space and action_space. We can use them to validate whether the environment meets the preconditions for the agent's operation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "z4g5xIwxdb4Z"
      },
      "outputs": [],
      "source": [
        "class StaticAgent(AbstractEpisodicRecommenderAgent):\n",
        "  def __init__(self, observation_space, action_space):\n",
        "    # Check if document corpus is large enough.\n",
        "    if len(observation_space['doc'].spaces) \u003c len(action_space.nvec):\n",
        "      raise RuntimeError('Slate size larger than size of the corpus.')\n",
        "    super(StaticAgent, self).__init__(action_space)\n",
        "\n",
        "  def step(self, reward, observation):\n",
        "    print(observation)\n",
        "    return list(range(self._slate_size))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vtpUsmLKj2Uj"
      },
      "source": [
        "This agent will statically recommend the first *K* documents of the corpus. For reasons that will become clear soon, we'll also have it print the observation.\n",
        "\n",
        "We can now run it in RecSim using *runner_lib* (See [tutorial](RecSim_Overview.ipynb) for details).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "tKV4mRcgiorx"
      },
      "outputs": [],
      "source": [
        "def create_agent(sess, environment, eval_mode, summary_writer=None):\n",
        "  return StaticAgent(environment.observation_space, environment.action_space)\n",
        "\n",
        "tmp_base_dir = '/tmp/recsim/'\n",
        "\n",
        "runner = runner_lib.EvalRunner(\n",
        "  base_dir=tmp_base_dir,\n",
        "  create_agent_fn=create_agent,\n",
        "  env=ie_environment,\n",
        "  max_eval_episodes=1,\n",
        "  max_steps_per_episode=5,\n",
        "  test_mode=True)\n",
        "\n",
        "# We won't run this, but we totally could\n",
        "# runner.run_experiment()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jae7rcS8nG5S"
      },
      "source": [
        "# Design: Hierarchical Agent Layers \n",
        "\n",
        "Now that we've gotten a basic agent off the ground, we might want to set our aims a little higher. That is, let's see if we can build an agent that actually does something useful. \n",
        "\n",
        "The way this problem is set up, a natural heuristic presents itself. We can run a bandit algorithm to reveal the average engagement of a user with each cluster of documents. That is, each cluster becomes an arm. Once the algorithm has chosen a cluster, we serve take the highest quality video from that cluster. This is a metaphor for a situation that occurs often in recommender systems that serve as a front end to multiple (sub-)products: within each session, the user will interact with the recommender with some intent in mind, that is, to realize some task that can be fulfilled by one of the possible sub-products. Sometimes, the user will issue an explicit query (e.g., enter search terms), which effectively makes that intent observable up ot query interpretation uncertainty. Most often, however, the intent will be latent -- the user will reveal it indirectly by chosing among a set of items from the slate. We assume that had the intent been observable, a product-specific policy would be available to fulfill it.   \n",
        "\n",
        "This set-up captures some typical features of practical recommender systems -- they tend to very hierarchical, often very heuristic due to the complexity of the environment they operate in, and also very idiosyncratic to the task at hand. For this reason, RecSim's approach to agent engineering is very modular. Instead of providing a wide array of agents, we provide an easily extendable set of agent building blocks, called Agent Layers, which could be combined into hierarchies to create more complex agents.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g8mhcm9jEHKv"
      },
      "source": [
        "## Hierarchical agent layers\n",
        "![Hierarchical agent architecture](https://github.com/google-research/recsim/blob/master/recsim/colab/figures/agent_architecture.png?raw=true)\n",
        "\n",
        "A hierarchical agent layer does not materialize a slate of documents, but relies on one or more base agents to do so. The hierarchical agent architecture in RecSim can roughly be described follows:\n",
        "* a hierarchical agent layer receives an observationand reward from the environment; it preprocesses the raw observation and passes it to one or more base agents.\n",
        "* Each base agent outputs either a slate or an abstract action (depending on the use case), which is then post-processed by the layer to create/output the slate (concrete action). \n",
        "\n",
        "Hierarchical layers are recursively stackable in a fashion similar to Keras layers. Hierarchical  layers  are  defined  by  their  pre-  and  post-processing functions and can play many roles dependinghow these are implemented. For example, a layer can beused as a pure feature injector — it can extract some feature from the (history of) observations and pass it to the base agent, while keeping the post-processing function vacuous. This allows decoupling of feature- and agent-engineering. Various regularizers can be implemented in a similar fashion by modifying the reward. Layers may also be stateful and dynamic, as the pre- or post-processing functions may implement parameter updates or learning mechanisms. \n",
        "\n",
        "We will not discuss how to implement these layers here (the reader is referred to examples in the *layers/* directory), rather, we will show their usage and benefits. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MB4VQFzjqKAq"
      },
      "source": [
        "## ClusterClickStats\n",
        "\n",
        "Recall that the *Interest Exploration* provides clicks as feedback, but does not keep track of cumulative click counts or impression counts. Since maintaining such statistics is generally useful, we provide an agent layer that does exactly that. That is, it monitors the stream of responses and retains the number of clicks and impressions from each cluster. The precondition is that the response space has a key 'click', as well as 'cluster_id'. If this is met, than the layer can be used with any environment/agent. Let's see how this works."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "AmKdauxrEGzY"
      },
      "outputs": [],
      "source": [
        "from recsim.agents.layers.cluster_click_statistics import ClusterClickStatsLayer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k2lvoE93ma8g"
      },
      "source": [
        "\n",
        "A hierarchical agent layer is instantiated in a smilar way to usual agents, except that it takes in a constructor for a base agent, that is, an agent whose abstract action it can interpret. In the case of cluster click stats, it will not do any post-processing of the abstract action, that is, it simply relays the action of the base agent to the environment. This implies that the base agent will need to provide a full slate. \n",
        "\n",
        "Once instantiated, the cluster click stats layer will inject a sufficient statistic to the base agent's observation space containing clicks and impressions. Thus, the combination of both will behave like as if the base agent had an additional field in its observation space. We showcase this using our StaticAgent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "height": 70
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 37,
          "status": "ok",
          "timestamp": 1568741349201,
          "user": {
            "displayName": "Martin Mladenov",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBGIUwOMhxCVcEq0Q5I9YhNVfoEjuuMOWzhNOPn=s64",
            "userId": "05569865531106976534"
          },
          "user_tz": 420
        },
        "id": "1EMysDE43nn8",
        "outputId": "810376cb-cf1e-42d8-9e2c-2a0e5bf4a8c2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'user': array([], dtype=float64), 'doc': {'30': {'quality': 2.489224450301943, 'cluster_id': 0}, '31': {'quality': 2.125926607579561, 'cluster_id': 0}, '32': {'quality': 1.27448138607991, 'cluster_id': 1}, '33': {'quality': 1.2179911236932994, 'cluster_id': 1}, '34': {'quality': 1.177703750911228, 'cluster_id': 1}, '35': {'quality': 2.079489146813576, 'cluster_id': 0}, '36': {'quality': 1.1416765236282371, 'cluster_id': 1}, '37': {'quality': 1.2052916542615082, 'cluster_id': 1}, '38': {'quality': 1.2424683972006194, 'cluster_id': 1}, '39': {'quality': 1.8727966807396805, 'cluster_id': 0}, '40': {'quality': 1.1964488835024119, 'cluster_id': 1}, '41': {'quality': 1.282540205315461, 'cluster_id': 1}, '42': {'quality': 2.015585394934561, 'cluster_id': 0}, '43': {'quality': 2.464004827721051, 'cluster_id': 0}, '44': {'quality': 1.33980633202097, 'cluster_id': 1}}, 'response': ({'click': 0, 'quality': 1.2272016322975663, 'cluster_id': 1}, {'click': 0, 'quality': 1.2925848895378007, 'cluster_id': 1})}\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "[0, 1]"
            ]
          },
          "execution_count": 66,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "static_agent = StaticAgent(ie_environment.observation_space,\n",
        "                           ie_environment.action_space)\n",
        "static_agent.step(reward, observation)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "height": 70
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 49,
          "status": "ok",
          "timestamp": 1568741349275,
          "user": {
            "displayName": "Martin Mladenov",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBGIUwOMhxCVcEq0Q5I9YhNVfoEjuuMOWzhNOPn=s64",
            "userId": "05569865531106976534"
          },
          "user_tz": 420
        },
        "id": "5H7wzqfD27Em",
        "outputId": "72fb909c-e8d9-4e5b-8e0d-856e8bde0e3a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'user': {'raw_observation': array([], dtype=float64), 'sufficient_statistics': {'impression_count': array([0, 2]), 'click_count': array([0, 0])}}, 'doc': {'30': {'quality': 2.489224450301943, 'cluster_id': 0}, '31': {'quality': 2.125926607579561, 'cluster_id': 0}, '32': {'quality': 1.27448138607991, 'cluster_id': 1}, '33': {'quality': 1.2179911236932994, 'cluster_id': 1}, '34': {'quality': 1.177703750911228, 'cluster_id': 1}, '35': {'quality': 2.079489146813576, 'cluster_id': 0}, '36': {'quality': 1.1416765236282371, 'cluster_id': 1}, '37': {'quality': 1.2052916542615082, 'cluster_id': 1}, '38': {'quality': 1.2424683972006194, 'cluster_id': 1}, '39': {'quality': 1.8727966807396805, 'cluster_id': 0}, '40': {'quality': 1.1964488835024119, 'cluster_id': 1}, '41': {'quality': 1.282540205315461, 'cluster_id': 1}, '42': {'quality': 2.015585394934561, 'cluster_id': 0}, '43': {'quality': 2.464004827721051, 'cluster_id': 0}, '44': {'quality': 1.33980633202097, 'cluster_id': 1}}, 'response': ({'click': 0, 'quality': 1.2272016322975663, 'cluster_id': 1}, {'click': 0, 'quality': 1.2925848895378007, 'cluster_id': 1})}\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "[0, 1]"
            ]
          },
          "execution_count": 67,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "cluster_static_agent = ClusterClickStatsLayer(StaticAgent,\n",
        "                                              ie_environment.observation_space,\n",
        "                                              ie_environment.action_space)\n",
        "cluster_static_agent.step(reward, observation)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AeyrpnNI3onL"
      },
      "source": [
        "Observe how the 'user' field of the observation dictionary (as printed from within the static agent's step function) now has a new key 'sufficient_statistics', whereas the old user observation (which is vacuous) went under the 'raw_observation' key. This is done to avoid naming conflicts."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_6FSXE_eqCMP"
      },
      "source": [
        "## AbstractClickBandit\n",
        "\n",
        "The ClusterClickStats layer takes care of computing the necessary sufficient statistics for exploration. To implement the actual bandit policy, RecSim offers an abstract bandit layer implementation. The *AbstractClickBandit* takes as input a list of base agents, which it treats as arms. It will then utilize one of a a few implemented bandit policies (UCB1, KL-UCB, ThompsonSampling) to mix the policies in a way that achieves sub-linear regret relative to the best policy (which is apriori unknown), subject to certain assumptions about the environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "k0vDml6hqXSS"
      },
      "outputs": [],
      "source": [
        "from recsim.agents.layers.abstract_click_bandit import AbstractClickBanditLayer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zYA94M_puOs1"
      },
      "source": [
        "To instantiate an abstract bandit, we must present a list of base agents. In our case, we will have one base agent for each cluster. That agent simply retrieves the documents of that cluster from the corpus and sorts them according to perceived quality."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DRRAhUTtun-S"
      },
      "outputs": [],
      "source": [
        "class GreedyClusterAgent(agent.AbstractEpisodicRecommenderAgent):\n",
        "  \"\"\"Simple agent sorting all documents of a topic according to quality.\"\"\"\n",
        "\n",
        "  def __init__(self, observation_space, action_space, cluster_id, **kwargs):\n",
        "    del observation_space\n",
        "    super(GreedyClusterAgent, self).__init__(action_space)\n",
        "    self._cluster_id = cluster_id\n",
        "\n",
        "  def step(self, reward, observation):\n",
        "    del reward\n",
        "    my_docs = []\n",
        "    my_doc_quality = []\n",
        "    for i, doc in enumerate(observation['doc'].values()):\n",
        "      if doc['cluster_id'] == self._cluster_id:\n",
        "        my_docs.append(i)\n",
        "        my_doc_quality.append(doc['quality'])\n",
        "    if not bool(my_docs):\n",
        "      return []\n",
        "    sorted_indices = np.argsort(my_doc_quality)[::-1]\n",
        "    return list(np.array(my_docs)[sorted_indices])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ph9XmH_1usdR"
      },
      "source": [
        "We will now instantiate one GreedyClusterAgent for each cluster."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "BQ7DO1yqus_f"
      },
      "outputs": [],
      "source": [
        "  num_topics = list(ie_environment.observation_space.spaces['doc']\n",
        "                    .spaces.values())[0].spaces['cluster_id'].n\n",
        "  base_agent_ctors = [\n",
        "      functools.partial(GreedyClusterAgent, cluster_id=i)\n",
        "      for i in range(num_topics)\n",
        "  ]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Lld6Y3KZvWVy"
      },
      "source": [
        "We can now instantiate our cluster bandit as a combination of ClusterClickStats, AbstractClickBandit, and GreedyClusterAgent:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HrgbdGUSvVcj"
      },
      "outputs": [],
      "source": [
        "bandit_ctor = functools.partial(AbstractClickBanditLayer,\n",
        "                                arm_base_agent_ctors=base_agent_ctors)\n",
        "cluster_bandit = ClusterClickStatsLayer(bandit_ctor,\n",
        "                                        ie_environment.observation_space,\n",
        "                                        ie_environment.action_space)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "R_HoVSVhww-n"
      },
      "source": [
        "Our ClusterBandit is ready to use!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "height": 66
        },
        "colab_type": "code",
        "executionInfo": {
          "elapsed": 25,
          "status": "ok",
          "timestamp": 1568741349500,
          "user": {
            "displayName": "Martin Mladenov",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBGIUwOMhxCVcEq0Q5I9YhNVfoEjuuMOWzhNOPn=s64",
            "userId": "05569865531106976534"
          },
          "user_tz": 420
        },
        "id": "8LVPU3ukw06u",
        "outputId": "4f9e9b4e-8ea0-4ca8-82d7-c2e16c27f9e0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cluster bandit slate 0:\n",
            "{'quality': 1.4686875120276195, 'cluster_id': 1}\n",
            "{'quality': 1.4226918183479484, 'cluster_id': 1}\n"
          ]
        }
      ],
      "source": [
        "observation0 = ie_environment.reset()\n",
        "slate = cluster_bandit.begin_episode(observation0)\n",
        "print(\"Cluster bandit slate 0:\")\n",
        "doc_list = list(observation0['doc'].values())\n",
        "for doc_position in slate:\n",
        "  print(doc_list[doc_position])"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "last_runtime": {
        "build_target": "",
        "kind": "local"
      },
      "name": "RecSim: Developing an Agent.ipynb",
      "provenance": [
        {
          "file_id": "1CnUxBncchgVq1tv1pXLFtNii4KT2bSnh",
          "timestamp": 1568927323331
        }
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}