Simulating_a_news_personalization_scenario_using_Contextual_Bandits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulating a news personalization scenario using Contextual Bandits\n",
    "\n",
    "In the Contextual Bandit(CB) introduction tutorial, we learnt about CB and different CB algorithms. In this tutorial we will simulate the scenario of personalizing news content on a site, using CB, to users. The goal is to maximize user engagement quantified by measuring click through rate (CTR).\n",
    "\n",
    "Let's recall that in a CB setting, a data point has four components,\n",
    "\n",
    "- Context\n",
    "- Action\n",
    "- Probability of choosing action\n",
    "- Reward/cost for chosen action\n",
    "\n",
    "In our simulator, we will need to generate a context, get an action/decision for the given context and also simulate generating a reward.\n",
    "\n",
    "In our simulator, our goal is to maximize reward (click through rate/CTR) or minimize loss (-CTR)\n",
    "\n",
    "- We have two website visitors: 'Tom' and 'Anna'\n",
    "- Each of them may visit the website either in the morning or in the afternoon\n",
    "\n",
    "The **context** is therefore (user, time_of_day)\n",
    "\n",
    "We have the option of recommending a variety of articles to Tom and Anna. Therefore, **actions** are the different choices of articles: \"politics\", \"sports\", \"music\", \"food\", \"finance\", \"health\", \"cheese\"\n",
    "\n",
    "The **reward** is whether they click on the article or not: 'click' or 'no click'\n",
    "\n",
    "Let's first start with importing the necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vowpalwabbit import pyvw\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import itertools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulate reward\n",
    "\n",
    "In the real world, we will have to learn Tom and Anna's preferences for articles as we observe their interactions. Since this is a simulation, we will have to define Tom and Anna's preference profile. The reward that we provide to the learner will follow this preference profile. Our hope is to see if the learner can take better and better decisions as we see more samples which in turn means we are maximizing the reward.\n",
    "\n",
    "We will also modify the reward function in a few different ways and see if the CB learner picks up the changes. We will compare the CTR with and without learning.\n",
    "\n",
    "VW optimizes to minimize **cost which is negative of reward**. Therefore, we will always pass negative of reward as cost to VW."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# VW tries to minimize loss/cost, therefore we will pass cost as -reward\n",
    "USER_LIKED_ARTICLE = -1.0\n",
    "USER_DISLIKED_ARTICLE = 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reward function below specifies that Tom likes politics in the morning and music in the afternoon whereas Anna likes sports in the morning and politics in the afternoon. It looks dense but we are just simulating our hypothetical world in the format of the feedback the learner understands: cost. If the learner recommends an article that aligns with the reward function, we give a positive reward. In our simulated world this is a click."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost(context,action):\n",
    "    if context['user'] == \"Tom\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'music':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE\n",
    "    elif context['user'] == \"Anna\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Understanding VW format\n",
    "\n",
    "There are some things we need to do to get our input into a format VW understands. This function handles converting from our context as a dictionary, list of articles and the cost if there is one into the text format VW understands.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function modifies (context, action, cost, probability) to VW friendly format\n",
    "def to_vw_example_format(context, actions, cb_label = None):\n",
    "    if cb_label is not None:\n",
    "        chosen_action, cost, prob = cb_label\n",
    "    example_string = \"\"\n",
    "    example_string += \"shared |User user={} time_of_day={}\\n\".format(context[\"user\"], context[\"time_of_day\"])\n",
    "    for action in actions:\n",
    "        if cb_label is not None and action == chosen_action:\n",
    "            example_string += \"0:{}:{} \".format(cost, prob)\n",
    "        example_string += \"|Action article={} \\n\".format(action)\n",
    "    #Strip the last newline\n",
    "    return example_string[:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand what's going on here let's go through an example. Here, it's the morning and the user is Tom. There are four possible articles. So in the VW format there is one line that starts with shared, this is the shared context, followed by four lines each corresponding to an article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = {\"user\":\"Tom\",\"time_of_day\":\"morning\"}\n",
    "actions = [\"politics\", \"sports\", \"music\", \"food\"]\n",
    "\n",
    "print(to_vw_example_format(context,actions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting a decision\n",
    "\n",
    "When we call VW we get a _pmf_, [probability mass function](https://en.wikipedia.org/wiki/Probability_mass_function), as the output. Since we are incorporating exploration into our strategy, VW will give us a list of probabilities over the set of actions. This means that the probability at a given index in the list corresponds to the likelihood of picking that specific action. In order to arrive at a decision/action, we will have to sample from this list.\n",
    "\n",
    "So, given a list `[0.7, 0.1, 0.1, 0.1]`, we would choose the first item with a 70% chance. `sample_custom_pmf` takes such a list and gives us the index it chose and what the probability of choosing that index was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_custom_pmf(pmf):\n",
    "    total = sum(pmf)\n",
    "    scale = 1/total\n",
    "    pmf = [x * scale for x in pmf]\n",
    "    draw = random.random()\n",
    "    sum_prob = 0.0\n",
    "    for index, prob in enumerate(pmf):\n",
    "        sum_prob += prob\n",
    "        if(sum_prob > draw):\n",
    "            return index, prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have all of the information we need to choose an action for a specific user and context. To use VW to achieve this, we will do the following:\n",
    "\n",
    "1. We convert our context and actions into the text format we need\n",
    "2. We pass this example to vw and get the pmf out\n",
    "3. Now, we sample this pmf to get what article we will end up showing\n",
    "4. Finally we return the article chosen, and the probability of choosing it (we are going to need the probability when we learn form this example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_action(vw, context, actions):\n",
    "    vw_text_example = to_vw_example_format(context,actions)\n",
    "    pmf = vw.predict(vw_text_example)\n",
    "    chosen_action_index, prob = sample_custom_pmf(pmf)\n",
    "    return actions[chosen_action_index], prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Simulation set up\n",
    "\n",
    "Now that we have done all of the setup work and know how to interface with VW, let's simulate the world of Tom and Anna. The scenario is they go to a website and are shown an article. Remember that the reward function allows us to define the worlds reaction to what VW recommends.\n",
    "\n",
    "\n",
    "We will choose between Tom and Anna uniformly at random and also choose their time of visit uniformly at random. You can think of this as us tossing a coin to choose between Tom and Anna (Anna if heads and Tom if tails) and another coin toss for choosing time of day.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "users = ['Tom', 'Anna']\n",
    "times_of_day = ['morning', 'afternoon']\n",
    "actions = [\"politics\", \"sports\", \"music\", \"food\", \"finance\", \"health\", \"camping\"]\n",
    "\n",
    "def choose_user(users):\n",
    "    return random.choice(users)\n",
    "\n",
    "def choose_time_of_day(times_of_day):\n",
    "    return random.choice(times_of_day)\n",
    "\n",
    "# display preference matrix\n",
    "def get_preference_matrix(cost_fun):\n",
    "    def expand_grid(data_dict):\n",
    "        rows = itertools.product(*data_dict.values())\n",
    "        return pd.DataFrame.from_records(rows, columns=data_dict.keys())\n",
    "\n",
    "    df = expand_grid({'users':users, 'times_of_day': times_of_day, 'actions': actions})\n",
    "    df['cost'] = df.apply(lambda r: cost_fun({'user': r[0], 'time_of_day': r[1]}, r[2]), axis=1)\n",
    "\n",
    "    return df.pivot_table(index=['users', 'times_of_day'], \n",
    "            columns='actions', \n",
    "            values='cost')\n",
    "\n",
    "get_preference_matrix(get_cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will instantiate a CB learner in VW and then simulate Tom and Anna's website visits `num_iterations` number of times. In each visit, we:\n",
    "\n",
    "1. Decide between Tom and Anna\n",
    "2. Decide time of day\n",
    "3. Pass context i.e. (user, time of day) to learner to get action i.e. article recommendation and probability of choosing action\n",
    "4. Receive reward i.e. see if user clicked or not. Remember that cost is just negative reward.\n",
    "5. Format context, action, probability, reward in VW format\n",
    "6. Learn from the example\n",
    "    - VW _reduces_ a CB problem to a cost sensitive multiclass classification problem.\n",
    "\n",
    "This is the same for every one of our simulations, so we define the process in the `run_simulation` function. The cost function must be supplied as this is essentially us simulating how the world works.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation(vw, num_iterations, users, times_of_day, actions, cost_function, do_learn = True):\n",
    "    cost_sum = 0.\n",
    "    ctr = []\n",
    "\n",
    "    for i in range(1, num_iterations+1):\n",
    "        # 1. In each simulation choose a user\n",
    "        user = choose_user(users)\n",
    "        # 2. Choose time of day for a given user\n",
    "        time_of_day = choose_time_of_day(times_of_day)\n",
    "\n",
    "        # 3. Pass context to vw to get an action\n",
    "        context = {'user': user, 'time_of_day': time_of_day}\n",
    "        action, prob = get_action(vw, context, actions)\n",
    "\n",
    "        # 4. Get cost of the action we chose\n",
    "        cost = cost_function(context, action)\n",
    "        cost_sum += cost\n",
    "\n",
    "        if do_learn:\n",
    "            # 5. Inform VW of what happened so we can learn from it\n",
    "            vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n",
    "            # 6. Learn\n",
    "            vw.learn(vw_format)\n",
    "\n",
    "        # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "        ctr.append(-1*cost_sum/i)\n",
    "\n",
    "    return ctr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We want to be able to visualize what is occurring, so we are going to plot the click through rate over each iteration of the simulation. If VW is showing actions the get rewards the ctr will be higher. Below is a little utility function to make showing the plot easier.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ctr(num_iterations, ctr):\n",
    "    plt.plot(range(1,num_iterations+1), ctr)\n",
    "    plt.xlabel('num_iterations', fontsize=14)\n",
    "    plt.ylabel('ctr', fontsize=14)\n",
    "    plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scenario 1\n",
    "\n",
    "We will use the first reward function `get_cost` and assume that Tom and Anna do not change their preferences over time and see what happens to user engagement as we learn. We will also see what happens when there is no learning. We will use the \"no learning\" case as our baseline to compare to.\n",
    "\n",
    "### With learning\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations = 5000\n",
    "ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost)\n",
    "\n",
    "plot_ctr(num_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aside: interactions\n",
    "You'll notice in the arguments we supply to VW, **we include `-q UA`**. This is telling VW to create additional features which are the features in the (U)ser namespace and (A)ction namespaces multiplied together. This allows us to learn the interaction between when certain actions are good in certain times of days and for particular users. If we didn't do that, the learning wouldn't really work. We can see that in action below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate learner in VW but without -q\n",
    "vw = pyvw.vw(\"--cb_explore_adf --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations = 5000\n",
    "ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost)\n",
    "\n",
    "plot_ctr(num_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Without learning\n",
    "Let's do the same thing again (with `-q`) but with do_learn set to False to show the effect if we don't learn from the rewards. The ctr never improves are we just hover around 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations = 5000\n",
    "ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost, do_learn=False)\n",
    "\n",
    "plot_ctr(num_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scenario 2\n",
    "\n",
    "In the real world people's preferences change over time. So now in the simulation we are going to incorporate two different cost functions, and swap over to the second one halfway through. Below is a a table of the new reward function we are going to use, `get_cost_1`:\n",
    "\n",
    "### Tom\n",
    "\n",
    "| | `get_cost` | `get_cost_new1` |\n",
    "|:---|:---:|:---:|\n",
    "| **Morning** | Politics | Politics |\n",
    "| **Afternoon** | Music | Sports |\n",
    "\n",
    "### Anna\n",
    "\n",
    "| | `get_cost` | `get_cost_new1`  |\n",
    "|:---|:---:|:---:|\n",
    "| **Morning** | Sports | Sports |\n",
    "| **Afternoon** | Politics | Sports |\n",
    "\n",
    "This reward function is still working with actions that the learner has seen previously.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost_new1(context,action):\n",
    "    if context['user'] == \"Tom\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE\n",
    "    elif context['user'] == \"Anna\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE\n",
    "        \n",
    "get_preference_matrix(get_cost_new1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make it easy to show the effect of the cost function changing we are going to modify the `run_simulation` function. It is a little less readable now, but it supports accepting a list of cost functions and it will operate over each cost function in turn. This is perfect for what we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation_multiple_cost_functions(vw, num_iterations, users, times_of_day, actions, cost_functions, do_learn = True):\n",
    "    cost_sum = 0.\n",
    "    ctr = []\n",
    "\n",
    "    start_counter = 1\n",
    "    end_counter = start_counter + num_iterations\n",
    "    for cost_function in cost_functions:\n",
    "        for i in range(start_counter, end_counter):\n",
    "            # 1. in each simulation choose a user\n",
    "            user = choose_user(users)\n",
    "            # 2. choose time of day for a given user\n",
    "            time_of_day = choose_time_of_day(times_of_day)\n",
    "\n",
    "            # Construct context based on chosen user and time of day\n",
    "            context = {'user': user, 'time_of_day': time_of_day}\n",
    "\n",
    "            # 3. Use the get_action function we defined earlier\n",
    "            action, prob = get_action(vw, context, actions)\n",
    "\n",
    "            # 4. Get cost of the action we chose\n",
    "            cost = cost_function(context, action)\n",
    "            cost_sum += cost\n",
    "\n",
    "            if do_learn:\n",
    "                # 5. Inform VW of what happened so we can learn from it\n",
    "                vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n",
    "                # 6. Learn\n",
    "                vw.learn(vw_format)\n",
    "\n",
    "            # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "            ctr.append(-1*cost_sum/i)\n",
    "        start_counter = end_counter\n",
    "        end_counter = start_counter + num_iterations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation_multiple_cost_functions(vw, num_iterations, users, times_of_day, actions, cost_functions, do_learn = True):\n",
    "    cost_sum = 0.\n",
    "    ctr = []\n",
    "\n",
    "    start_counter = 1\n",
    "    end_counter = start_counter + num_iterations\n",
    "    for cost_function in cost_functions:\n",
    "        for i in range(start_counter, end_counter):\n",
    "            # 1. in each simulation choose a user\n",
    "            user = choose_user(users)\n",
    "            # 2. choose time of day for a given user\n",
    "            time_of_day = choose_time_of_day(times_of_day)\n",
    "\n",
    "            # Construct context based on chosen user and time of day\n",
    "            context = {'user': user, 'time_of_day': time_of_day}\n",
    "\n",
    "            # 3. Use the get_action function we defined earlier\n",
    "            action, prob = get_action(vw, context, actions)\n",
    "\n",
    "            # 4. Get cost of the action we chose\n",
    "            cost = cost_function(context, action)\n",
    "            cost_sum += cost\n",
    "\n",
    "            if do_learn:\n",
    "                # 5. Inform VW of what happened so we can learn from it\n",
    "                vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n",
    "                # 6. Learn\n",
    "                vw.learn(vw_format)\n",
    "\n",
    "            # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "            ctr.append(-1*cost_sum/i)\n",
    "        start_counter = end_counter\n",
    "        end_counter = start_counter + num_iterations\n",
    "\n",
    "    return ctr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With learning\n",
    "Let us now switch to the second reward function after a few samples (running the first reward function). Recall that this reward function changes the preferences of the web users but it is still working with the same action space as before. We should see the learner pick up these changes and optimize towards the new preferences.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use first reward function initially and then switch to second reward function\n",
    "\n",
    "# Instantiate learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new1]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions)\n",
    "\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** The initial spike in CTR depends on the rewards received for the first few examples. When you run on your own, you may see something different initially because our simulator is designed to have randomness.\n",
    "\n",
    "### Without learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do not learn\n",
    "# use first reward function initially and then switch to second reward function\n",
    "\n",
    "# Instantiate learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new1]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions, do_learn=False)\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scenario 3\n",
    "In this scenario we are going to start rewarding actions that have never seen a reward previously when we change the cost function.\n",
    "\n",
    "### Tom\n",
    "\n",
    "| | `get_cost` | `get_cost_new2` |\n",
    "|:---|:---:|:---:|\n",
    "| **Morning** | Politics |  Politics|\n",
    "| **Afternoon** | Music |   Food |\n",
    "\n",
    "### Anna\n",
    "\n",
    "| | `get_cost` | `get_cost_new2` |\n",
    "|:---|:---:|:---:|\n",
    "| **Morning** | Sports | Food|\n",
    "| **Afternoon** | Politics |  Food |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost_new2(context,action):\n",
    "    if context['user'] == \"Tom\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'food':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE\n",
    "    elif context['user'] == \"Anna\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'food':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'food':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### With learning\n",
    "Let us now switch to the third reward function after a few samples (running the first reward function). Recall that this reward function changes the preferences of the users and is working with a **different** action space than before. We should see the learner pick up these changes and optimize towards the new preferences\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use first reward function initially and then switch to third reward function\n",
    "\n",
    "# Instantiate learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new2]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions)\n",
    "\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Without Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do not learn\n",
    "# use first reward function initially and then switch to third reward function\n",
    "\n",
    "# Instantiate learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new2]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions, do_learn=False)\n",
    "\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This tutorial aimed at showcasing a real world scenario where contextual bandit algorithms can be used. We were able to take a context and set of actions and learn what actions worked best for a given context. We saw that the learner was able to respond rapidly to changes in the world.  We showed that allowing the learner to interact with the world resulted in higher rewards than the no learning baseline.\n",
    "\n",
    "This tutorial worked with simplistic features. VW supports high dimensional sparse features, different exploration algorithms and policy evaluation approaches."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}