cats_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulating a smart thermostat using CATS: a Contextual Bandit algorithm with a continuous action space\n",
    "\n",
    "In this tutorial we will simulate the scenario of personalizing a thermostat for a household with two rooms using Contextual Bandits in a continuous action space. The goal is to maximize user satisfaction with the thermostat quantified by measuring thermostat accuracy or reward (TR). The thermostat proposes a temperature and the user will either accept the temperature or adjust it to fit their needs.\n",
    "\n",
    "Let's recall that in a CB setting, a data point has four components,\n",
    "\n",
    "- Context\n",
    "- Chosen Action\n",
    "- Probability of chosen action\n",
    "- Reward/cost for chosen action\n",
    "\n",
    "In our simulator we will need to generate a context, get an action/decision for the given context, and also simulate generating a reward.\n",
    "\n",
    "The goal of the learning agent is to maximize the reward or to minimize the loss.\n",
    "\n",
    "The thermostat tracks two rooms: 'Living Room' and 'Bedroom'. \n",
    "Each room will need temperature adjustment either in the morning or in the afternoon. \n",
    "The context is therefore (room, time_of_day). \n",
    "\n",
    "In a continuous range we can't specify actions since there are infinite actions we can take across the continuous range. We do however provide the minimum value and the maximum value of the range. Here we will range between 0 degrees Celsius and 32 degrees Celsius using 1 degree increments which gives us a continuous range of 33 degrees.\n",
    "\n",
    "The reward is measured using the absolute difference between the proposed temperature and the one that was actually set by the people living in the house.\n",
    "\n",
    "Let's first start with importing the necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from vowpalwabbit import pyvw\n",
    "import pylibvw\n",
    "import random\n",
    "import math\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# VW minimizes loss/cost, therefore we will pass cost as -reward\n",
    "USER_LIKED_TEMPERATURE = -1.0\n",
    "USER_DISLIKED_TEMPERATURE = 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulate reward\n",
    "\n",
    "In the real world we will have to learn the room temperature preferences as we observe the interactions between the proposed temperature for each room and the one selected by the people living in the house. Since this is a simulation we will have to define the preference profile for each room. The reward that we provide to the learner will follow this preference profile. Our hope is to see if the learner can take better and better decisions as we see more samples which in turn means we are maximizing the reward.\n",
    "\n",
    "We will also modify the reward function in a few different ways and see if the CB learner picks up the changes. We will compare the TR with and without learning.\n",
    "\n",
    "VW minimizes the cost, which is defined as -reward. Therefore, we will pass the cost associated to each chosen action to VW.\n",
    "\n",
    "\n",
    "The reward function below specifies that we want the living room to be cold in the morning but warm in the afternoon. In reverse, we prefer the bedroom to be warm in the morning and cold in the afternoon. It looks dense but we are just simulating our hypothetical world in the format of the feedback the learner understands: cost. If the learner recommends a temperature that aligns with the reward function, we give a positive reward. Max reward is -1.0, min reward is 0 since VW learns in terms of cost, so we return a negative reward. In our simulated world this is the difference between the temperature recommended and the temperature chosen. If the difference is smaller than 5 degrees then we give a reward to the thermostat. This is a steep cost function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost(context, temperature, min_value, max_value):\n",
    "    range = float(max_value - min_value)\n",
    "    if context['room'] == \"Living Room\":\n",
    "        if context['time_of_day'] == \"morning\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        elif context['time_of_day'] == \"afternoon\":\n",
    "            selected_temperature = random.uniform(25, 29)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        else:\n",
    "            return USER_DISLIKED_TEMPERATURE\n",
    "    elif context['room'] == \"Bedroom\":\n",
    "        if context['time_of_day'] == \"morning\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(22, 29)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        elif context['time_of_day'] == \"afternoon\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        else:\n",
    "            return USER_DISLIKED_TEMPERATURE\n",
    "    else:\n",
    "        return USER_DISLIKED_TEMPERATURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function modifies (context, temperature (i.e. action), cost, probability) to VW friendly json format\n",
    "def to_vw_example_format(context, cats_label=None):\n",
    "    example_dict = {}\n",
    "    if cats_label is not None:\n",
    "        chosen_temp, cost, pdf_value = cats_label\n",
    "        example_dict['_label_ca'] = {'action' : chosen_temp, 'cost': cost, 'pdf_value': pdf_value}\n",
    "    example_dict['c'] = {'room={}'.format(context['room']): 1, 'time_of_day={}'.format(context['time_of_day']) : 1}\n",
    "    return json.dumps(example_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting a decision\n",
    "\n",
    "We call VW and get a predicted temperature and the value of the probability density function at that temperature. Since we are predicting over a continuous range VW will sample a pdf before returning the predicted value and the density of the pdf at that point. We are incorporating exploration into our strategy so the pdf will be more dense around the value that VW chooses to predict, and less dense in the rest of the continuous range. So it is more likely that VW will choose an action around the predicted value.\n",
    "\n",
    "We have all of the information we need to choose a temperature for a specific room and time of day. To use VW to achieve this, we will do the following:\n",
    "\n",
    "We convert our context into the json format we need. \n",
    "We pass this example to VW and get the chosen action and the probability of chosing that action. \n",
    "Finally we return the chosen temperature and the probability of choosing it (we are going to need the probability when we learn form this example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_temperature(vw, context):\n",
    "    vw_text_example = to_vw_example_format(context)\n",
    "    return vw.predict(vw_text_example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulation set up\n",
    "\n",
    "Now that we have done all of the setup work and know how to interact with VW, let's simulate the world of our two rooms. The scenario is that the thermostat it turned on in each room and it has to propose a temperature. Remember that the reward function allows us to define the worlds reaction to what VW recommends.\n",
    "\n",
    "We will choose between 'Living Room' and 'Bedroom' uniformly at random and also choose the time of day uniformly at random. We can think of this as tossing a coin to choose between the rooms ('Living Room' if heads and 'Bedroom' if tails) and another coin toss for choosing time of day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rooms = ['Living Room', 'Bedroom']\n",
    "times_of_day = ['morning', 'afternoon']\n",
    "\n",
    "def choose_room(rooms):\n",
    "    return random.choice(rooms)\n",
    "\n",
    "def choose_time_of_day(times_of_day):\n",
    "    return random.choice(times_of_day)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will instantiate a CB learner in VW and then simulate the thermostat interaction num_iterations number of times. In each interaction, we:\n",
    "\n",
    "1. Decide between 'Living Room' and 'Bedroom'\n",
    "2. Decide time of day\n",
    "3. Pass context i.e. (room, time of day) to learner to get a temperature i.e. a value between min (0 degrees) and max (32 degrees) and probability of choosing that temperature\n",
    "4. Receive reward i.e. see if the proposed temperature was adjusted or not, and by how much. Remember that cost is just negative reward.\n",
    "5. Format context, action (temperature), probability, and reward into VW format\n",
    "6. Learn from the example\n",
    "\n",
    "The above steps are repeatedly executed during our simulations, so we define the process in the run_simulation function. The cost function must be supplied as this is essentially us simulating how the world works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation(vw, num_iterations, rooms, times_of_day, cost_function, min_value, max_value, do_learn=True):\n",
    "\n",
    "    reward_rate = []\n",
    "    hits = 0\n",
    "    cost_sum = 0.\n",
    "\n",
    "    for i in range(1, num_iterations + 1):\n",
    "        # 1. In each simulation choose a room\n",
    "        room = choose_room(rooms)\n",
    "        # 2. Choose time of day for a given room\n",
    "        time_of_day = choose_time_of_day(times_of_day)\n",
    "        # 3. Pass context to vw to get a temperature\n",
    "        context = {'room': room, 'time_of_day': time_of_day}\n",
    "        temperature, pdf_value = predict_temperature(vw, context)\n",
    "        \n",
    "        # 4. Get cost of the action we chose\n",
    "        cost = cost_function(context, temperature, min_value, max_value)\n",
    "        if cost <= -0.75: # count something as a hit only if it has a high reward\n",
    "            hits += 1\n",
    "        cost_sum += cost\n",
    "\n",
    "        if do_learn:\n",
    "            # 5. Inform VW of what happened so we can learn from it\n",
    "            txt_ex = to_vw_example_format(context, cats_label=(temperature, cost, pdf_value))\n",
    "            vw_format = vw.parse(txt_ex, pyvw.vw.lContinuous)\n",
    "            # 6. Learn\n",
    "            vw.learn(vw_format)\n",
    "            # 7. Let VW know you're done with these objects\n",
    "            vw.finish_example(vw_format)\n",
    "\n",
    "        # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "        reward_rate.append(-1*cost_sum/i)\n",
    "\n",
    "    return reward_rate, hits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to be able to visualize what is occurring, so we are going to plot the reward rate over each iteration of the simulation. If VW is showing temperatures the that are close to what the simulated world wants, the reward will be higher. Below is a little utility function to make showing the plot easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_reward_rate(num_iterations, reward_rate, title):\n",
    "    plt.show()\n",
    "    plt.plot(range(1, num_iterations + 1), reward_rate)\n",
    "    plt.xlabel('num_iterations', fontsize=14)\n",
    "    plt.ylabel('reward rate', fontsize=14)\n",
    "    plt.title(title)\n",
    "    plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 1\n",
    "\n",
    "We will use the first reward function get_cost and assume that the preferences for room temperatures do not change over time and see what happens to the smart thermostat as we learn. We will also see what happens when there is no learning. We will use the \"no learning\" case as our baseline to compare to.\n",
    "\n",
    "We will be using the CATS algorithm which does tree based learning with smoothing. That means that we need to provide the number of actions (buckets/tree leaves) that the continuous range will be discretized into, and then we need to define the bandwidth which is the radius around the chosen discreet action that the algorithm will sample a temperature from with higher probability.\n",
    "\n",
    "For example, in our current range of 32 degrees celsius if we select the number of actions to be 8 that means that the algorithm will initially predict an action from the centre of one of 8 buckets: \n",
    "\n",
    "`(0 - 2 - 4), (4 - 6 - 8), (8 - 10 - 12), (12 - 14 - 16), (16 - 18 - 20), (20 - 22 - 24), (24 - 26 - 28), (28 - 30 - 32)`\n",
    "\n",
    "Let's say that for a given context, it selects the third bucket that starts from 8 degrees celsius, goes until 12 degrees celsius, and has a center of 10 degrees celsius. For a smoothing radius (bandwidth) of 1 the resulting probability density function (pdf) that VW will have to sample from will have a higher density around \n",
    "\n",
    "`[bucket_centre - bandwidth, bucket_centre + bandwidth]`\n",
    "\n",
    "i.e. \\[9, 11\\]. If bandwidth was bigger, for example 5 then we would have higher density (and therefore higher probability of selecting an action) in the range \\[5, 15\\], providing a smoothing range that spans the discretized buckets. The bandwidth is defined in terms of the continuous range (max_value - min_value)\n",
    "\n",
    "## With Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "num_iterations = 5000\n",
    "\n",
    "num_actions = 32\n",
    "bandwidth = 1\n",
    "\n",
    "# Instantiate VW learner\n",
    "vw = pyvw.vw(\"--cats \" + str(num_actions) + \"  --bandwidth \" + str(bandwidth) + \" --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \")\n",
    "ctr, hits = run_simulation(vw, num_iterations, rooms, times_of_day, get_cost, 0, 32, do_learn=True)\n",
    "vw.finish()\n",
    "plot_reward_rate(num_iterations, ctr, 'reward rate with num_actions = 32 and bandwidth = 1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Without Learning\n",
    "\n",
    "Let's do the same but without learning. The reward rate never improves and just hovers around 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "num_iterations = 5000\n",
    "\n",
    "num_actions = 32\n",
    "bandwidth = 1\n",
    "\n",
    "# Instantiate VW learner\n",
    "vw = pyvw.vw(\"--cats \" + str(num_actions) + \"  --bandwidth \" + str(bandwidth) + \" --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \")\n",
    "ctr, hits = run_simulation(vw, num_iterations, rooms, times_of_day, get_cost, 0, 32, do_learn=False)\n",
    "vw.finish()\n",
    "plot_reward_rate(num_iterations, ctr, 'click through rate with num_actions = 32 and bandwidth = 1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parameter sweep\n",
    "\n",
    "Next let's do a parameter sweep for different values of `num_actions` and `bandwidth`. We will use the below function to help us plot the reward rates for different combinations of `num_actions` and `bandwidths`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_reward_sweep(num_iterations, actions, bandwidths, data):\n",
    "    plt.show()\n",
    "    n_actions = len(actions)\n",
    "    n_bandwidths = len(bandwidths)\n",
    "    fig, axs = plt.subplots(n_actions, n_bandwidths)\n",
    "    for i in range(0, len(actions)):\n",
    "        for j in range(0, len(bandwidths)):\n",
    "            if bandwidths[j] >= actions[i]:\n",
    "                axs[i, j].set_title('NA')\n",
    "                continue\n",
    "            reward_rate, hits = data[str(actions[i])][str(bandwidths[j])]\n",
    "            hits_percentage = (hits/(num_iterations))*100\n",
    "            axs[i, j].plot(range(1, num_iterations + 1), reward_rate)\n",
    "            axs[i, j].set_title('hits {:.2f}% TR {:.2f}%'.format(hits_percentage, reward_rate[-1]*100))\n",
    "            axs[i, j].set_ylim([0,1])\n",
    "\n",
    "    for i, row in enumerate(axs):\n",
    "        for j, ax in enumerate(row):\n",
    "            ax.set_xlabel('b: ' + str(bandwidths[j%len(bandwidths)]), fontsize=14)\n",
    "            ax.set_ylabel('k: ' + str(actions[i%len(actions)]), fontsize=14)\n",
    "\n",
    "    fig.text(0.5, 0.04, 'num_iterations', ha='center', fontsize=14)\n",
    "    fig.text(0.04, 0.5, 'reward_rate', va='center', rotation='vertical', fontsize=14)\n",
    "    fig.set_figheight(18)\n",
    "    fig.set_figwidth(30)\n",
    "    plt.suptitle('#examples {}'.format(num_iterations))\n",
    "\n",
    "    # Hide x labels and tick labels for top plots and y ticks for right plots.\n",
    "    for ax in axs.flat:\n",
    "        ax.label_outer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With Learning\n",
    "\n",
    "We will try all the number of actions as powers of 2 from 8 until 2048. Since our continuous range stays the same (0-32) we are creating smaller range buckets as the number of actions grows. The number of actions needs to be a power of 2 as it represents the number of leaves that the internal binary tree will have. Small number of actions might result in coarser discretizaton leading to results similar to uniform random. On the other hand really large number of actions could mean that we need a lot more data in order to train all of the buckets.\n",
    "\n",
    "We will also try all the combinaitons of the above action numbers with bandwidths ranging from 0 to 25. The smaller the bandwidth the smaller the smoothing range around the selected continuous value. Really large bandwidths will result in large smoothing ranges and could lead to results similar to uniform random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# do parameter sweeping\n",
    "data = {}\n",
    "num_actions = [8, 32, 64, 128, 256, 512, 1024, 2048]\n",
    "bandwidths = [0, 1, 2, 3, 25]\n",
    "\n",
    "num_iterations = 5000\n",
    "\n",
    "for actions in num_actions:\n",
    "    for bd in bandwidths:\n",
    "        if str(actions) not in data:\n",
    "            data[str(actions)] = {}\n",
    "        if bd >= actions:\n",
    "            continue\n",
    "        print(\"Starting simulation for: --cats {} --bandwidth {} --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \".format(actions, bd))\n",
    "        vw = pyvw.vw(\"--cats \" + str(actions) + \"  --bandwidth \" + str(bd) + \" --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \")\n",
    "        rr, hits = run_simulation(vw, num_iterations, rooms, times_of_day, get_cost, 0, 32, do_learn=True)\n",
    "        vw.finish()\n",
    "\n",
    "        print(\"Done with simulation for num_actions: {} and bandwidth: {}\".format(actions, bd))\n",
    "        print()\n",
    "        data[str(actions)][str(bd)] = (rr, hits)\n",
    "\n",
    "print(\"Plotting...\")\n",
    "plot_reward_sweep(num_iterations, num_actions, bandwidths, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Without Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# do parameter sweeping\n",
    "data = {}\n",
    "num_actions = [8, 32, 64, 128, 256, 512, 1024, 2048]\n",
    "bandwidths = [0, 1, 2, 3, 25]\n",
    "\n",
    "num_iterations = 5000\n",
    "\n",
    "for actions in num_actions:\n",
    "    for bd in bandwidths:\n",
    "        if str(actions) not in data:\n",
    "            data[str(actions)] = {}\n",
    "        if bd >= actions:\n",
    "            continue\n",
    "        print(\"Starting simulation for: --cats {} --bandwidth {} --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \".format(actions, bd))\n",
    "        vw = pyvw.vw(\"--cats \" + str(actions) + \"  --bandwidth \" + str(bd) + \" --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \")\n",
    "        rr, hits = run_simulation(vw, num_iterations, rooms, times_of_day, get_cost, 0, 32, do_learn=False)\n",
    "        vw.finish()\n",
    "\n",
    "        print(\"Done with simulation for num_actions: {} and bandwidth: {}\".format(actions, bd))\n",
    "        print()\n",
    "        data[str(actions)][str(bd)] = (rr, hits)\n",
    "\n",
    "print(\"Plotting...\")\n",
    "plot_reward_sweep(num_iterations, num_actions, bandwidths, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 2\n",
    "\n",
    "In the real world peoples preferences change as e.g. the seasons change. So now in the simulation we are going to incorporate two different cost functions, and swap over to the second one halfway through. Below is a a table of the new cost function we are going to use, get_cost_1:\n",
    "\n",
    "### Living Room\n",
    "\n",
    " | | get_cost | get_cost_1 |\n",
    " |:---|:---:|:---:|\n",
    " | **Morning** | Cold | Hot |\n",
    " | **Afternoon** | Hot | Cold |\n",
    "\n",
    " \n",
    " ### Bedroom\n",
    "\n",
    " | | get_cost | get_cost_1 |\n",
    " |:---|:---:|:---:|\n",
    " | **Morning** | Hot | Cold |\n",
    " | **Afternoon** | Cold | Cold |\n",
    "\n",
    "\n",
    "Below we define the new cost function\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost_1(context, temperature, min_value, max_value):\n",
    "    range = float(max_value - min_value)\n",
    "    if context['room'] == \"Living Room\":\n",
    "        if context['time_of_day'] == \"morning\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(25, 29)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        elif context['time_of_day'] == \"afternoon\":\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        else:\n",
    "            return USER_DISLIKED_TEMPERATURE\n",
    "    elif context['room'] == \"Bedroom\":\n",
    "        if context['time_of_day'] == \"morning\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        elif context['time_of_day'] == \"afternoon\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            if math.fabs(selected_temperature - temperature) < 5.0:\n",
    "                return USER_LIKED_TEMPERATURE\n",
    "            else:\n",
    "                return USER_DISLIKED_TEMPERATURE\n",
    "        else:\n",
    "            return USER_DISLIKED_TEMPERATURE\n",
    "    else:\n",
    "        return USER_DISLIKED_TEMPERATURE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make it easy to show the effect of the cost function changing we are going to modify the run_simulation function. It is a little less readable now, but it supports accepting a list of cost functions and it will operate over each cost function in turn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation_multiple_cost_functions(vw, num_iterations, rooms, times_of_day, cost_functions, min_value, max_value, do_learn=True):\n",
    "\n",
    "    reward_rate = []\n",
    "    hits = 0\n",
    "    cost_sum = 0.\n",
    "\n",
    "    start_counter = 1\n",
    "    end_counter = start_counter + num_iterations\n",
    "    for cost_function in cost_functions:\n",
    "        for i in range(start_counter, end_counter):\n",
    "            # 1. In each simulation choose a room\n",
    "            room = choose_room(rooms)\n",
    "            # 2. Choose time of day for a given room\n",
    "            time_of_day = choose_time_of_day(times_of_day)\n",
    "            # 3. Pass context to vw to get a temperature\n",
    "            context = {'room': room, 'time_of_day': time_of_day}\n",
    "            temperature, pdf_value = predict_temperature(vw, context)\n",
    "            \n",
    "            # 4. Get cost of the action we chose\n",
    "            cost = cost_function(context, temperature, min_value, max_value)\n",
    "            if cost <= -0.75: # count something as a hit only if it has a high reward\n",
    "                hits += 1\n",
    "            cost_sum += cost\n",
    "\n",
    "            if do_learn:\n",
    "                # 5. Inform VW of what happened so we can learn from it\n",
    "                txt_ex = to_vw_example_format(context, cats_label=(temperature, cost, pdf_value))\n",
    "                vw_format = vw.parse(txt_ex, pyvw.vw.lContinuous)\n",
    "                # 6. Learn\n",
    "                vw.learn(vw_format)\n",
    "                # 7. Let VW know you're done with these objects\n",
    "                vw.finish_example(vw_format)\n",
    "\n",
    "            # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "            reward_rate.append(-1*cost_sum/i)\n",
    "        \n",
    "        start_counter = end_counter\n",
    "        end_counter = start_counter + num_iterations\n",
    "\n",
    "    return reward_rate, hits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With Learning\n",
    "\n",
    "Now that we have run a parameter sweep we can better pick the values of num_actions and bandwidth. For the next scenario we will pick `num_actions 128` and `bandwidth 2`.\n",
    "\n",
    "Let us now switch to the second cost function after a few samples (running the first cost function). Recall that this cost function changes the preferences of the room temperatures but it is still working with the same continuous action space as before. We should see the learner pick up these changes and optimize towards the new preferences.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# use first reward function initially and then switch to second reward function\n",
    "\n",
    "# Instantiate learner in VW\n",
    "num_actions = 128\n",
    "bandwidth = 2\n",
    "\n",
    "# Instantiate VW learner\n",
    "vw = pyvw.vw(\"--cats \" + str(num_actions) + \"  --bandwidth \" + str(bandwidth) + \" --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_1]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr, hits = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, rooms, times_of_day, cost_functions, 0, 32, do_learn=True)\n",
    "vw.finish()\n",
    "plot_reward_rate(total_iterations, ctr, 'reward rate with num_actions = 32 and bandwidth = 1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Without Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# use first reward function initially and then switch to second reward function\n",
    "\n",
    "# Instantiate learner in VW\n",
    "num_actions = 128\n",
    "bandwidth = 2\n",
    "\n",
    "# Instantiate VW learner\n",
    "vw = pyvw.vw(\"--cats \" + str(num_actions) + \"  --bandwidth \" + str(bandwidth) + \" --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_1]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr, hits = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, rooms, times_of_day, cost_functions, 0, 32, do_learn=False)\n",
    "vw.finish()\n",
    "plot_reward_rate(total_iterations, ctr, 'reward rate with num_actions = 32 and bandwidth = 1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 3\n",
    "\n",
    "## Better cost function\n",
    "\n",
    "The cost function we have been using until now has been a bit too simplistic but has served us well enough to showcase the differences in learning and also in showing CB pickup the new cost cost function and adjust to it.\n",
    "\n",
    "A slightly better cost function for our simulated world could be the difference between the temperature recommended and the temperature chosen. The smaller the difference the better the thermostat is doing. We are going to model that by taking the absolute cost: `1.0 - |selected_temperature - predicted_temerature| / range` and the transforming that cost into a reward by multiplying it with `-1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_smooth_cost(context, temperature, min_value, max_value):\n",
    "    range = float(max_value - min_value)\n",
    "    if context['room'] == \"Living Room\":\n",
    "        if context['time_of_day'] == \"morning\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(25, 29)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            cost = 1.0 - math.fabs(selected_temperature - temperature) / range\n",
    "            return -1.0 * cost\n",
    "        elif context['time_of_day'] == \"afternoon\":\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            cost = 1.0 - math.fabs(selected_temperature - temperature) / range\n",
    "            return -1.0 * cost\n",
    "        else:\n",
    "            return USER_DISLIKED_TEMPERATURE\n",
    "    elif context['room'] == \"Bedroom\":\n",
    "        if context['time_of_day'] == \"morning\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            cost = 1.0 - math.fabs(selected_temperature - temperature) / range\n",
    "            return -1.0 * cost\n",
    "        elif context['time_of_day'] == \"afternoon\":\n",
    "            # randomly pick a temperature in this range\n",
    "            selected_temperature = random.uniform(15, 18)\n",
    "            # the absolute difference between selected temperature and proposed temperature\n",
    "            cost = 1.0 - math.fabs(selected_temperature - temperature) / range\n",
    "            return -1.0 * cost\n",
    "        else:\n",
    "            return USER_DISLIKED_TEMPERATURE\n",
    "    else:\n",
    "        return USER_DISLIKED_TEMPERATURE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try the original paramter sweep with the new cost function `get_smooth_cost`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# do parameter sweeping\n",
    "data = {}\n",
    "num_actions = [8, 32, 64, 128, 256, 512, 1024, 2048]\n",
    "bandwidths = [0, 1, 2, 3, 25]\n",
    "\n",
    "num_iterations = 5000\n",
    "\n",
    "for actions in num_actions:\n",
    "    for bd in bandwidths:\n",
    "        if str(actions) not in data:\n",
    "            data[str(actions)] = {}\n",
    "        if bd >= actions:\n",
    "            continue\n",
    "        print(\"Starting simulation for: --cats {} --bandwidth {} --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \".format(actions, bd))\n",
    "        vw = pyvw.vw(\"--cats \" + str(actions) + \"  --bandwidth \" + str(bd) + \" --min_value 0 --max_value 32 --json --chain_hash --coin --epsilon 0.2 -q :: \")\n",
    "        rr, hits = run_simulation(vw, num_iterations, rooms, times_of_day, get_smooth_cost, 0, 32, do_learn=True)\n",
    "        vw.finish()\n",
    "\n",
    "        print(\"Done with simulation for num_actions: {} and bandwidth: {}\".format(actions, bd))\n",
    "        print()\n",
    "        data[str(actions)][str(bd)] = (rr, hits)\n",
    "\n",
    "print(\"Plotting...\")\n",
    "plot_reward_sweep(num_iterations, num_actions, bandwidths, data)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}