CB Simulation v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Personalizing news content using Contextual Bandits\n",
    "\n",
    "In the [Contextual Bandit(CB) introduction tutorial](https://vowpalwabbit.github.io/guides/contextual_bandits.html), we learnt about CB and different CB algorithms. In this tutorial we will simulate the scenario of personalizing news content on a site, using CB, to users. The goal is to maximize user engagement quantified by measuring click through rate (ctr).\n",
    "\n",
    "Let's recall that a CB setting has four components: \n",
    "\n",
    "- Context\n",
    "- Action\n",
    "- Probability of choosing action\n",
    "- Reward/cost for chosen action\n",
    "\n",
    "We want to maximize reward (ctr) or minimize loss (-ctr)\n",
    "\n",
    "- We have two website visitors: 'Tom' and 'Anna'\n",
    "- They visit the website at two different times in a day: 'morning' and 'afternoon'\n",
    "\n",
    "The **context** is therefore (user, time_of_day)\n",
    "\n",
    "We have the option of recommending a variety of articles to Tom and Anna. Therefore, **actions** are the different choices of articles: \"politics\", \"sports\", \"music\", \"food\", \"finance\", \"health\", \"cheese\"\n",
    "\n",
    "The **reward** is whether they click on the article or not: 'click' or 'no click'\n",
    "\n",
    "In the real world, we will have to learn Tom and Anna's preferences for articles i.e. the \"cost function\". Since this is a simulation we will define a cost function and see if the learner can optimize towards the cost function as we see more samples.\n",
    "\n",
    "We will also modify the cost function in a few different ways and see if the CB learner picks up the changes. We will compare the CTR with and without learning.\n",
    "\n",
    "Let's first start with importing the necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vowpalwabbit import pyvw\n",
    "import random\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reward function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# VW tries to minimize loss/cost, therefore we will pass cost as -reward\n",
    "USER_LIKED_ARTICLE = -1.0\n",
    "USER_DISLIKED_ARTICLE = 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reward function below specifies that Tom likes politics in the morning and music in the afternoon whereas Anna likes sports in the morning and politics in the afternoon. It looks dense but we are just simulating our hypothetical world in the format of the feedback the learner understands: cost. If the learner recommends an article that aligns with the reward function, we give a positive reward. In our simulated world this is a click."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost(context,action):\n",
    "    if context['user'] == \"Tom\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'music':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE\n",
    "    elif context['user'] == \"Anna\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding VW format\n",
    "\n",
    "There are some things we need to do to get our input into a format VW understands. This function handles converting from our context as a dictionary, list of articles and the cost if there is one into the text format VW understands. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function modifies (context, action, cost, probablity) to VW friendly format\n",
    "def to_vw_example_format(context, actions, cb_label = None):\n",
    "    if cb_label is not None:\n",
    "        chosen_action, cost, prob = cb_label\n",
    "    example_string = \"\"\n",
    "    example_string += \"shared |User user={} time_of_day={}\\n\".format(context[\"user\"], context[\"time_of_day\"])\n",
    "    for action in actions:\n",
    "        if cb_label is not None and action == chosen_action:\n",
    "            example_string += \"0:{}:{} \".format(cost, prob)\n",
    "        example_string += \"|Action article={} \\n\".format(action)\n",
    "    #Strip the last newline\n",
    "    return example_string[:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand what's going on here let's go through an example. Here, it's the morning and the user is Tom. There are four possible articles. So in the VW format there is one line that starts with shared, this is the shared context, followed by four lines each corresponding to an article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shared |User user=Tom time_of_day=morning\n",
      "|Action article=politics \n",
      "|Action article=sports \n",
      "|Action article=music \n",
      "|Action article=food \n"
     ]
    }
   ],
   "source": [
    "context = {\"user\":\"Tom\",\"time_of_day\":\"morning\"}\n",
    "actions = [\"politics\", \"sports\", \"music\", \"food\"]\n",
    "\n",
    "print(to_vw_example_format(context,actions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting an article/action recommendation\n",
    "\n",
    "When we call VW we get a _pmf_, or probability mass function, as the output. Since we are incorporating exploration into our strategy VW is giving us a list of probabilities of choosing each option. Our job here is to simply sample this list based on the probabilities.\n",
    "\n",
    "So, given a list `[0.7, 0.1, 0.1, 0.1]`, we would choose the first item with a 70% chance. `sample_custom_pmf` takes such a list and gives us the index it chose and what the probability of choosing that index was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function .....\n",
    "def sample_custom_pmf(pmf):\n",
    "    total = sum(pmf)\n",
    "    scale = 1/total\n",
    "    pmf = [x * scale for x in pmf]\n",
    "    draw = random.random()\n",
    "    sum_prob = 0.0\n",
    "    for index, prob in enumerate(pmf):\n",
    "        sum_prob += prob\n",
    "        if(sum_prob > draw):\n",
    "            return index, prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can actually get a prediction from VW!\n",
    "\n",
    "1. We convert our context and actions into the text format we need\n",
    "2. We pass this example to vw and get the pmf out\n",
    "3. Now, we sample this pmf to get what article we will end up showing\n",
    "4. Finally we return the article chosen, and the probability of choosing it (we are going to need the probability when we learn form this example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_action(vw, context, actions):\n",
    "    vw_text_example = to_vw_example_format(context,actions)\n",
    "    pmf = vw.predict(vw_text_example)\n",
    "    chosen_action_index, prob = sample_custom_pmf(pmf)\n",
    "    return actions[chosen_action_index], prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulation set up\n",
    "\n",
    "Now that we have done all of the setup work and know how to interface with VW, let's simulate the world of Tom and Anna. The scenario is they go to a website and are shown an article. Remember that the reward function allows us to define the worlds reaction to what VW recommends.\n",
    "\n",
    "\n",
    "We will choose between Tom and Anna uniformly at random and also choose their time of visit uniformly at random. You can think of this as us tossing a coin to choose between Tom and Anna (Anna if heads and Tom if tails) and another coin toss for choosing time of day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "users = ['Tom', 'Anna']\n",
    "times_of_day = ['morning', 'afternoon']\n",
    "actions = [\"politics\", \"sports\", \"music\", \"food\", \"finance\", \"health\", \"camping\"]\n",
    "\n",
    "def choose_user(users):\n",
    "    return random.choice(users)\n",
    "\n",
    "def choose_time_of_day(times_of_day):\n",
    "    return random.choice(times_of_day)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will initialize the CB learner in VW and then simulate their website visits `num_iterations` number of times. In each visit, we:\n",
    "\n",
    "1. Decide between Tom and Anna\n",
    "2. Decide time of day\n",
    "3. Pass context i.e. (user, time of day) to learner to get action i.e. article recommendation and probability of choosing action\n",
    "4. Receive reward i.e. see if user clicked or not. Remember that cost is just negative reward.\n",
    "5. Format context, action, probablity, reward in VW format\n",
    "6. Learn from the example\n",
    "\n",
    "This is the same for every one of our simulations, so we define the process in the `run_simulation` function. The cost function must be supplied as this is essentially us simulating how the world works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation(vw, num_iterations, users, times_of_day, actions, cost_function, should_learn = True):\n",
    "    cost_sum = 0.\n",
    "    ctr = []\n",
    "    \n",
    "    for i in range(1, num_iterations+1):\n",
    "        # 1. In each simulation choose a user \n",
    "        user = choose_user(users) \n",
    "        # 2. Choose time of day for a given user\n",
    "        time_of_day = choose_time_of_day(times_of_day)\n",
    "\n",
    "        # 3. Pass context to vw to get an action \n",
    "        context = {'user': user, 'time_of_day': time_of_day}\n",
    "        action, prob = get_action(vw, context, actions)\n",
    "        \n",
    "        # 4. Get cost of the action we chose\n",
    "        cost = cost_function(context, action)\n",
    "        cost_sum += cost\n",
    "        \n",
    "        if should_learn:\n",
    "            # 5. Inform VW of what happened so we can learn from it\n",
    "            vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n",
    "            # 6. Learn\n",
    "            vw.learn(vw_format)\n",
    "        \n",
    "        # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "        ctr.append(-1*cost_sum/i)\n",
    "        \n",
    "    return ctr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to be able to visualize what is occuring, so we are going to plot the click through rate over each iteration of the simulation. If VW is showing actions the get rewards the ctr will be higher. Below is a little utility function to make showing the plot easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ctr(num_iterations, ctr):\n",
    "    plt.plot(range(1,num_iterations+1), ctr)\n",
    "    plt.xlabel('num_iterations', fontsize=14)\n",
    "    plt.ylabel('ctr', fontsize=14)\n",
    "    plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 1\n",
    "\n",
    "We will use the first reward function `get_cost` and assume that Tom and Anna do not change their preferences over time and see what happens to user engagement as we learn.\n",
    "\n",
    "#### With learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcdZnv8c9TVV29dzr7DkkgAgmyRkRFYBSYMChwFVlmvOMoMzgLznDVmQnDiAo44jYOjDjIZbgoKriM4wSNsoiCIFvYkyBJJ2TpBNKddKeT3mt57h/ndKeql6Qb0qe6ur7v16teXWep08+v0qlvnd8553fM3REREekTK3QBIiIyvigYREQkj4JBRETyKBhERCSPgkFERPIoGEREJE9kwWBmd5hZk5mtGWa5mdnNZtZgZi+a2UlR1SYiIvtFucdwJ7D8AMvPBRaHjyuA/4igJhERGSCyYHD3R4CWA6xyAfAdDzwB1JvZ7GiqExGRPolCF5BjLrAtZ7oxnPfawBXN7AqCvQqqq6tPPvrooyMpUERkonjmmWd2ufv0oZaNp2CwIeYNOV6Hu98G3AawbNkyX7169VjWJSIy4ZjZluGWjaezkhqB+TnT84AdBapFRKRkjadgWAn8aXh20qlAm7sP6kYSEZGxFVlXkpndDZwJTDOzRuCzQBmAu98KrAL+CGgAOoGPRlWbiIjsF1kwuPtlB1nuwN9EVI6IiAxjPHUliYjIOKBgEBGRPAoGERHJo2AQEZE8CgYREcmjYBARkTwKBhERyaNgEBGRPAoGERHJo2AQEZE8CgYREcmjYBARkTwKBhERyaNgEBGRPAoGERHJo2AQEZE8CgYREcmjYBARkTwKBhERyaNgEBGRPAoGERHJo2AQEZE8CgYREcmjYBARkTwKBhERyaNgEBGRPAoGERHJo2AQEZE8CgYREcmjYBARkTwKBhERyaNgEBGRPAoGERHJo2AQEZE8iUIXICIy3mSzTktnL4mY4Q496SypTJapNUmqkvkfm5msEzNIZZxkYv937VQmS2dvhr1dKdp70uztStHa2cuezhTN+3p4bW836UyWyrI4M+oqqE7G6U5nybpjGB09acygtiLB7vZe4jGjtqKMmEFHb4ZMNsvZS2Zxwvz6Q97+SIPBzJYDNwFx4HZ3v3HA8sOAbwP14Tor3H1VlDWKSPHr6Emzc283ZsbOvd3s606TyWbJOv0fzp29aXpSWVo6e6lKxmntTNHenWZbSyeNrV30ZrKDtmsGdRVldKUylCdixGNGW1eKmBmZrFNbniCddVKZLOmsH7DGqdVJzIzuVIb2nvSg5TGDvk0k4zHSYf19EjFjbn1VcQeDmcWBW4CzgUbgaTNb6e7rclb7Z+CH7v4fZrYEWAUsiKpGERk77s72PV3935ib9nUzqbKMvd1ptu7upLM3w9SaJK+3dfNaWzfuTmtnL3u702SyTldvhkmVwYdyJutUJePEYxZ8K+9O0ZvOEjejO50hlTnwh3KfirIYVckE6UyW+qokkyrLOGZ2HWcvncmM2gosXC+ZiJGIGY2tXbR09lKdjJPKOOlslvrKJKls8M2/paOXZDxGMhGjoixOZVmcSZVl1FQkqKsoo74qeEyrKaeiLN5fR1tninQ2G/6eGI5TkYjT3hsERm15AnfoTGXIulOTTBCL2RAtOjSi3GM4BWhw900AZnYPcAGQGwwO1IXPJwE7IqxPREagO5VhX3eaLbs7SGWceMxIZbLs7uilaW83rZ299KazdPRmaGnvZU9X8A29sbVryG/GAPGYEY8ZveksdRUJZk2qwB2m1iRZPKOGeMwwMzp70lQm48TCb9pZh6pk8OEbDz8oy8ti1FcmmVqTBIc59ZXUViRIxI2YGZOrktRXBesnwu0W2qSqsiHn11Xsn28GNeXRfGRHGQxzgW05043A2wes8zngfjP7BFANnDXUhszsCuAKgMMOO+yQFypSqto6U+xo6+K1ti42NXewu6OXbS2dNO3roTuVYXv4jdkP8oW8LG5UlsWZXltOTUUZc+srefvCKSyeWcu0miTTasqZUVvB3u4UNeUJ5tRX9vfTVybjB964jLkog2GoWB7453UZcKe7f83M3gHcZWbHunteZ5+73wbcBrBs2bKR7TOKlKiWjl72dafY05lia0snU6qT7NjTxYuNbTQ0tZNMxGje10Nzew/N+3ryXhszmDu5kll1FUE3y6w6ZtaVU1dZxqLp1STjcVKZLOWJGFNryplRW059VRnuvKGujoQyYVyIMhgagfk50/MY3FV0ObAcwN0fN7MKYBrQFEmFIuNcW1eKF7bt4alXW3h9bzf1YZ97bzrL1pZONja3k8560E/tzp6uFJlhDoLWlCdYMK2KdKcza1IFS+bUccT0GuZNrmTWpAqOmF5DfWXZG/qAHwe9M/ImRBkMTwOLzWwhsB24FPjjAetsBd4L3GlmxwAVQHOENYocMulMlkQ8RiqT5ZXX99GTzrJ4Zg2pdHCQsTedpaWjl/lTqsIP8+AMmqZ9PWzY2c6OPV2Ywe6OXjp60jy/bQ9bdncCQZ98VTJOR0+asniMqmScw6ZW8+7F00nGg1MmHWdKdTlTqsuoLk9QX5lk7uRK9nWnmD2pgoXTavr75UVyRRYM7p42syuB+whORb3D3dea2XXAandfCXwK+L9m9n8Iupn+zP1gvZkiYyudyZLKOOt37qMyGefI6TXEYoa7096TZu2OvWxt6SSTdZr29rDutTae2dLKrvZe6ioStPekOciZi8Myg4pEnOryBCcdVs/Fy+Zz3LxJnHTYZKrLE7j7uDh4KhNLpNcxhNckrBow79qc5+uAd0VZk5SmTNZp2tfN623d/ac/ugfnv/dmsqzZ3sbujl62t3bxQuMeulP557TXViRIZbKD5gPMnlTBO46YxtTqJI2tXSycVsXSOZOIx4wNTe1UJeNkst5/gHZrSyd1FWWks87Mugqm15ZzxPRq5tRXknUfdEFVLoWCjAVd+Swlo6MnzYMv7+SR9bv41e93sqczNey6ZlCdTDCjtpzlS2cxp76So2bVsru9lw1N7fSkMlSVx5lSlWTJnEksmFYFwKy6CuqrklE1SWRMKBhkQnF31mzfy+bdHcRjxvbWLn7bsIvntrSyLzyHvjwR45ylszhmdi3TaspJxIJv7hXJOHhwHvyJ8yfrtEkpWQoGKUo793bz8CvNbGxuZ8vuTnZ39LC1pZPd7b2DhiKYW1/JO4+cyoKp1bz3mJksO3zymF41KlLsFAwyrrV1pujJZLhvzeu8tL2N1s4Um5rb2djc0b/O5KoyJlclOW5ePTPrylk8o5YjZ9TQm85y+NQqFk2vKWALRIqPgkHGjdfaumjtSPGrl3fy9JZWXmzcM+g4wIzaco6YXsN5b53N2UtmsXhmTd6YMyLy5ikYpKDaulLc89RWfrthF4827OqfXxY3Fk2r4dg5k1g6t47j59Vz4mH1zJ5UWcBqRUqDgkEil8k6dzz6Ko9saOa3G4IwSMSMs5fMZOmcOs5ZMoslc+oOshURGSsKBonMmu1t/MV3VvNaW3f/vPPeOpuLls3jhHn1TK7WaZ4i44GCQcaMu/Pw+mb+/aEGntnS2j/fDD5++hFcdPJcjpxRW8AKRWQoCgZ5w+5f+zo3/uL3JOLGsXMmcdriaZx8+GRufXgT8yZX8tPntrOhqb1//VMWTOGmy07QcQKRcU7BICOyq72Hmx7cwF1PbBly+fqd7fzkue2D5leWxfn6JSdwzpKZunZApEgoGGRYG5vbebFxD9f+dG3/VcO5Tlk4hfcdN5t4zPjGQw281tZNdTLOZ9+/lIbmdj508jwWz1RXkUixUTDIIL9r2MUf3/7koPnLl87iaxcfT/UQtxf8k7cfHkVpIhIBBYP0605luOqe5/nl2tf75y2eUcP1Fx7LqYumFrAyEYmSgkHY3d7Dx+58mhca2/rnPbbiPcyt10FikVKkYChh7s7n713Hnb/b3D/v42cs4upzjylcUSJScAqGEnXdveu447FX+6fPXjKTr1x0nO4lICIKhlLT0ZPm5oc29IfCB06ay1cvOl6nkopIPwVDibjzsVf53L3r+qdPPKyeu//iVI1MKiKDKBhKwNU/eYm7n9raPz2lOsk9V5xKeUKhICKDKRgmuJt/tYG7n9rKtJpyHv3HP9AegogclIJhAspmnf94eCNfue8VINhDUCiIyEgpGCaQ7lSGS771eN71CAAPfeoMhYKIjJiCYYLY2NzOhbc8xr7u/WMa/fHbD+MLFx6Lmc44EpGRUzBMAD9avY2///GLANRVJPjs+5fywZPnFbgqESlWCoYi95mfrukfCvv84+fwtYuPpyweK3BVIlLMFAxF7O6ntvaHws2Xncj5x88pcEUiMhEoGIrU7b/dxA0/fxmAlz53DrUVZQWuSEQmCgVDEfrkD57vv1vafVedrlAQkUNKndFF5tmtrf2hcP0FSzlqlu6QJiKHlvYYikg26/xjePbRg588nSNnKBRE5NBTMBSJ1o5eTrz+AQC+9MG3KhREZMwoGMa5bNZZ9E+r8uZ96OT5BapGREqBjjGMY+75oVCVjLP5xvN07wQRGVORBoOZLTezV8yswcxWDLPOxWa2zszWmtn3o6xvPNm5t5uFV+8Phd+teA/rrltewIpEpFRE1pVkZnHgFuBsoBF42sxWuvu6nHUWA1cD73L3VjObEVV948mjG3bx4f98sn/6xc+dQ51OSRWRiES5x3AK0ODum9y9F7gHuGDAOn8B3OLurQDu3hRhfeNCbzqbFwrPfeZshYKIRCrKYJgLbMuZbgzn5XoL8BYze8zMnjCzIftOzOwKM1ttZqubm5vHqNzCuPeFHQBcdsphbL7xPCZXJwtckYiUmiiDYagjpj5gOgEsBs4ELgNuN7P6QS9yv83dl7n7sunTpx/yQgvl2a2tfOpHLwDwhQuPLXA1IlKqogyGRiD3PMt5wI4h1vkfd0+5+6vAKwRBMeFls84Hvvk7AK46a7HOPBKRgokyGJ4GFpvZQjNLApcCKwes81PgDwDMbBpB19KmCGssmO8+GYySOquugqvOekuBqxGRUhZZMLh7GrgSuA94Gfihu681s+vM7PxwtfuA3Wa2Dvg18PfuvjuqGgtp1UuvAfDYivcUuBIRKXWRXvns7quAVQPmXZvz3IFPho+S8eC6nTyxqYW/OvMI4upCEpEC05XPBfbohl38+XdWA/DRdy4obDEiIigYCsrd+69ZmFaTZEZdRYErEhFRMBTUC41tABw1s5bV/3x2gasREQkoGAqkN53lwlseA+CHf/mOAlcjIrKfgqFAbvh5/xBRTKrUkBciMn4oGAogncnynceD6xbW33BugasREcmnYIjYQ7/fyZHX/AKAD5w4l2RC/wQiMr7oDm4RyGSdv73nOdKZLPet3dk//++XH1XAqkREhqZgiMDbvvAgLR29efPW33Cu9hZEZFwa0SeTmZWZ2etmtnSsC5po3D0vFGorEmy+8TyFgoiMWyPaY3D3lJmlGDxMthzEU6+2APChk+ex4tyjmVpTXuCKREQObDRfW/8duNrM1P00Cn/9vWcB+PN3L1IoiEhRGM2H/LuBM4DtZrYG6Mhd6O7nD/mqEubu7A67kY6aVVvgakRERmY0wbAL+K+xKmQi2r6nC4CPn76owJWIiIzciIPB3T86loVMRM9saQXg/BPmFLgSEZGRG/ExBjN7aKj7L5tZnZk9dGjLmhie3txCTXmCo2fVFboUEZERG01X0plAcoj5FQTHHyTHpuZ2vvvEVgDdfEdEispBg8HMTsqZPM7MWnKm48AfAtsPdWHF7gertwFw8uGTC1yJiMjojGSPYTXB9QsO3D/E8i7gE4eyqIngWw9vAuDHGlJbRIrMSIJhIWDAJuBtBGcn9ekFmtw9Mwa1Fa293an+52bqRhKR4nLQYHD3LQBmdiPwNne/NXe5mf2lmc1198+MUY1F54mNuwG46qzFBa5ERGT0RnPl84eB54aY/yzwp4emnIlh/c59AHzstIUFrkREZPRGEwwzgOYh5u8CZh6aciaGr96/HoC6Ct2ZTUSKz2iCYStDn5Z6OtB4aMopfqlMttAliIi8KaO5juFbwNfNLAn0XdD2XuCLwJcOdWHFqqGpHYCbLj2hwJWIiLwxoxkS42tmNg24mf0XuvUCN7n7l8eiuGL00+eDSzqWztHVziJSnEY1hLa7X21mNwBLCE5hXefu7WNSWZHqu35h4bSaAlciIvLGjPreCu7eATw9BrUUvWx2/32MNAyGiBQr3V/yEOobZvviZfMKXImIyBunYDiE+q5fuORt8wtciYjIG6dgGEbT3m7aulJsa+lkwYqf87MXdxz0Net3Bodbjpyhu7WJSPHS/ZuH8L0nt3DNf69hclUZrZ3BuEdXfv853nfcgW+4s2HnPmbVVTCpUhe2iUjx0h7DAE37urnmv9cA9IfCSK1v2sfimTobSUSKm4JhgL1dw4dBd2r4QWSzWaehqZ3F6kYSkSKnYBhgzwH2Er77xJZhlzU0t9OdyvIW7TGISJGLNBjMbLmZvWJmDWa24gDrXWRmbmbLoqwPoKWjd9hldzz66rDLzvn6IwDMrq885DWJiEQpsmAwszhwC3AuwZXTl5nZkiHWqwX+Fngyqtpy7cnpSjr58Mn8+WkL+f31ywHY0dbNghU/55drXst7zerN++92esZbpkdTqIjIGIlyj+EUoMHdN7l7L3APcMEQ610PfBnojrC2fm1hV9L/OnEud11+Cv/8viVUlMXz1vnL7z6bN33RrY8D8L9PPTyaIkVExlCUwTAX2JYz3RjO62dmJwLz3f1nB9qQmV1hZqvNbHVz81C3iHjjbn14IwD/evHxVCX3n8372ffn79xksk46k2XBip/3z7v+wmMPaS0iIoUQ5XUMQw0e1D+4kJnFgK8Df3awDbn7bcBtAMuWLfODrD4qu8NjDAPv1fzRdy3ksYZdPPhyEwBH/NMq6ir2v31vnTvpUJYhIlIwUe4xNAK5Y0XMA3IvJ64FjgV+Y2abgVOBlVEegHY/cMbc/pG3cd9Vp/dP7+1OAzClOsm9nzhtTGsTEYlKlMHwNLDYzBaGN/u5FFjZt9Dd29x9mrsvcPcFwBPA+e6+OqoC+05VPXrW8NciHDXEsmc/c/aY1SQiErXIgsHd08CVwH3Ay8AP3X2tmV1nZudHVceBnHj9AwAsP3bWAddr+MK5/c9/8+kzx7IkEZHIRTpWkruvAlYNmHftMOueGUVNQN4BZIBntrQecP1EPMatHz6ZDTv3sWBa9ViWJiISOQ2iN4Tj5h38QPLyY2cddM9CRKQYlfyQGPu6Bw+B8cmzjypAJSIi40PJB8MXf/H7vOn3Hz9Ht+UUkZJW8sHQHp5y2udAo6uKiJSCkg+GzIBrFx5ef2ivpBYRKTYKhswhvXBaRKToKRgG7DHcoPGORKTElfzpqtns/mB46pr3MqO2ooDViIgUXsnvMRw/vx6AWz98skJBRAQFAzXlwU7TOxZNLXAlIiLjQ8kHQybsSoqV/DshIhIo+Y/DdBgMCSWDiAigYOBLvwyufNbVziIigZIPhj4JBYOICKBg6BdTMIiIAAoGTl00pdAliIiMKyUfDFNryjlium62IyLSp+SDIZ3JUhYv+bdBRKRfyX8ipjJOMlHyb4OISL+S/0RMZbI6I0lEJEfJB0NvOktCXUkiIv1K/hMxk3XK4tpjEBHpo2BwJ2YKBhGRPiUfDNmsazgMEZEcJR8MGXfi2mMQEelX8sGQzmiPQUQkV8kHQ9YVDCIiuUo+GDJZ1wB6IiI5FAxZHWMQEcmlYHDXlc8iIjlKPhiyWd2LQUQkV8kHg7qSRETylXwwpHXwWUQkT8kHQ1bHGERE8kQaDGa23MxeMbMGM1sxxPJPmtk6M3vRzH5lZoePdU0ZDYkhIpInsmAwszhwC3AusAS4zMyWDFjtOWCZux8H/Bj48ljXlclqED0RkVxR7jGcAjS4+yZ37wXuAS7IXcHdf+3uneHkE8C8sS4q2GMY698iIlI8ovxInAtsy5luDOcN53LgF0MtMLMrzGy1ma1ubm5+U0Vl3InHlAwiIn2i/EQcqr/Gh1zR7MPAMuArQy1399vcfZm7L5s+ffqbKiqrPQYRkTyJCH9XIzA/Z3oesGPgSmZ2FnANcIa794x1UWldxyAikifK78pPA4vNbKGZJYFLgZW5K5jZicC3gPPdvWmsC0plsgB09mbG+leJiBSNyILB3dPAlcB9wMvAD919rZldZ2bnh6t9BagBfmRmz5vZymE2d0j8cs3rANz+6Ktj+WtERIpKlF1JuPsqYNWAedfmPD8rynrS2WyUv05EpCiU9GHXMh11FhEZpKQ/GadUJQE4elZtgSsRERk/SjoYqsqDnrRPn3NUgSsRERk/SjoY3IPLKOJxna4qItKnpIMhG15ep7GSRET2K+lg6NtjUCyIiOxX2sEQ/tQeg4jIfiUdDNmwL0m3YxAR2a+0g6Fvl0HBICLSr6SDwenbY1AyiIj0Ke1g0FlJIiKDlHQwZPvOSlIuiIj0K+lg2L/HUNg6RETGk5IOhr49Bh19FhHZr6SDQXsMIiKDlXYw6KwkEZFBSjoY+u7To1wQEdmvpINhxU9eArTHICKSq6SDYVd7D6A9BhGRXCUdDH1MZyWJiPRTMAAxvQsiIv30kYiOMYiI5FIwoMvbRERyKRgA0x6DiEg/BQP7b/EpIiIKBgD2dqcLXYKIyLihYACOmF5d6BJERMaNRKELKKQ5kyp455HTqK9KFroUEZFxo+T3GHTYWUQkX0kHgw45i4gMVtrB4BonSURkoNIOBlzjJImIDFDawaA9BhGRQUo7GFAwiIgMVNLBEFAyiIjkijQYzGy5mb1iZg1mtmKI5eVm9oNw+ZNmtmAs69FIGCIig0UWDGYWB24BzgWWAJeZ2ZIBq10OtLr7kcDXgS+NVT2dvWn2dPaqK0lEZIAo9xhOARrcfZO79wL3ABcMWOcC4Nvh8x8D77UxGvr0rse3kM46ybh600REckU5JMZcYFvOdCPw9uHWcfe0mbUBU4FduSuZ2RXAFeFku5m98gZrmvZ52PX5N/jiIjWNAe9nCVCbS4PaPDqHD7cgymAY6pv/wF7+kayDu98G3PamCzJb7e7L3ux2ionaXBrU5tIwVm2Osh+lEZifMz0P2DHcOmaWACYBLZFUJyIiQLTB8DSw2MwWmlkSuBRYOWCdlcBHwucXAQ+57qIjIhKpyLqSwmMGVwL3AXHgDndfa2bXAavdfSXwn8BdZtZAsKdw6RiX9aa7o4qQ2lwa1ObSMCZtNn0hFxGRXDpXU0RE8igYREQkT8kGw8GG5ygmZnaHmTWZ2ZqceVPM7AEz2xD+nBzONzO7OWz3i2Z2Us5rPhKuv8HMPjLU7xoPzGy+mf3azF42s7Vm9nfh/Inc5goze8rMXgjb/Plw/sJw+JgN4XAyyXD+sMPLmNnV4fxXzOwPC9OikTOzuJk9Z2Y/C6cndJvNbLOZvWRmz5vZ6nBetH/b7l5yD4KD3xuBRUASeAFYUui63kR7TgdOAtbkzPsysCJ8vgL4Uvj8j4BfEFwzcirwZDh/CrAp/Dk5fD650G0bpr2zgZPC57XAeoJhViZymw2oCZ+XAU+GbfkhcGk4/1bgr8Lnfw3cGj6/FPhB+HxJ+PdeDiwM/x/EC92+g7T9k8D3gZ+F0xO6zcBmYNqAeZH+bZfqHsNIhucoGu7+CIOv98gdXuTbwIU587/jgSeAejObDfwh8IC7t7h7K/AAsHzsqx89d3/N3Z8Nn+8DXia4an4it9ndvT2cLAsfDryHYPgYGNzmoYaXuQC4x9173P1VoIHg/8O4ZGbzgPOA28NpY4K3eRiR/m2XajAMNTzH3ALVMlZmuvtrEHyQAjPC+cO1vSjfk7C74ESCb9ATus1hl8rzQBPBf/SNwB53T4er5NafN7wM0De8TFG1Gfg34B+AbDg9lYnfZgfuN7NnLBj+ByL+245ySIzxZERDb0xQw7W96N4TM6sB/gu4yt332vDjLU6INrt7BjjBzOqB/waOGWq18GfRt9nM3gc0ufszZnZm3+whVp0wbQ69y913mNkM4AEz+/0B1h2TNpfqHsNIhucodjvDXUrCn03h/OHaXlTviZmVEYTC99z9J+HsCd3mPu6+B/gNQZ9yvQXDx0B+/cMNL1NMbX4XcL6ZbSbo7n0PwR7ERG4z7r4j/NlE8AXgFCL+2y7VYBjJ8BzFLnd4kY8A/5Mz/0/DsxlOBdrCXdP7gHPMbHJ4xsM54bxxJ+w3/k/gZXf/15xFE7nN08M9BcysEjiL4NjKrwmGj4HBbR5qeJmVwKXhGTwLgcXAU9G0YnTc/Wp3n+fuCwj+jz7k7n/CBG6zmVWbWW3fc4K/yTVE/bdd6CPwhXoQHM1fT9BPe02h63mTbbkbeA1IEXxTuJygb/VXwIbw55RwXSO4YdJG4CVgWc52PkZwYK4B+Gih23WA9p5GsFv8IvB8+PijCd7m44DnwjavAa4N5y8i+JBrAH4ElIfzK8LphnD5opxtXRO+F68A5xa6bSNs/5nsPytpwrY5bNsL4WNt32dT1H/bGhJDRETylGpXkoiIDEPBICIieRQMIiKSR8EgIiJ5FAwiIpJHwSByEGa2wMzczMbVjebHa11S/BQMIge3jWBE1+cBzOzM8AN5WlQFmNlvzOwbB6pL5FAp1bGSREbMgzGKXh+LbZtZmbun3shrx7IuKW3aY5BxJfxm/E0z+xcz22XBDYi+amaxcPlmM/v0EK/5Rs70ZjO71szuNLN9ZrbNzC4xs3ozu8fM2sObl5wzwpr6u2zC0Vx/HS5qDuffGa5nZvYPZrbRzLosuNnKh4fYzmVm9pCZdQEfN7OpZna3mTWGr1trZh/Ned2dwBnA34Sv93Bbg7qSzOx0C25S021mO83s6+GwLyN6f8N1PmDBTV+6zKzFzB42s5kjea9kYlAwyHj0J0AaeCdwJXAVcMkot3EVwbAIJxHc2OXbBDd7WQWcADwCfNfMKka53W3AB8PnSwm6cv4unL6BYDiSvyG4OcwXgW+Z2XkDtvFF4JvhOj8lGMrhWeB94Vjab8EAAAL9SURBVDZvCl/33nD9vwMeB/5f+Ptmkz+kMgBmNpfgpi3PEQxFfjlwWfj7cg37/prZLIIB675NMHrr6cBdI3hfZCIp9NggeuiR+yAYNfTxAfMeAG4Pn28GPj3Ea76RM70ZuDtnuoZgbKWbc+YtCOctG0FNeesSjNvj5NxlC6gGuoB3D3jtvwGrBmznUyP4nff0tXmoNg5T1xcIxsWJ5azzZ0APUDXC9/ekcJuHF/pvQY/CPXSMQcajFwdM72D/jUlGvQ13bzezToJBxvrsDH+OdrvDWULwzf+XZpY7AFkZQVDlWp07YWZxgts1XkJwM5VyglvO/maUNRxD8KGfzZn3aLitI9n/nhzo/X0BeBBYY2b3h89/7O7No6xFipiCQcajgQdjnf3dnlkG34SkbITbSA2YhkPXndq3nfcDWw9SS8eA6U8DnyLoMnoJaAf+hdGHljH8zVhy5w/7/rp7Jjz2cirBUM2XA180szPc/YVR1iNFSsEgxaaZoI8dgPAYwdEE/epR6Q1/xnPmrSPosjnc3R8a5fZOA+5197ug/34TbwH2DPid8SFem2sdcLGZxXL2Gk4LX7txpMW4uxMc03jczK4jGP75EoK9CSkBCgYpNg8BHzOzlQQhcQ1D7zGMpS0E37LPM7N7gS5332dmXwW+Gn6wP0JwbONUIOvutx1ge+uBS8zsNGAX8AlgIflhtxk4JTwrqp3gzmQDfZPgQPI3zewmgrH9byQ4NtE5koaFN3s5i+CmLjsJDmLPJwgdKRE6K0mKzRcJwuF/gPsJ+tCfjbIAd98OfJbgYO9OoO9U2c8AnyPoGlpLcFD3g8CrB9nkDQRnUP2CIFA6gO8NWOerBN/81xEE4mHD1HUuwYf588AdBDdx+qdRNK+N4JaaPyO4KczXgOvd/buj2IYUOd2oR0RE8miPQURE8igYpOSZ2a3h1dBDPW4tdH0iUVNXkpQ8M5sB1A2zeK+7N0VZj0ihKRhERCSPupJERCSPgkFERPIoGEREJI+CQURE8vx/7IA9vq063FIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Initialize learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations = 5000\n",
    "ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost)\n",
    "\n",
    "plot_ctr(num_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Aside: interactions\n",
    "You'll notice in the arguments we supply to VW, we include `-q UA`. This is telling VW to create new features which are the features in the (U)ser namespace and (A)ction namespaces multiplied together. This allows us to learn the interaction between when certain actions are good in certain times of days and for particular users. If we didn't do that, the learning wouldn't really work. We can see that in action below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXQc5Znv8e+j7tYueZEXjG1sA2YxgbAowCUESIBgCAOTZQLcMFmGE3LvQCaZbAPhJmHIQshGQoABH5Ihw0wgZHeCMwQChAlhMzs2GIQXLGxsCduy9lZ3P/ePKsldUsu2jFUtqX+fc3TUVV1d/bxtuX9d71v9lrk7IiIi/cqKXYCIiIwtCgYREYlQMIiISISCQUREIhQMIiISoWAQEZGI2ILBzH5sZpvN7Plh7jczu87MmszsWTM7Oq7aRERkhziPGG4FFu/k/jOBheHPxcC/xVCTiIgMElswuPuDwJadbHIu8B8eeASYbGaz4qlORET6JYtdQJ7ZwPq85eZw3cbBG5rZxQRHFdTU1BxzyCGHxFKgiMhE8cQTT7S6+/RC942lYLAC6wrO1+HuS4AlAI2Njb58+fLRrEtEZMIxs3XD3TeWzkpqBubmLc8BNhSpFhGRkjWWgmEp8OHw7KTjgTZ3H9KNJCIioyu2riQzux04BZhmZs3AV4AUgLvfBCwDzgKagC7gY3HVJiIiO8QWDO5+wS7ud+CSmMoREZFhjKWuJBERGQMUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkYhYg8HMFpvZKjNrMrPLCty/n5ndb2ZPmdmzZnZWnPWJiEiMwWBmCeAG4ExgEXCBmS0atNn/A+5096OA84Eb46pPREQCcR4xHAs0uftqd08DdwDnDtrGgfrw9iRgQ4z1iYgI8QbDbGB93nJzuC7flcCFZtYMLAM+WWhHZnaxmS03s+UtLS2jUauISMmKMxiswDoftHwBcKu7zwHOAm4zsyE1uvsSd29098bp06ePQqkiIqUrzmBoBubmLc9haFfRRcCdAO7+MFAJTIulOhERAeINhseBhWa2wMzKCQaXlw7a5lXgVAAzO5QgGNRXJCISo9iCwd0zwKXA3cALBGcfrTCzq8zsnHCzzwIfN7NngNuBj7r74O4mEREZRck4n8zdlxEMKuev+3Le7ZXA2+OsSUREovTNZxERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISESswWBmi81slZk1mdllw2zzQTNbaWYrzOyncdYnIiKQjOuJzCwB3ACcDjQDj5vZUndfmbfNQuBy4O3uvtXMZsRVn4iIBOI8YjgWaHL31e6eBu4Azh20zceBG9x9K4C7b46xPhERId5gmA2sz1tuDtflOwg4yMweMrNHzGxxoR2Z2cVmttzMlre0tIxSuSIipSnOYLAC63zQchJYCJwCXADcYmaThzzIfYm7N7p74/Tp0/d6oSIipSzOYGgG5uYtzwE2FNjmt+7e5+5rgFUEQSEiIjGJMxgeBxaa2QIzKwfOB5YO2uY3wDsBzGwaQdfS6hhrFBEpebEFg7tngEuBu4EXgDvdfYWZXWVm54Sb3Q28YWYrgfuBz7v7G3HVKCIiYO6Du/nHl8bGRl++fHmxyxARGVfM7Al3byx0n775LCIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhG7FQxmljKz183ssNEuSEREimu3gsHd+4A+hk6TLSIiE8xIupJ+CFxuZrFdDlREROI3kjf5dwAnA6+Z2fNAZ/6d7n5OwUeJiMi4MpJgaAV+OVqFiIjI2LDbweDuHxvNQkREZGzY7TEGM7uv0PWXzazezO7bu2WJiEixjGTw+RSgvMD6SoLxBxERmQB22ZVkZkfnLR5hZlvylhPAGcBre7swEREpjt0ZY1hO8P0FB/5Y4P5u4JN7sygRESme3QmGBYABq4G3EZyd1C8NbHb37CjUNmas3LCd17Z1c/qimcUuRURk1O0yGNx9HYCZfRN4m7vflH+/mf0fM5vt7l8apRqL7qzr/geAtd98T5ErEREZfSMZfL4QeKrA+ieBD++dcsa2byx7odgliIiMupEEwwygpcD6VmBC9rH0ZrJ84rblA8tLHlxNNqfpokRkYhtJMLxK4dNSTwKa9045Y8sjq7dw94pNkXU6ahCRiW4kwXAzcK2ZfdzMDgh/Lga+CywZnfKK547HXuUjP35syPof/WUNAO6Ou9Pe08f8y+5i/mV3cclPn8RdRxQiMr6NZEqM75rZNOA6dnzRLQ38wN2/NRrFFdNlv3ousvzoF0/luG/8CYD5l90FwFmH78OfV+3oXbvr2Y1ksjlu/vvGgvt8qKmVnz2+nh+cfyRmRkt7L+09few/vXaUWiEiMnIjmkLb3S83s68BiwhOYV3p7h2jUtkYsubqszCzIeuXPff6kHV3r9iEu0e2b+3oZVptBR+65VEAlj6zATModHDxwwuO4pO3B2P8h86q58XXt/PclWdQW5Hk2K/fy+b2XlZ/4yzKyobWIyKyN9h47/pobGz05cuX73rDEeo/KoAdp6mu2NDGe677y5Btz2ucyzUfOCLyGIAnv3Q6n7rjKf7n5dYhj3mznvrS6UypKTRDiYjIrpnZE+5esHtDwVCAu7Pg8mUDy/nfX+jL5lh4xR8i2/ff/+SrW3nfjX/d6b4fu+JUTvn2A3Slg+8EVpcnOOGABq485zBOvOb+N137P592EP906oEFj3BERPrtLBh0NbYCOsM37Q8cM4dr3n9E5L5UooyXvnYmq1s7+Jsf/oXLzzx04L63zhky+WzEvyw+hBl1lTx/5Rnk3EkmomP/w32B7vW2Hh5qauW9R82mM53h8CsLzUwSuPbel7j23pdY9k/vYNG+9TutR0SkEB0xFHDHY69y2a+e48i5k/nNJW8f0WObNneQShiVqQQPNbVy2L6TOHBGLZu29zBrUuVe/STf3tPH9+55idMOnckJBzTw9PptvDfviOX6/30UZWYsPmwf1r7RybbuPqbXVvC7Zzfw/qPnMLO+csh4yGDpTA6A8uRITmATkbFOXUkjdM1/v8i/PfAKXzp7EReduGCv7jsO1/3pZb53z0sjesznzziYkw+aztk/HDqGAjC1ppx3HjyDz59xMD9fvp6eTJYXN7bT0Zshk3MOmlnLF844ROMeMuZ19GZ4rrmNjW3dpDM5su7sN7WaabUVpBJGT1+OZMLoSmcpT5Rx0My6CfnBSMEwQv2DyA9f/i5mTaraq/uOy/0vbuZjtz4e+/MeM28K5zXO5dyj9qUimYj9+aU43J1Vm9p5YeN25kypZp/6Sl58vZ2qVIJ5DdXMrK8ceHN1d7r7gu7anr4c2ZxTX5Vk47YeNm3voXlrN60dvWzv6SOTdeZPq2HWpErqKoOe71mTqjCDabUVbOlMs6UzzQsbt9Pa0cum7b10pbO0dacB2LS9l7buPnozWdKZHFs7+0hncyNqW0WyjIpkGdPrKpg9pZqGmnIqU2UDbZpaXc7k6hSbtvfS05elszfD1q4+5kypoq4yRUWyjOryBJOry5laU051eYL6yhTtvX2sae1k47Ye0tkcVakEk6pSZN3pzexYrkiWkXOnO52lJ5PFMDa395DO5Dhx4XQWTKvZo38zjTHsoYaaimKXsMfeecgMmr5+5pBxjH65nPP42i0snFlH89Yuzrn+IQCuu+AoTj1kBq0dvcxrqMHdWftGF7c9vI7tPX384ongS+6N86Zw4fHzSGdyHDN/Cjfe/wq/fLKZJ9Zt5Yl1W/nCL58F4JB96nilpYMj5kxmyd8fQ0NtxcDzt3T00lBTTpmZTr8dJdmck3iTr627s7Gth9e2dfPYmi3sO7mSGXWVNG3u4P5Vm1n3RhetHb2092R2up/y8G+xrCwIhHzDnb493PrhpBJGTUWSyVUpABpqK9hvajW1FUnKDGbWV1JfleLQWXXsN7WGRJmRShjr3uhi0/YeMjmnvjJFOpujMllGd1+WJ9dtpS/ntLb3sqGtm9UtHQNv/sOpLk8MnGAymq5+3+F7HAw7oyOGAhZcfhfumk11Tzz56la+9JvnWbFh+7DbpBJGX3bo392U6hSL3zKLT5y0P/MaqnVm1U7kck46m+Pp9dt4eXMHT63byoa2blo70nT1Zujuy9Ldl6WnL8f0ugr6X8ktnWkyOacqlaCmIgEYFcmygU/jyYSRywVjSk2bO4KTJMqM7cO86c9vqObgfeqorUhxxJxJHLf/VJo2d7ClM82B02vJurOmtZMN23roDT/tAkytSeEOZWXB82/pTDOvoZrq8iRvmT2J+sokU2vK6cs6r7f1sLm9h/aeDOlsjpb2XtydlvZeaiuT7DOpioNn1jGzvoJJVanY/m629wTB0NbVR1t3H1NryqmrTFKRTJBKGB29GTp6M8En/b4cW7vStHb0ks7kaOvuoyJZxvxpNcydUk1lKkFnOsMbHWnKk2VUhcvtPRnSmRwGVKYSVJWXYWZMqkpRmUpQW55kUnVqj+ofM11JZrYY+AHBld9ucfdvDrPdB4CfE0zzvdN3/b0dDBvbuvlfVweXsFYwvDnuzm2PrOPw2ZPo7M1y4Y8ejdxfV5Eknc3Rmxl6aJ8oM+ork5xwwDROXzSTMw/fZ9S6ptydvqzj+B4/R3tPH0+9uo2KZBkPr36D6vIE+0+r5bDZ9XSlszyxdiu9YRfGI6vfYFNbDw215cyeXM2M+grKE2VUphJs6ewl5/D69uCNcNP2HtZv6aKmIklnb4aeviypRBmb23sjz19mML+hhtlTqqhKJUiUGVXlCaZWl7OmtZNJVSl6szmm11ZgBp3h2FBXb3aga8XdBz7Jt/f2ccD0WiZVpehKZ1k4o5b9plZz5H6TaW1Ps7Urzb6Tqzhgeo0CfJwaE8FgZgngJeB0gkn3HgcucPeVg7arA+4imHbj0riD4bE1W/jgzQ8DCobR0JfN8drWbuYXOPx1d+56biPf/MOLNG/tjtxXXZ5g8Vv2oaY8Sdad6bUVnPe2uew7efgxoLauPrr6Mqxu6eSOx9eTzeUGugdbO3pZ09pJZSrBig1tA0cwdZVJplSXs2BaDQfOqOXgfepoqCmnM52lriLJ69t7WNPaSW9flte3B/28ZWY8+1obLYPerIdTV5mktiJJd1+WbcN0R1SlEuTcmTWpktlTquhOZ6mpSFJfmaK7L8vM+kpqyhMcNruehTPqWDizVmM6MiJjZYzhWKDJ3VeHRd0BnAusHLTdV4FvAZ+LsbYB/UF54oHTivH0E14qUVYwFADMjLOP2Jezj9h3YF06k+OLv36Ovza1suy5jZG+6evvb+KkhdOY11DDvS9s4rVt3TTUlNNQU0Eml+OVls5hngcmV6WYUl1Oe0+G4/dv4OCZdTiwuqWDjt4ML76+nT+/VGiW+f52BKckG1BdnuSA6TV85vSDSCXKOG7BVLr7sqx6vZ11b3RiZhy3YCo1FcF/t4Nm1g30+2dzzoZt3cGn93SGabXB0cOkqpTGXaRo4gyG2cD6vOVm4Lj8DczsKGCuu//ezIYNhnBW14sB9ttvv71a5Ia24JPqvyw+ZK/uV/ZMebKM7/zdWweWezNZ3KF5azc3//kVlj6zgftXtXDA9Bo+eMxc1m/t4o2ONOlsjhMPnMaifeuZUVfBmYfPYvZOji4K6ejNsOK1NrZ29VFflaSlvZeZ9ZUctd/k3fp0ftDMul1ukygz5k6tHlFdIqMtzmAo9PFnoB/LzMqAa4GP7mpH7r6EcKrvxsbGvdYX1tmb4Z9/9gwAk/dwQEdGV/8b8oEzavn2372Vr/7tW9i0vYd5DXv/zIzaiiTH7d+w1/crMtbF+a2NZmBu3vIcYEPech3wFuABM1sLHA8sNbPCc1iPgsO+cvfA7T0d6Zd4VaYSoxIKIqUszmB4HFhoZgvMrBw4H1jaf6e7t7n7NHef7+7zgUeAc3Y1+Ly3DL5kZ225vuIhIqUptmBw9wxwKXA38AJwp7uvMLOrzOycuOoYzj0rd1zC8+KT9tfAn4iUrFg/Frv7MmDZoHVfHmbbU+Koqd+/P7Rm4Pa2rnScTy0iMqZMvJmh9tCzzW0Dt5ev3VrESkREikvBEOqf1AvQmSgiUtIUDAVcdqa+wyAipUvBUEB9pc5IEpHSpWAY5JYPN2pSMBEpaQqGQU5bNLPYJYiIFJWCgR1fbjv1kBlFrkREpPgUDARTQQMcM39KkSsRESk+BQMMXKikfJjLYIqIlBK9EwJ94RXEUgoGEREFAzBw9S4Fg4iIggEIrhIGwUVhRERKnd4JgXQ2mA5DwSAiomAAoDejwWcRkX56JwTee8NfAajQEYOIiIKhaXP7jtNVFQwiIgqG79/78sBtBYOIiIIhMq6gMQYREQUDv3rqtYHbOmIQEVEwRCgYREQUDLw7b5ptdSWJiCgYItd31umqIiIKBrK53MBtdSWJiCgYyIQX6QFIqitJRETBkAlnVj1y7mSqU4kiVyMiUnzJYhdQbP1HDL/+xxMwsyJXIyJSfCV/xJDN5UiWmUJBRCRU8sGQyTmJMoWCiEi/kg+GbNZJKhhERAaUfDDoiEFEJErBkMvpNFURkTwl/46YzakrSUQkX8kHQ0ZjDCIiESX/PYZfPNmM+663ExEpFbEeMZjZYjNbZWZNZnZZgfs/Y2YrzexZM/uTmc0b7ZoUCiIiUbEFg5klgBuAM4FFwAVmtmjQZk8Bje5+BPAL4Ftx1SciIoE4jxiOBZrcfbW7p4E7gHPzN3D3+929K1x8BJgTY30iIkK8wTAbWJ+33ByuG85FwB8K3WFmF5vZcjNb3tLSshdLFBGROAefC536U7CH38wuBBqBkwvd7+5LgCUAjY2Nb2qUYH5DNW+dO/nN7EJEZEKJMxiagbl5y3OADYM3MrPTgCuAk929d7SLyrpTpgn0REQGxNmV9Diw0MwWmFk5cD6wNH8DMzsKuBk4x903x1FULgfKBRGRHWILBnfPAJcCdwMvAHe6+wozu8rMzgk3+zZQC/zczJ42s6XD7G5v1kVCySAiMiDWL7i5+zJg2aB1X867fVqc9XT2ZtjQ1sPja7fE+bQiImNaSU+JsWpTOwBr3+jaxZYiIqWjpIOhMqlrPIuIDFbawZAq6eaLiBRU0u+Mus6ziMhQJR0MuXAGvUP2qStyJSIiY0dJB4OHwXDJOw8sciUiImNHSQdDLpxMQz1KIiI7lHQw9F+LQVNiiIjsUNLB0D/GoCt7iojsoGBAZyeJiOQr6WBQV5KIyFAlHQzqShIRGarEgyH4rQMGEZEdSjoYXGMMIiJDlHQwPPhSKwBbO9NFrkREZOwo6WC49a9rAHjq1W1FrkREZOwo6WCoq0wBkMnlilyJiMjYUdLBcNGJCwD46AkLilyJiMjYUdLBUJUKLtRTU6EL9oiI9CvpYHD6v8egs5JERPqVdDDoewwiIkOVdDBoSgwRkaFKOhgGJtErch0iImNJSQeD999QMoiIDCjpYMA1+CwiMlhJB8PA4HNxyxARGVNKOhhcRwwiIkOUdDDodFURkaFKOhj6B59NnUkiIgNKOxj6T1ct6VdBRCSqpN8SXYPPIiJDlHYwaK4kEZEhSjoYNPgsIjJUSQfDjq4kJYOISL/SDoawK0lHDCIiO8QaDGa22MxWmVmTmV1W4P4KM/tZeP+jZjZ/NOtxdSWJiAwRWzCYWQK4ATgTWARcYGaLBm12EbDV3Q8ErgWuGa16utIZWtp7AQ0+i4jki/OI4Vigyd1Xu3sauAM4d9A25wI/CW//AjjVbHTetW97eB23/nUtoGAQEcmXjPG5ZgPr85abgeOG28bdM2bWBjQArfkbmdnFwMXhYoeZrdrDmqYBrclROy4Zk6Yx6PUsAWpzaVCbR2becHfEGQyFPpb7HmyDuy8BlrzpgsyWu3vjm93PeKI2lwa1uTSMVpvj7EpqBubmLc8BNgy3jZklgUnAlliqExERIN5geBxYaGYLzKwcOB9YOmibpcBHwtsfAO7z/gmNREQkFrF1JYVjBpcCdwMJ4MfuvsLMrgKWu/tS4EfAbWbWRHCkcP4ol/Wmu6PGIbW5NKjNpWFU2mz6QC4iIvlK+pvPIiIylIJBREQiSjYYdjU9x3hiZj82s81m9nzeuqlmdo+ZvRz+nhKuNzO7Lmz3s2Z2dN5jPhJu/7KZfaTQc40FZjbXzO43sxfMbIWZfSpcP5HbXGlmj5nZM2Gb/zVcvyCcPublcDqZ8nD9sNPLmNnl4fpVZnZGcVq0+8wsYWZPmdnvw+UJ3WYzW2tmz5nZ02a2PFwX79+2u5fcD8Hg9yvA/kA58AywqNh1vYn2nAQcDTyft+5bwGXh7cuAa8LbZwF/IPjOyPHAo+H6qcDq8PeU8PaUYrdtmPbOAo4Ob9cBLxFMszKR22xAbXg7BTwatuVO4Pxw/U3A/w1v/yNwU3j7fOBn4e1F4d97BbAg/H+QKHb7dtH2zwA/BX4fLk/oNgNrgWmD1sX6t12qRwy7Mz3HuOHuDzL0+x7504v8BPjbvPX/4YFHgMlmNgs4A7jH3be4+1bgHmDx6Fc/cu6+0d2fDG+3Ay8QfGt+IrfZ3b0jXEyFPw68i2D6GBja5kLTy5wL3OHuve6+Bmgi+P8wJpnZHOA9wC3hsjHB2zyMWP+2SzUYCk3PMbtItYyWme6+EYI3UmBGuH64to/L1yTsLjiK4BP0hG5z2KXyNLCZ4D/6K8A2d8+Em+TXH5leBuifXmZctRn4PvAFIBcuNzDx2+zAH83sCQum/4GY/7bjnBJjLNmtqTcmqOHaPu5eEzOrBX4JfNrdt9vwkyFOiDa7exY40swmA78GDi20Wfh73LfZzM4GNrv7E2Z2Sv/qAptOmDaH3u7uG8xsBnCPmb24k21Hpc2lesSwO9NzjHebwkNKwt+bw/XDtX1cvSZmliIIhf9y91+Fqyd0m/u5+zbgAYI+5ckWTB8D0fqHm15mPLX57cA5ZraWoLv3XQRHEBO5zbj7hvD3ZoIPAMcS8992qQbD7kzPMd7lTy/yEeC3ees/HJ7NcDzQFh6a3g2828ymhGc8vDtcN+aE/cY/Al5w9+/l3TWR2zw9PFLAzKqA0wjGVu4nmD4Ghra50PQyS4HzwzN4FgALgcfiacXIuPvl7j7H3ecT/B+9z90/xARus5nVmFld/22Cv8nniftvu9gj8MX6IRjNf4mgn/aKYtfzJttyO7AR6CP4pHARQd/qn4CXw99Tw22N4IJJrwDPAY15+/kHgoG5JuBjxW7XTtp7IsFh8bPA0+HPWRO8zUcAT4Vtfh74crh+f4I3uSbg50BFuL4yXG4K798/b19XhK/FKuDMYrdtN9t/CjvOSpqwbQ7b9kz4s6L/vSnuv21NiSEiIhGl2pUkIiLDUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDyC6Y2XwzczMbUxeaH6t1yfinYBDZtfUEM7o+DWBmp4RvyNPiKsDMHjCz63dWl8jeUqpzJYnsNg/mKHp9NPZtZil379uTx45mXVLadMQgY0r4yfhGM/uGmbVacAGi75hZWXj/WjP7XIHHXJ+3vNbMvmxmt5pZu5mtN7PzzGyymd1hZh3hxUvevZs1DXTZhLO53h/e1RKuvzXczszsC2b2ipl1W3CxlQsL7OcCM7vPzLqBT5hZg5ndbmbN4eNWmNnH8h53K3AycEn4eA/3NaQrycxOsuAiNT1mtsnMrg2nfdmt1zfc5n0WXPSl28y2mNmfzWzm7rxWMjEoGGQs+hCQAU4ALgU+DZw3wn18mmBahKMJLuzyE4KLvSwDjgQeBP7TzCpHuN/1wPvD24cRdOV8Klz+GsF0JJcQXBzmauBmM3vPoH1cDdwYbvMbgqkcngTODvf5g/Bxp4bbfwp4GPj38PlmEZ1SGQAzm01w0ZanCKYivwi4IHy+fMO+vma2D8GEdT8hmL31JOC23XhdZCIp9twg+tFP/g/BrKEPD1p3D3BLeHst8LkCj7k+b3ktcHveci3B3ErX5a2bH65r3I2aItsSzNvj5F1lC6gBuoF3DHrs94Flg/bz2d14zjv621yojcPU9XWCeXHK8rb5KNALVO/m63t0uM95xf5b0E/xfjTGIGPRs4OWN7DjwiQj3oe7d5hZF8EkY/02hb9Hut/hLCL45P/fZpY/AVmKIKjyLc9fMLMEweUazyO4mEoFwSVnHxhhDYcSvOnn8tb9JdzXgex4TXb2+j4D3As8b2Z/DG//wt1bRliLjGMKBhmLBg/GOju6PXMMvQhJajf30TdoGfZed2r/fv4GeHUXtXQOWv4c8FmCLqPngA7gG4w8tIzhL8aSv37Y19fds+HYy/EEUzVfBFxtZie7+zMjrEfGKQWDjDctBH3sAIRjBIcQ9KvHJR3+TuStW0nQZTPP3e8b4f5OBH7n7rfBwPUmDgK2DXrORIHH5lsJfNDMyvKOGk4MH/vK7hbj7k4wpvGwmV1FMP3zeQRHE1ICFAwy3twH/IOZLSUIiSsofMQwmtYRfMp+j5n9Duh293Yz+w7wnfCN/UGCsY3jgZy7L9nJ/l4CzjOzE4FW4JPAAqJhtxY4NjwrqoPgymSD3UgwkHyjmf2AYG7/bxKMTXTtTsPCi72cRnBRl00Eg9hzCUJHSoTOSpLx5mqCcPgt8EeCPvQn4yzA3V8DvkIw2LsJ6D9V9kvAlQRdQysIBnXfDwJkziUAAACWSURBVKzZxS6/RnAG1R8IAqUT+K9B23yH4JP/SoJA3G+Yus4keDN/GvgxwUWcvjiC5rURXFLz9wQXhfku8FV3/88R7EPGOV2oR0REInTEICIiEQoGKXlmdlP4behCPzcVuz6RuKkrSUqemc0A6oe5e7u7b46zHpFiUzCIiEiEupJERCRCwSAiIhEKBhERiVAwiIhIxP8Hg17I2rLx4HwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Initialize learner in VW but without -q\n",
    "vw = pyvw.vw(\"--cb_explore_adf --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations = 5000\n",
    "ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost)\n",
    "\n",
    "plot_ctr(num_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without learning\n",
    "Let's do the same thing again (but with `-q`, but this time show the effect if we don't learn from what happens. The ctr never improves are we just hover around 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfAklEQVR4nO3de5RddXn/8fdzbnPNJJPMEHKBJGAUQrQljICKgqISoD+wP10CSvHCktaK1VUsK2illVpEpaJWKMYbahVEbSVqFJFLU60gA4RLoMAkBDIkJJNkMpn7uT39Y++Esydnkpkks+dyPq+1Zs3Ze3/Pnud71pn9OftyvtvcHRERkT0S412AiIhMLAoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRiNiCwcy+bWbbzOyJYZabmX3VzNrM7DEzWxZXbSIi8rI49xhuAZbvZ/nZwOLw5zLg32KoSUREhogtGNx9DbBzP03OB77ngfuBGWY2J57qRERkj9R4F1BiHrCpZLo9nLdlaEMzu4xgr4K6urqTjjvuuFgKFBGZKh566KHt7t5cbtlECgYrM6/seB3uvhJYCdDS0uKtra1jWZeIyJRjZs8Pt2wiXZXUDhxVMj0f2DxOtYiIVKyJFAyrgEvCq5NOBbrcfZ/DSCIiMrZiO5RkZrcCZwBNZtYO/AOQBnD3m4HVwDlAG9AHfCCu2kRE5GWxBYO7X3SA5Q58JKZyRERkGBPpUJKIiEwACgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEhErMFgZsvN7GkzazOzFWWWH21m95rZI2b2mJmdE2d9IiISYzCYWRK4ETgbWAJcZGZLhjT7e+B2dz8RuBC4Ka76REQkEOcew8lAm7tvcPcscBtw/pA2DjSEj6cDm2OsT0REiDcY5gGbSqbbw3ml/hG42MzagdXAR8utyMwuM7NWM2vt6OgYi1pFRCpWnMFgZeb5kOmLgFvcfT5wDvB9M9unRndf6e4t7t7S3Nw8BqWKiFSuOIOhHTiqZHo++x4quhS4HcDd/wBUA02xVCciIkC8wfAgsNjMFplZhuDk8qohbV4AzgQws+MJgkHHikREYhRbMLh7HrgcuBN4iuDqo3Vmdo2ZnRc2uwL4kJk9CtwKvN/dhx5uEhGRMZSK84+5+2qCk8ql864uefwk8IY4axIRkSh981lERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhIRazCY2XIze9rM2sxsxTBt3m1mT5rZOjP7YZz1iYgIpOL6Q2aWBG4E3ga0Aw+a2Sp3f7KkzWLgKuAN7t5pZkfEVZ+IiATi3GM4GWhz9w3ungVuA84f0uZDwI3u3gng7ttirE9ERIg3GOYBm0qm28N5pV4JvNLMfm9m95vZ8nIrMrPLzKzVzFo7OjrGqFwRkcoUZzBYmXk+ZDoFLAbOAC4CvmlmM/Z5kvtKd29x95bm5ubDXqiISCWLMxjagaNKpucDm8u0ucPdc+7+HPA0QVCIiEhM4gyGB4HFZrbIzDLAhcCqIW1+BrwZwMyaCA4tbYixRhGRihdbMLh7HrgcuBN4Crjd3deZ2TVmdl7Y7E5gh5k9CdwL/J2774irRhERAXMfeph/cmlpafHW1tbxLkNEZFIxs4fcvaXcMn3zWUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCJGFAxmljazl8zshLEuSERExteIgsHdc0COfYfJFhGRKWY0h5L+FbjKzGK7HaiIiMRvNBv5NwKnAy+a2RNAb+lCdz+v7LNERGRSGU0wbAd+OlaFiIjIxDDiYHD3D4xlISIiMjGM+ByDmd1T7v7LZtZgZvcc3rJERGS8jObk8xlApsz8aoLzDyIiMgUc8FCSmS0rmXyNme0smU4CZwEvHu7CRERkfIzkHEMrwfcXHPhNmeX9wEcPZ1EiIjJ+RhIMiwADNgCvJbg6aY8ssM3dC2NQm4iIjIMDBoO7Pw9gZtcBr3X3m0uXm9lfmdk8d//0GNUoIiIxGs3J54uBR8rMfxi45PCUIyIi4200wXAE0FFm/nZg9uEpR0RExttoguEFyl+W+iag/fCUIyIi4200Q2J8HbjBzDLAni+0nQl8Dvj84S5MRETGx2iGxPgXM2sCvsrLX3TLAl9x9y+MRXEiIhK/UQ2h7e5XmdlngSUEl7A+6e49Y1KZiIiMi1HfW8Hde4EHx6AWERGZAHTPZxERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkItZgMLPlZva0mbWZ2Yr9tHuXmbmZtcRZn4iIxBgMZpYEbgTOJvjm9EVmtqRMu2nA3wAPxFWbiIi8LM49hpOBNnff4O5Z4Dbg/DLt/gn4AjAQY20iIhKKMxjmAZtKptvDeXuZ2YnAUe7+i/2tyMwuM7NWM2vt6Ch3iwgRETlYcQaDlZnnexeaJYAbgCsOtCJ3X+nuLe7e0tzcfBhLFBGROIOhHTiqZHo+sLlkehqwFLjPzDYCpwKrdAJaRCRecQbDg8BiM1sU3uznQmDVnoXu3uXuTe6+0N0XAvcD57l7a4w1iohUvNiCwd3zwOXAncBTwO3uvs7MrjGz8+KqQ0RE9m/U92M4FO6+Glg9ZN7Vw7Q9I46aREQkSt98FhGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRiIoNhkLR+dXjW3D38S5FRGRCqdhg+M7vn+PDP3iYn619cbxLERGZUCo2GLbuHgCgo3twnCsREZlYKjYYzAyAoo4kiYhEVG4whL91ikFEJKpig2FPMmTzRW646xkWrvglfdn8+NYkIjIBVGwwWJgMN/z2Gb5y97MALLn6Tu5+aut4liUiMu4qNhgSVn7+pd9txd33npwWEak0FRsMNkwwACy6ajWnXHs3d+hSVhGpQLEGg5ktN7OnzazNzFaUWf63ZvakmT1mZneb2YI46xtq9eNbAPjW757jlGt/qy/DiUhFSMX1h8wsCdwIvA1oBx40s1Xu/mRJs0eAFnfvM7MPA18ALhiLehL722UI3bluKwtX/HLvdNu2HhbPnjYW5YiITBhx7jGcDLS5+wZ3zwK3AeeXNnD3e929L5y8H5g/VsWUi4WTFjTu9zm/a9s+NsWIiEwgcQbDPGBTyXR7OG84lwK/KrfAzC4zs1Yza+3o6Di4asrsMXzvgyfv9ym/b9txcH9LRGQSiTMYyn1IL3vQ3swuBlqAL5Zb7u4r3b3F3Vuam5sPqphkmWCoq3r5yNrn3/nqyLJjmuu4f8MO8oXiQf09EZHJIs5gaAeOKpmeD2we2sjM3gp8CjjP3cdsIKPkkJ7/3Vmvikxf8Nqjed/rgnPft37oVK4861X0DOZ5cGPnWJUkIjIhxHbyGXgQWGxmi4AXgQuB95Q2MLMTga8Dy91921gWYyV7DK8/dhYfefMrAFj3mbP27sZ85vylfOb8pQD0ZfNkkgku+sb9PPDJM5ndUD2W5YmIjJvY9hjcPQ9cDtwJPAXc7u7rzOwaMzsvbPZFoB74sZmtNbNVY1VPsuQbbqWP66pS1Fftm5e1mRTzGmsAeM837h+rskRExl2cewy4+2pg9ZB5V5c8fmtctZR+87l1hIeHzn31HL52bxvrO3rJ5otkUhX7/UARmcIqdstW+j2G/lxhRM/5mzMX87YlswFYuWb9mNQlIjLeKjYYbARfcBsqk0rwb+9dBsD1v3mGbd0aT0lEpp6KDYbhBtE7kFQywXtPORqAt31pDQXd6UdEppiKDYaDzAUA/vnPg+84dPXnOOXa3/L8jl5++lD74SlMRGScVWwwJA52lyF09xWnA7C9J8vpX7yPK378KL96fAtbuvpp29Z9wOcXis7KNet5/3f+yOZd/YdUi4jI4RTrVUkTyUgG0dufY5vr+fXH38jyL//33nkf/sHD+7T7nxVvYe6M4DLX/myB46/+9T5tXn/dPfzkr17HZ3/5FIuPqOfDZxzLMc31h1SfiMjBqthgKM2FL737Tw5qHccd2cB9nziDM66/b9g2r7/uHi553QLeevxsLvn2H/dZfvLCmTyyqZN33fwHANZu2sWPH2rn7KVHcuXy46jLJKmrSlFXlaJnME+h4EyvTUfWMZArcNyng8BZMKuW8/5kLu855Wgefn4X7Z19nLa4iRPmTj+oPkqgP1vg+Z29bNk1wIbtvTy3vYfnd/ThDt0DOZIJo+gwmC+yvWeQuTNqaKgOvhOzszdLKmkYxvSaNDPrMsxuqOLI6TXMnVFNU30V3QN5svki1ekEhaKzpWuA+qoU9dUpplWnmDM9WF/Rg8Ogh7rHKxNTNl8knTRyBSedtGEvkhnMF8jmi1SlkmNy2bxN9nsMtLS0eGtr66if98MHXuCT//k4ADe9dxnnvHrOQdfg7pgZ/dkCz2ztpr2zn67+HGue6eDX617ap/21f/5qXjN/OkvnBRvrrbsHOOvLa9jVl+OkBY2s29zFQG74MZn+/tzjOe7IBi7+1gOjqrMmneS6d76a7oE8x8+ZxrKjGyNvvE07+7jlfzZy2uImFh9Rz/zG2lGtf7Jwd57Z2sOO3kEaazP0ZQt0D+To6B5kIFfghZ19bOjopaNnkN7BPN0DebZ17zs6y9zp1dRWpWioTjGQKzK9Jk0qadSkk/QM5unoHiRfdGozSVLJBLjT2Zejsy9L98DB3188nTRm1VXRNC1DY22GTDKBWXCl3UCuQEf3IAkz0klj/sxamuuraJ5WRW0miQHzGmuZUZvGgN0DOVKJBH3ZAnOmVzOzLng9OroHae/so7MvR1N9hvmNtcybUUMiEXwQ6R7Is3X3IH3ZPPVVKfpzBXoHCxTcwZ1cwamvSpEvOoP5Qvga58kXivTnCuQLzrTqFBu295JJJphRm2ZadYpcwZlek2Z2QzU1mQSZZJKGmhS1mRQ16SRFdwbzRXb1ZVnf0UPCjMbaDMmE0T2YZ1dvlnS4oezLFsCdqnSSZMKYVp2iUAzqqk4n6csWWN/RQ3/Y30LRqU4n2NWfI51MkM0XKboH4T+YZ2fvIMVi8KFyIFdg90DQ95l1GWbWZkinjFQiwUCuQFd/jsF88D/c2ZelUHRm1Kapy6SoSiVwIF9wcoUiyYTR2ZdlZ2+WXCG6Pa5JJ5lek6Y6nSBXCPo+kCvQMxi8fz77jqVcfOrB3bbGzB5y95ayyyo1GG774wus+I8gGG6+eBnLlx58MOzPHWtf5MqfPLb3TbLxunNH9Lz/fraDv/jWvnsYw5lWneLhT7+N57b38sgLnXx9zQY2dPRy0oJGGmsz/PYA97JOJqzsFVaNtWnyBWfpvOksmFXL8qVHsmRuA8919NJQk2ZHT5ajZtbQ0T3IneteYum86bxxcTOD+QJHNlTz+7YdPLG5i4TBEdOqcZyjZ9Yyv7GWpvqqyLfOSxWLzsYdvdRmUvQM5tjSNcD0mjRzptfQH/5D9wzm+d+XdrN20y5e7Oxn90CeWXUZmqcFn8D3bJS2dg2QTiWYWZehP1tgR0+W7AEGQ2yqz3Bscz3pZIJp1Sma6qs4bs405k6v4ZjmOo5qrD2kT+192WDD2t7Zx87eLO7QVF9Ff65AoVhkway6IJQG8+zuz7F19wBd/TmSZmQLzrbdA+wMNybZ8L3VMxhsqGrDvcx8wdnc1U9H92CwkRxnyYSRSSaoTidIJRN0D+SYM72GhEFXf57ugWCD3JvNE9dmKZkwEgbzG2tJJYz+XIEZ4Xs+nUyQShrFojOtOk1TfYZEwnAPwqGpvorewTydfVl29GTJF4MNfXU6SUN1mkzKSCYS1FcFYdDVn6NnML83cNLJBFWpBPmi01CdZnZDFVWpJPlikXQysTdEu/pz9GYLVKeSZFJGVSpJU32G6nSS1x/bxJK5DQfVdwVDGbe3buLKnzwGwNf/4iTOOuHIw11aRKHow24ER8PduWPtZj7+o7VcfOrRfOqcJdRkkiN63vaeLK0bd/LpO57gDa9o4okXu1jf0Rtpd/ormzlhbgMPv9DJIy/s2htoY2XejBoWzKrlpAWNZJIJfr3uJdZ39Ox3j2moZMJYOreBRU119GULvLR7gKI71algA1lfHf5j9gUbntqqJK+aPY2jZ9bSly1QdGd2Q/Xe8a8WH1E/5Q7V9GXzdPUHewcv7gr2aPOFIg01aQbDvZ2Xdg+wo2eQ2qoU02vSHNNUR0N1ml39Wdo7+3mxsz94XdNJajPJ8FN98Mm7KpWgoSZNOmFgkEok6B3Mk0wYVakEtZkUmVRiRP8DuUJx795btlCkqy9HX7bAQK6AmVGdTlCVSvLK2fWkEgl29mVxd2ozKRrr0hSKTtGhNp3EDLKFIoWis7s/+JTdnyvQny2QSSVY1FR3WP4vJyMFQxk/f3QzH731EQC+cUnL3m80V6qXugb27rKXky8U6R7I81/PdLDm2Q5yBWdW+Am8oSbFCzv7WHZ0IyfMnc7aTZ109efY3pPlue29/OWbjuFVR07jue297OjJsq17gGy+yM6+LM9u7aGje5AN218OqOZpVbz5VcFw6vNm1LJ5Vz8Lm+qoq0rSPZBnIFdgUVMd1ekki4+o59jmqbchFxlr+wuGij35XFVywqZQ1D0Wjpy+/9FiU8kEjXUZ3nHiPN5x4v7urwSnLW4qO39/V1p19gbHjGfWZXRFlsg4q9hgKDWawxYyNhrrMrTUzRzvMkSECv6CW+kBtJEOoiciUgkqNhhKDSgYRET2UjCgPQYRkVIKBqBQmNxXZomIHE4VGwx7rtJ97cJGPnDaovEtRkRkAqnYYNjjM+ctLXuPZxGRSlXxwSAiIlEVHAw6ryAiUk4FB0PgEG/LICIy5VR8MIiISJSCQUREIio2GCb5oLIiImOmYoNhD51jEBGJqvhgEBGRqIoNBh1JEhEpr2KDYQ9Dx5JEREpVfDCIiEhUxQaDrkoSESmvYoNhD12VJCISVfHBICIiUQoGERGJqNhgcF2wKiJSVqzBYGbLzexpM2szsxVllleZ2Y/C5Q+Y2cIxr2ms/4CIyCQTWzCYWRK4ETgbWAJcZGZLhjS7FOh091cANwCfH6t6+rKFsVq1iMikFucew8lAm7tvcPcscBtw/pA25wPfDR//BDjTbGyuG3pgw04A0smKPZomIlJWnDc7ngdsKpluB04Zro27582sC5gFbC9tZGaXAZeFkz1m9vRB1tS06PPRdVeAJlCfK4D6XBkOpc8LhlsQZzCU++Q/9AzwSNrg7iuBlYdckFmru7cc6nomE/W5MqjPlWGs+hzncZR24KiS6fnA5uHamFkKmA7sjKU6EREB4g2GB4HFZrbIzDLAhcCqIW1WAe8LH78LuMddg1eIiMQptkNJ4TmDy4E7gSTwbXdfZ2bXAK3uvgr4FvB9M2sj2FO4cIzLOuTDUZOQ+lwZ1OfKMCZ9Nn0gFxGRUrpWU0REIhQMIiISUbHBcKDhOSYTM/u2mW0zsydK5s00s7vM7Nnwd2M438zsq2G/HzOzZSXPeV/Y/lkze1+5vzURmNlRZnavmT1lZuvM7GPh/Knc52oz+6OZPRr2+TPh/EXh8DHPhsPJZML5ww4vY2ZXhfOfNrOzxqdHI2dmSTN7xMx+EU5P6T6b2UYze9zM1ppZazgv3ve2u1fcD8HJ7/XAMUAGeBRYMt51HUJ/3gQsA54omfcFYEX4eAXw+fDxOcCvCL4zcirwQDh/JrAh/N0YPm4c774N0985wLLw8TTgGYJhVqZynw2oDx+ngQfCvtwOXBjOvxn4cPj4r4Gbw8cXAj8KHy8J3+9VwKLw/yA53v07QN//Fvgh8Itwekr3GdgINA2ZF+t7u1L3GEYyPMek4e5r2Pf7HqXDi3wXeEfJ/O954H5ghpnNAc4C7nL3ne7eCdwFLB/76kfP3be4+8Ph427gKYJvzU/lPru794ST6fDHgbcQDB8D+/a53PAy5wO3ufuguz8HtBH8P0xIZjYfOBf4ZjhtTPE+DyPW93alBkO54TnmjVMtY2W2u2+BYEMKHBHOH67vk/I1CQ8XnEjwCXpK9zk8pLIW2Ebwj74e2OXu+bBJaf2R4WWAPcPLTKo+A18GrgSK4fQspn6fHfiNmT1kwfA/EPN7O84hMSaSEQ29MUUN1/dJ95qYWT3wU+Dj7r7bhh9vcUr02d0LwJ+a2QzgP4HjyzULf0/6PpvZnwHb3P0hMztjz+wyTadMn0NvcPfNZnYEcJeZ/e9+2o5Jnyt1j2Ekw3NMdlvDXUrC39vC+cP1fVK9JmaWJgiFH7j7f4Szp3Sf93D3XcB9BMeUZ1gwfAxE6x9ueJnJ1Oc3AOeZ2UaCw71vIdiDmMp9xt03h7+3EXwAOJmY39uVGgwjGZ5jsisdXuR9wB0l8y8Jr2Y4FegKd03vBN5uZo3hFQ9vD+dNOOFx428BT7n7l0oWTeU+N4d7CphZDfBWgnMr9xIMHwP79rnc8DKrgAvDK3gWAYuBP8bTi9Fx96vcfb67LyT4H73H3d/LFO6zmdWZ2bQ9jwnek08Q93t7vM/Aj9cPwdn8ZwiO035qvOs5xL7cCmwBcgSfFC4lOLZ6N/Bs+Htm2NYIbpi0HngcaClZzwcJTsy1AR8Y737tp7+nEewWPwasDX/OmeJ9fg3wSNjnJ4Crw/nHEGzk2oAfA1Xh/Opwui1cfkzJuj4VvhZPA2ePd99G2P8zePmqpCnb57Bvj4Y/6/Zsm+J+b2tIDBERiajUQ0kiIjIMBYOIiEQoGEREJELBICIiEQoGERGJUDCIHICZLTQzN7MJdaP5iVqXTH4KBpED20QwoutaADM7I9wgN8VVgJndZ2Zf219dIodLpY6VJDJiHoxR9NJYrNvM0u6eO5jnjmVdUtm0xyATSvjJ+CYzu9bMtltwA6LrzSwRLt9oZp8o85yvlUxvNLOrzewWM+s2s01mdoGZzTCz28ysJ7x5ydtHWNPeQzbhaK73hos6wvm3hO3MzK40s/Vm1m/BzVYuLrOei8zsHjPrB/7SzGaZ2a1m1h4+b52ZfaDkebcApwMfCZ/v4br2OZRkZm+y4CY1A2a21cxuCId9GdHrG7b5/xbc9KXfzHaa2X+Z2eyRvFYyNSgYZCJ6L5AHXg9cDnwcuGCU6/g4wbAIywhu7PJdgpu9rAb+FFgD/LuZVY9yvZuAd4aPTyA4lPOxcPqzBMORfITg5jCfA75uZucOWcfngJvCNj8jGMrhYeDPwnV+JXzemWH7jwF/AL4T/r05RIdUBsDM5hHctOURgqHILwUuCv9eqWFfXzM7kmDAuu8SjN76JuD7I3hdZCoZ77FB9KOf0h+CUUP/MGTeXcA3w8cbgU+Uec7XSqY3AreWTNcTjK301ZJ5C8N5LSOoKdKWYNwep+QuW0Ad0A+8cchzvwysHrKeK0bwN2/b0+dyfRymrn8mGBcnUdLm/cAgUDvC13dZuM4F4/1e0M/4/egcg0xEjw2Z3szLNyYZ9TrcvcfM+ggGGdtja/h7tOsdzhKCT/6/NrPSAcjSBEFVqrV0wsySBLdrvIDgZipVBLecvW+UNRxPsNEvlsz7XbiuV/Dya7K/1/dR4LfAE2b2m/DxT9y9Y5S1yCSmYJCJaOjJWOflw55F9r0JSXqE68gNmYbDdzh1z3r+H/DCAWrpHTL9CeAKgkNGjwM9wLWMPrSM4W/GUjp/2NfX3QvhuZdTCYZqvhT4nJmd7u6PjrIemaQUDDLZdBAcYwcgPEdwHMFx9bhkw9/JknlPEhyyWeDu94xyfacBP3f378Pe+028Etg15G8myzy31JPAu80sUbLXcFr43PUjLcbdneCcxh/M7BqC4Z8vINibkAqgYJDJ5h7gg2a2iiAkPkX5PYax9DzBp+xzzeznQL+7d5vZ9cD14YZ9DcG5jVOBoruv3M/6ngEuMLPTgO3AR4FFRMNuI3ByeFVUD8GdyYa6ieBE8k1m9hWCsf2vIzg30TeSjoU3e3krwU1dthKcxD6KIHSkQuiqJJlsPkcQDncAvyE4hv5wnAW4+4vAPxCc7N0K7LlU9tPAPxIcGlpHcFL3ncBzB1jlZwmuoPoVQaD0Aj8Y0uZ6gk/+TxIE4tHD1HU2wcZ8LfBtgps4fXIU3esiuKXmLwhuCvMvwD+5+7+PYh0yyelGPSIiEqE9BhERiVAwSMUzs5vDb0OX+7l5vOsTiZsOJUnFM7MjgIZhFu92921x1iMy3hQMIiISoUNJIiISoWAQEZEIBYOIiEQoGEREJOL/AL1MH1d3w2a1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Initialize learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations = 5000\n",
    "ctr  = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost, should_learn=False)\n",
    "\n",
    "plot_ctr(num_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 2\n",
    "\n",
    "In the real world people's preferences change over time. So now in the simulation we are going to incorporate two different cost functions, and swap over to the second one halfway through. Below is a a table of the new reward function we are going to use, `get_cost_1`:\n",
    "\n",
    "### Tom\n",
    "| | `get_cost` | `get_cost_new1` | \n",
    "|---|---|---|\n",
    "| **Morning** | Politics | Politics | \n",
    "| **Afternoon** | Music | Sports | \n",
    "\n",
    "### Anna\n",
    "| | `get_cost` | `get_cost_new1`  |\n",
    "|---|---|---|\n",
    "| **Morning** | Sports | Sports |\n",
    "| **Afternoon** | Politics | Sports |\n",
    "\n",
    "This reward function is still working with actions that the learner has seen previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost_new1(context,action):\n",
    "    if context['user'] == \"Tom\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE\n",
    "    elif context['user'] == \"Anna\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'sports':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make it easy to show the effect of the cost function changing we are going to modify the `run_simulation` function. It is a little less readable now, but it supports accepting a list of cost functions and it will operate over each cost function in turn. This is perfect for what we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation_multiple_cost_functions(vw, num_iterations, users, times_of_day, actions, cost_functions, should_learn = True):\n",
    "    cost_sum = 0.\n",
    "    ctr = []\n",
    "    \n",
    "    start_counter = 1\n",
    "    end_counter = start_counter + num_iterations\n",
    "    for cost_function in cost_functions:\n",
    "        for i in range(start_counter, end_counter):\n",
    "            # 1. in each simulation choose a user \n",
    "            user = choose_user(users) \n",
    "            # 2. choose time of day for a given user\n",
    "            time_of_day = choose_time_of_day(times_of_day)\n",
    "\n",
    "            # Construct context based on chosen user and time of day\n",
    "            context = {'user': user, 'time_of_day': time_of_day}\n",
    "\n",
    "            # 3. Use the get_action function we defined earlier\n",
    "            action, prob = get_action(vw, context, actions)\n",
    "\n",
    "            # 4. Get cost of the action we chose\n",
    "            cost = cost_function(context, action)\n",
    "            cost_sum += cost\n",
    "\n",
    "            if should_learn:\n",
    "                # 5. Inform VW of what happened so we can learn from it\n",
    "                vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n",
    "                # 6. Learn\n",
    "                vw.learn(vw_format)\n",
    "\n",
    "            # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "            ctr.append(-1*cost_sum/i)\n",
    "        start_counter = end_counter\n",
    "        end_counter = start_counter + num_iterations\n",
    "        \n",
    "    return ctr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With learning\n",
    "Let us now switch to the second reward function after a few samples (running the first reward function). Recall that this reward function changes the preferences of the web users but it is still working with the same action space as before. We should see the learner pick up these changes and optimize towards the new preferences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxcdb3/8ddnZrI1aZu2SQt0ry1LFbyFWIugoKAsXuGhosK9KirKbxGvXlQeIP7EiwuouKGgoiCIynLRn1Z+bMriitgAAm2lkEJt0pa2oVv2ZGY+vz/OSTonnaRJm5xJMu/n4zGPnm3OfE5Omvec7fs1d0dERKRXotAFiIjI2KJgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiYgtGMzsJjPbZmarB5hvZnatmTWY2dNmdmxctYmIyF5xHjHcDJw+yPwzgCXh60LgezHUJCIi/cQWDO7+B2DHIIucDfzEA38Fqs3s0HiqExGRXqlCF5BjNtCYM94UTtvSf0Ezu5DgqILKysrjjjzyyFgKFBGZKB5//PFmd6/NN28sBYPlmZa3vQ53vwG4AaCurs7r6+tHsy4RkQnHzP450LyxdFdSEzA3Z3wOsLlAtYiIFK2xFAwrgfeHdyetAHa7+z6nkUREZHTFdirJzG4DTgZqzKwJuAIoAXD37wP3AGcCDUA78MG4ahMRkb1iCwZ3P28/8x34aEzliIjIAMbSqSQRERkDFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZGIWIPBzE43s3Vm1mBml+aZP8/MHjazJ83saTM7M876RHK5O+4+6Pymne007mjnpd2dbNvTye6OnkHfIzIepOL6IDNLAtcBbwaagFVmttLd1+Ys9lngTnf/npktBe4BFsRVo0xcG19up/6fO9i6p4vDZ1Uxf8Yksg67O3pobunixZfbWLNpDy80t7GrvZvdHT109GRImjG9spQpFSXMrq4gmTBaOntYv72N1s403ZnsPp81qTTJ5PIUZakkF5y4kPeumE8yYQXYapEDE1swAMuBBnd/AcDMbgfOBnKDwYEp4fBUYHOM9ckY05XOsL2li7JUkvbuNG1dGV5obmXTzg6601nawz/cFaVJOrozZN2ZWlHC9pYuNrzcRlVZij2daZp2tvPc1tb9ft6caRUsnlnFkYdMpqosRXlJIvic7gw72rrZsruTrDtVZSlOPqKW6opS5k6v6KsPoKM7w0t7OmnrSvNicxtXrFzDHasa+cSpS3jz0lmYKSBk7IszGGYDjTnjTcBr+y3zeeABM/sYUAmcmm9FZnYhcCHAvHnzRrxQGR3uTk/G2bijjX++3M62li627OrgsRd38GJz8Id84452HJg2qYTm1u4D+pzSVII51RXs6exhSkUJtVVlfPatR3HC4hqmV5by3NYWNu3swAxmTSmnelIpC2ZMonpS6Yhv712PN3Hdww1ceOvjLF8wnf84ZQknLqkZ0c8RGWkW1/lQM3sXcJq7fzgcfx+w3N0/lrPMxWFNXzez44EbgVe5+77H66G6ujqvr68f5erlQDXuaOc3T2/m4We38VTjbjDoTkd356LaShbXVtHRk6GqLEUqmaAnnWV+zSRmTi6nO52loiRBeUmSxTOrOLS6gmmTSkglEphBe3eGSaVJdrZ3YwSnfsbSqZuudIbbHtvI9Y+sZ1tLF/OmT+KCExfyvhXzSYyhOqUwejJZ2rrSdPRkmDaplPKSZHh9CzLuZLJOS2ea7S1dbN3TSUtXmkw2i2Esm1fN/BmVB/S5Zva4u9flmxfnEUMTMDdnfA77niq6ADgdwN0fNbNyoAbYFkuFMizuzupNe/juw8+zZXcnx82fxn2rXyKVNBp3dESWnT9jEsvmVbOwppKj50zlsOoKaqvKWDyzivKS5EHVMbUiuIdi5uTyg1rPaClLJfnACQs577Xz+PljG/l/T2/hipVruLO+kbcvm83yhdN55WFTx1SYTRS9f2BzA9jd6UpncYfNuzto6UxTWZok69DS2UNPxkkljZ5MlvauDK1d6b737ukMrkm1dKVJmIU3G0BZSYJs1smGR8XlJQl2tffQlc7Slc7Q2ZOloztYlxmUpRJ09mTZ0dYdWX/CIJkwejJD+8L+5bcffcDBMJg4jxhSwHPAKcAmYBXwb+6+JmeZe4E73P1mMzsKeBCY7YMUqSOG+L3Y3MZ3HnyeXz65Ke/80mSCOdMrmFJewomLazjnuDksqBn5X97xKpt17qhv5PpHGvoCdHJZiiMOmcyyedXMm1HJoppKjps/7aBDc7xp707T3p1hZ1s3tZPL2NXeQypppDPO1j2ddKazbNrZQWdPho6e4A/t1j2dNLd2k8lmaelM053Oks46Hd0Zmlu7yLpTWZbq+0OeDcPiQJlBZWkKd2dyeQkWHgUnE0bCjFTS6EpnmRLegFAeHu1WlAQ3JWQ9OIqsKEkyrbKU6opSJpenKC9JsnlXBxl3SpIJDEgljGTSqCpLUVNVxqwp5UytSJFMJMi6M6Oy9IBPgQ52xBBbMISFnAl8C0gCN7n7l8zsSqDe3VeGdyL9EKgiuBB9ibs/MNg6FQyjp7UrTTqT5ZdPbOKr9z9LZ0/0FFBZKsHFbz6cY+ZUk0wE/yGWza3WBdZh2PhyO39Z38xTTbt5qnEX67a2kMkG/yfLUglOWFzDGa86hLcvm00qOXYeO3J3mlu72byrg9JUou/0XeOOdrrSWaZWlFCSTJDJOns6e2jvzpDOZPu+Hbd2pdm8q4O2rgybd3WwraWLpp3tw76ulEoYtZPLwlMwCSaXl1CWSpBKGhUlKWZUlZIwo707TU8my7RJwXhFaRC4h1WXM6W8hJbONMmEMaWihFTCSGedkqRRWZqiqnzviZXJZSlmVJVNiKO7MRMMo0HBcHB6D7UfXreNnz+2kQefHfys3bRJJZx8xEw+8LoFvHpudUxVFo/udJZNuzpYvWk3j77wMvc+s4Wd7T3Mmz6JU46aydGzpzJzcjklSeOQqeWjchoBoLm1i4072tm6u5MXX25j084OdrR1s3l3JxtfbqOtK5P3Vt3hSBhUlCQ5rLqCmVPKmF1dwYKaSiaF36Q37eqgprIMx0klEsycUkZ5uPykkiQVpUnKUgl9ETlACgahO53ljvpG/vjcdh5Yu3VI71kQ3ut/1TuOZsWiGRPiW9J44+7ct/olbv7LBv7euIuufhfuayeXUTd/GnULprOotpKkGelslnTGSZjRmc7Q3NJFT8bpDr+xd3Rn2NbSSUtnMAzBN/ieTJaSZILm1m6aW7sinzO1ooSaqlIOmVrOvOmVTClPMWtKOXOmVdCZzrKzrRt3Z/a0SSQT0NKZpifjlKUSTKkooaIkSUl4SiSRMCpKksyaUq7fqQJSMBShbXs6ueCWep7ZtHu/y56weAZLZk7mda+YwalHzcIMfQsbg9q60mzZ3UlzaxedPRnWbtnDupda+HPDy/v8IR9IKmGUphLUVJUxrbKUslSChEFVWQklSaO1K82hU8s5fNZkFtVWUlNVxqLaKqrK4rxPReIwVu5KklHWnQ5ue/vz+mYu+vmT+8x/9ZypfPCEhSyeWUVLZ5oVi6YrAMaRyrIUi2dWsXhmFQAnHzETCI4qtuzupGlnB5msY+GdLRCcqjlkajmlqQSlyeClW2Rlf4o6GPZ09jClvKTQZRwUd+fzK9dwy6P/3Gfep087ggtOXFh0d7YUGzPjsOoKDquuKHQpMkEUbTD8paGZf/vRY/zkQ8t5w+G1hS7ngHzt/me57uH1+0x/7cLp3PzB5X13XoiIDEfRBsOqDTvDf3eMu2Bwd95349/4U0MzAEceMpmffGg5M6eMzQe8RGR8Kdpg6DXezra+tLuTFVc9CMAbDq/lpvPrxtT97SIy/hV9MIwXPZksr7zi/r52hs45bg5fO+cYXTwWkRFXtMHgjJ/bdH/5RBMX3/lU3/g173o15xw3p4AVichEVrTBMB40bGvl1G/8vm/8ojcu5lOnHVHAikSkGCgYxqjP/uoZfvrXjX3jKy86gWPmqAkKERl9RRsMfQ98j7Fz9D2ZLEs/d19fs7sfP2UJ//nmwwtclYgUk6INhl5jJRZ6Mlled/VDbG/Z27TBXy59kx5aEpHYFX0wjBUf/dkTfaHw2oXTuf3CFbrjSEQKQsEwBtxZ38gDa7dyxKzJfO1dx+hagogUVNEGw1i5WXX99lYuuetpAH710RPUjIWIFFzRBkPv1edCna25b/UW/udPn+gbv/H8OoWCiIwJRd+WQtPOjv0vNMKe3LgzEgonHV7LKUfNir0OEZF8ijYYek8l3fV4U6yfu+6lFt5+/V+AoP/Yz5x5JLd8aHmsNYiIDKZoTyVlC9Rz3Wnf+gMA575mLle/85iC1CAiMpiiPWLIFiAXvv275wFYNq9aoSAiY1YRB0O8ydDaleabv3sOgJ99+LWxfraIyHAUbTDcVR/ftYXfP7edV11xPwCffetRTCot2jN4IjIOFO1fqJfbumP5nA/8+G88sm573/iHX78ols8VETlQRXvEEIfd7T19ofCOY2ez4eq3FrgiEZH9K9ojhjj85unNAFz9jqN5d93cAlcjIjI0CoZRsmrDDj77q9UAvOc1c9UgnoiMGzqVNArSmSzv+v6jAMydXqFQEJFxRUcMI+zy//sMP3tsb89rf7zkTQWsRkRk+HTEMILcPRIK6754egGrERE5MAqGEbRm856+4Uc+dTJlKbWWKiLjj04ljYB0JksqmeDaB4MmL1Zdfiq1k8sKXJWIyIHREcNBunNVI4svv5eGba08sHYrgEJBRMY1BcNBuuQXQe9rp37j9wC8R88riMg4F2swmNnpZrbOzBrM7NIBlnm3ma01szVm9vM46xuuzbv27eTn82e9sgCViIiMnNiCwcySwHXAGcBS4DwzW9pvmSXAZcAJ7v5K4BNx1Xcg3nrtHwF44xG1AHz73H9R95wiMu7FefF5OdDg7i8AmNntwNnA2pxlPgJc5+47Adx9W4z1DcuOtm52tvcA8MP315FK6qyciEwMcf41mw005ow3hdNyHQ4cbmZ/NrO/mlneBwHM7EIzqzez+u3bt+dbZP/FVFcc0Pt6fe+Rhr5hhYKITCRx/kXL1y5E/95yUsAS4GTgPOBHZla9z5vcb3D3Onevq62tPaBijps/7YDe16u31dRnPv+Wg1qPiMhYE2cwNAG5t+zMATbnWebX7t7j7i8C6wiCYsQdTP9tO9u6eX5bKwCTy0tGpiARkTEizmBYBSwxs4VmVgqcC6zst8yvgDcCmFkNwamlF0ajmLLUgW/6si/8FoDXLDi4ow4RkbEotmBw9zRwEXA/8A/gTndfY2ZXmtlZ4WL3Ay+b2VrgYeDT7v7yaNTz6rn7nKEaknQm2zf87XOXjVQ5IiJjRqxNYrj7PcA9/aZ9LmfYgYvD16g60Iaw/7x+b04ddpAXsEVExiLdTjMMXekM59/0NwDu+8TrC1yNiMjoUDAMwx+fa+4bPvKQKQWsRERk9CgYhmFnezcAF71xcYErEREZPQoGIJsd2s2rn74raDDvY6coGERk4lIwAOmsk9lPOLzY3NY3rA54RGQiUzAAF95azys+c8+gy7zxmkfiKUZEpMCKNhhyjw96m7cYSO6ppr9dfsooVSQiMjYUbTAMx41/erFveObk8gJWIiIy+hQMOQa6CP2le/4BwLmvUe9sIjLxDSkYzKzEzF4yswndPVlXOjvo/C+//eiYKhERKZwhBYO79wA9HFyjpGPeUZ+7L9IWEkBHd6ZvOJE40IY0RETGj+GcSvoOcJmZxdq+UtyebNwVGb8+p0MeEZFiMJw/8q8HTgI2mdlqoC13pruflfdd40xPv9NJ33koCIbfXXxSIcoREYndcIKhGfjFaBUyVlz3SAOvW1wDwCPr9nY5vbCmslAliYjEasjB4O4fHM1Cxoo/N+xtVvsDP17VN5zU9QURKRJDvsZgZg/l63/ZzKaY2UMjW1ZhbdndAUBpMvjxNHzpjEKWIyISq+FcfD4ZKM0zvZzg+sP44tEbrC45/Yi+4eOvCnKuO7xDKZXU4x4iUjz2eyrJzI7NGT3GzHbkjCeB04BNI11Y3I6dN40TFs/oO5XUuKO9wBWJiBTGUK4x1BM8v+DAA3nmdwAfG8miCmHFohms3rS7Lxj+Ht62evmZRxWyLBGR2A3lHMlC4BUE3SS/Jhzvfc0Gprj7TaNWYYzeeeycvuGP3fYkAFMrSgpVjohIQew3GNz9n+6+AbgaeE043vvaAnzEzL4w2oWOpt47jqZVlvLqudHr62979WGFKElEpGCGc1X1vcCTeaY/Abx/ZMqJ35TyFA/mPLx2xduWRuZXlKpTHhEpLsN5wG0mkK/jgmZg1siUE7+HPnUyNVVlfeNdPYM3pCciMtEN54hhI/lvS30D0DQy5RRe3YJphS5BRKSghnPE8APgm2ZWCvQ+0HYKcBXwlZEurFBKkglu+8gKfvjHF/jB+44rdDkiIrEbTpMYXzezGuBa9j7o1g18292/OhrFjabB2g8//hUzOP4VM2KrRURkLBlWE9rufpmZfRFYSnD76lp3bx2VymKiFpBERKKG3beCu7cBq/a7oIiIjEtqBEhERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhFFGww+2BNuIiJFLNZgMLPTzWydmTWY2aWDLHeOmbmZ1cVQ02h/hIjIuBJbMJhZErgOOIPgyenzzGxpnuUmA/8BPBZXbSIislecRwzLgQZ3f8Hdu4HbgbPzLPcF4KtAZ4y1iYhIKM5gmA005ow3hdP6mNkyYK673z3YiszsQjOrN7P67dvzdREhIiIHKs5gyHcyv+8SsJklgG8Cn9zfitz9Bnevc/e62traESxRRETiDIYmYG7O+Bxgc874ZOBVwCNmtgFYAayM4wK0iIjsFWcwrAKWmNnCsLOfc4GVvTPdfbe717j7AndfAPwVOMvd62OsUUSk6MUWDO6eBi4C7gf+Adzp7mvM7EozOyuuOkREZHDD7o/hYLj7PcA9/aZ9boBlTx7lWkZz9SIi41bRPvncS4+3iYhEFX0wiIhIlIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRRRsMerxNRCS/og2GXurATUQkquiDQUREohQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJKNpgUAduIiL5FW0w9DL14SYiElH0wSAiIlEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRBRtMOj5NhGR/Io2GPro+TYRkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREImINBjM73czWmVmDmV2aZ/7FZrbWzJ42swfNbH6c9YmISIzBYGZJ4DrgDGApcJ6ZLe232JNAnbsfA9wFfHW06nF14SYiklecRwzLgQZ3f8Hdu4HbgbNzF3D3h929PRz9KzBntIsyPeAmIhIRZzDMBhpzxpvCaQO5ALg33wwzu9DM6s2sfvv27SNYooiIxBkM+b6b5z2fY2bvBeqAr+Wb7+43uHudu9fV1taOYIkiIpKK8bOagLk543OAzf0XMrNTgcuBk9y9K6baREQkFOcRwypgiZktNLNS4FxgZe4CZrYM+AFwlrtvi7E2EREJxRYM7p4GLgLuB/4B3Onua8zsSjM7K1zsa0AV8N9m9nczWznA6kREZJTEeSoJd78HuKfftM/lDJ8aZz0iIrIvPfksIiIRCgYREYko+mDQ820iIlFFHwwiIhKlYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISETRBoM6cBMRya9og6GXqQs3EZGIog8GERGJUjCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCSiaIPB0RNuIiL5FG0w9NLjbSIiUUUfDCIiEqVgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIRNEGg3pwExHJr2iDoZc6cBMRiSr6YBARkSgFg4iIRCgYREQkQsEgIiIRCgYREYmINRjM7HQzW2dmDWZ2aZ75ZWZ2Rzj/MTNbEGd9IiISYzCYWRK4DjgDWAqcZ2ZL+y12AbDT3RcD3wS+Eld9IiISiPOIYTnQ4O4vuHs3cDtwdr9lzgZuCYfvAk4x05MGIiJxSsX4WbOBxpzxJuC1Ay3j7mkz2w3MAJpzFzKzC4ELw9FWM1t3gDXVVH4luu4iUAPa5iKgbS4OB7PN8weaEWcw5Pvm379hiqEsg7vfANxw0AWZ1bt73cGuZzzRNhcHbXNxGK1tjvNUUhMwN2d8DrB5oGXMLAVMBXbEUp2IiADxBsMqYImZLTSzUuBcYGW/ZVYC54fD5wAPuau5OxGROMV2Kim8ZnARcD+QBG5y9zVmdiVQ7+4rgRuBW82sgeBI4dxRLuugT0eNQ9rm4qBtLg6jss2mL+QiIpJLTz6LiEiEgkFERCKKNhj21zzHeGFmc83sYTP7h5mtMbOPh9Onm9lvzez58N9p4XQzs2vD7X7azI7NWdf54fLPm9n5A33mWGFmSTN70szuDscXhk2pPB82rVIaTh+wqRUzuyycvs7MTivMlgyNmVWb2V1m9my4v4+f6PvZzP4z/L1ebWa3mVn5RNvPZnaTmW0zs9U500Zsv5rZcWb2TPiea4f00LC7F92L4OL3emARUAo8BSwtdF0HuC2HAseGw5OB5wiaHPkqcGk4/VLgK+HwmcC9BM+MrAAeC6dPB14I/50WDk8r9PbtZ9svBn4O3B2O3wmcGw5/H/hf4fD/Br4fDp8L3BEOLw33fRmwMPydSBZ6uwbZ3luAD4fDpUD1RN7PBA+8vghU5OzfD0y0/Qy8ATgWWJ0zbcT2K/A34PjwPfcCZ+y3pkL/UAq0I44H7s8Zvwy4rNB1jdC2/Rp4M7AOODScdiiwLhz+AXBezvLrwvnnAT/ImR5Zbqy9CJ6DeRB4E3B3+EvfDKT672OCO+GOD4dT4XLWf7/nLjfWXsCU8I+k9Zs+Yfcze1tCmB7ut7uB0ybifgYW9AuGEdmv4bxnc6ZHlhvoVaynkvI1zzG7QLWMmPDQeRnwGDDL3bcAhP/ODBcbaNvH28/kW8AlQDYcnwHscvd0OJ5bf6SpFaC3qZXxtM2LgO3Aj8PTZz8ys0om8H52903ANcBGYAvBfnucib2fe43Ufp0dDvefPqhiDYYhNb0xnphZFfAL4BPuvmewRfNM80Gmjzlm9q/ANnd/PHdynkV9P/PGzTYTfAM+Fvieuy8D2ghOMQxk3G9zeF79bILTP4cBlQStM/c3kfbz/gx3Gw9o24s1GIbSPMe4YWYlBKHwM3f/ZTh5q5kdGs4/FNgWTh9o28fTz+QE4Cwz20DQSu+bCI4gqi1oSgWi9Q/U1Mp42uYmoMndHwvH7yIIiom8n08FXnT37e7eA/wSeB0Tez/3Gqn92hQO958+qGINhqE0zzEuhHcY3Aj8w92/kTMrt3mR8wmuPfROf394d8MKYHd4qHo/8BYzmxZ+U3tLOG3McffL3H2Ouy8g2HcPufu/Aw8TNKUC+25zvqZWVgLnhnezLASWEFyoG3Pc/SWg0cyOCCedAqxlAu9nglNIK8xsUvh73rvNE3Y/5xiR/RrOazGzFeHP8P056xpYoS+6FPBiz5kEd/CsBy4vdD0HsR0nEhwaPg38PXydSXBu9UHg+fDf6eHyRtBh0nrgGaAuZ10fAhrC1wcLvW1D3P6T2XtX0iKC//ANwH8DZeH08nC8IZy/KOf9l4c/i3UM4W6NAm/rvwD14b7+FcHdJxN6PwP/BTwLrAZuJbizaELtZ+A2gmsoPQTf8C8Yyf0K1IU/v/XAd+l3A0O+l5rEEBGRiGI9lSQiIgNQMIiISISCQUREIhQMIiISoWAQEZEIBYPIfpjZAjNzMxtTHc2P1bpk/FMwiOxfI0FjZH8HMLOTwz/INXEVYGaPmNl3B6tLZKTE1uezyHjl7hngpdFYt5mVeNDcw7CNZl1S3HTEIGNK+M34ejP7spk1hx2YXGNmiXD+BjP7VJ73fDdnfIOZfc7MbjazFjNrNLP3WNDRze1m1hp2ZvKWIdbUd8ombMH24XDW9nD6zeFyZmaXmNl6M+sIO0d5b571nGdmD5lZB/A/zGyGBZ3QNIXvW2NmH8x5383AScBHw/d7uK59TiWZ2Rss6KSm08y2mtk3w2ZfhvTzDZd5hwWdwHSY2Q4z+72ZzRrKz0omBgWDjEX/DqQJGky7CPgE8J5hruMTBM0iHEvQscstBJ363EPQtMQfgJ+aWfkw19sIvDMcfiXBqZyPh+NfJGjO4KMEncNcBfzAzN7abx1XAdeHy/yKoCmHJ4B/Ddf57fB9p4TLfxx4FPhx+HmHEm1iGQAzm03QEcuTBM2vX0DQ/v5V/RYd8OdrZocQNEx4C3AUQScytw7h5yITSaHbCdFLr9wX8AjwaL9pvwV+FA5vAD6V5z3fzRnfANyWM15F0J7UtTnTFoTT6oZQU2RZgvaZHKjJWaYS6ABe3++93wLu6beeTw7hM2/v3eZ82zhAXV8iaCcnkbPMB4AuYNIQf77HhuucX+jfBb0K99I1BhmLnu43vpm9HZUMex3u3mpm7QSNjvXaGv473PUOZCnBN//7zCy3AbISgqDKVZ87YmZJgr4V3kPQiUoZQdedjwyzhqMI/uhnc6b9KVzXYvb+TAb7+T4F/A5YbWYPhMN3ufv2YdYi45iCQcai/hdjnb2nPbPs2/lIyRDX0dNvHEbudGrvet5G0Fz0YLW09Rv/FPBJglNGzwCtwJcZfmgZA3fCkjt9wJ+vu2fCay8rCJpuvgC4ysxOcvenhlmPjFMKBhlvthOcYwcgvEZwJMF59bh0h/8mc6atJThlM9/dHxrm+k4EfuPut0JfH1X8OQ0AAAFlSURBVBuHA7v6fWYyz3tzrQXebWaJnKOGE8P3rh9qMe7uBNc0HjWzK4E1BEczCoYioWCQ8eYh4ENmtpIgJC4n/xHDaPonwbfst5rZb4AOd28xs2uAa8I/7H8guLaxAsi6+w2DrO854D1mdiJBB/YfI+jOMjfsNgDLw7uiWgl6JuvveoILydeb2bcJ+i24muDaRPtQNizs/OVUgo5fthJcxJ5LEDpSJHRXkow3VxGEw6+BBwjOoT8RZwEedFJ/BcHF3q0EnZ8A/B/g8wSnhtYQXNR9J/Diflb5RYI7qO4lCJQ24Gf9lrmG4Jv/WoJAnDdAXWcQ/DH/O3ATQScwnxnG5u0m6Dr1boJOYr4OfMHdfzqMdcg4p456REQkQkcMIiISoWCQomdm3w+fhs73+n6h6xOJm04lSdEzs5nAlAFm73H3bXHWI1JoCgYREYnQqSQREYlQMIiISISCQUREIhQMIiIS8f8BG3nJU2Sc20oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# use first reward function initially and then switch to second reward function\n",
    "\n",
    "# Initialize learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new1]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions)\n",
    "\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdFUlEQVR4nO3de5hddX3v8fdn3+aW+4UQA5KgQQjWU3BALLaiULnogeexPgpHH2+0nFbx6KnaA8ejbakVtVorFQocRTjYgtT6aOSgeAFr7QFLAEESjIRwSQIkE3KdzG1fvuePtZLsNZlJZmBmzUzm83qe/cxev/Xba75rr5n12XutvX9LEYGZmdlehYkuwMzMJhcHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWUZuwSDpBklbJD0yzHxJukrSOkkPSzo5r9rMzGy/PN8x3Aicc5D55wLL09slwD/kUJOZmQ2SWzBExM+AbQfpcgHwfyJxLzBH0uJ8qjMzs71KE11AkyXAhqbpjWnbs4M7SrqE5F0FHR0drz7++ONzKdDM7HBx//33b42IhUPNm0zBoCHahhyvIyKuB64H6OzsjFWrVo1nXWZmhx1JTw03bzJ9KmkjcHTT9FHAMxNUi5nZtDWZgmEl8O7000mnATsj4oDDSGZmNr5yO5Qk6RbgDGCBpI3AnwNlgIi4FrgDOA9YB/QA78urNjMz2y+3YIiIiw4xP4AP5lSOmZkNYzIdSjIzs0nAwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8vINRgknSNpraR1ki4bYv5LJd0t6UFJD0s6L8/6zMwsx2CQVASuBs4FVgAXSVoxqNv/Am6LiJOAC4Fr8qrPzMwSeb5jOBVYFxHrI2IAuBW4YFCfAGal92cDz+RYn5mZkW8wLAE2NE1vTNua/QXwLkkbgTuADw21IEmXSFolaVVXV9d41GpmNm3lGQwaoi0GTV8E3BgRRwHnATdLOqDGiLg+IjojonPhwoXjUKqZ2fSVZzBsBI5umj6KAw8VXQzcBhAR9wCtwIJcqjMzMyDfYLgPWC5pmaQKycnllYP6PA2cCSDpBJJg8LEiM7Mc5RYMEVEDLgXuBB4l+fTRaklXSDo/7fZR4I8kPQTcArw3IgYfbjIzs3FUyvOXRcQdJCeVm9s+1XR/DXB6njWZmVmWv/lsZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLyDUYJJ0jaa2kdZIuG6bP2yWtkbRa0j/lWZ+ZmUEpr18kqQhcDfw+sBG4T9LKiFjT1Gc5cDlwekRsl3REXvWZmVkiz3cMpwLrImJ9RAwAtwIXDOrzR8DVEbEdICK25FifmZmRbzAsATY0TW9M25odBxwn6d8l3SvpnKEWJOkSSaskrerq6hqncs3Mpqc8g0FDtMWg6RKwHDgDuAj4qqQ5Bzwo4vqI6IyIzoULF455oWZm01mewbAROLpp+ijgmSH6fDciqhHxBLCWJCjMzCwneQbDfcByScskVYALgZWD+nwHeAOApAUkh5bW51ijmdm0l1swREQNuBS4E3gUuC0iVku6QtL5abc7geclrQHuBj4eEc/nVaOZmYEiBh/mn1o6Oztj1apVE12GmdmUIun+iOgcap6/+WxmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZIwoGSWVJz0k6cbwLMjOziTWiYIiIKlDlwGGyzczsMDOaQ0l/D1wuKbfLgZqZWf5Gs5P/XeD1wCZJjwB7mmdGxPlDPsrMzKaU0QTDVuBfxqsQMzObHEYcDBHxvvEsxMzMJocRn2OQdNdQ11+WNEvSXWNblpmZTZTRnHw+A6gM0d5Kcv7BzMwOA4c8lCTp5KbJV0na1jRdBM4GNo11YWZmNjFGco5hFcn3FwL44RDze4EPjWVRZmY2cUYSDMsAAeuBU0g+nbTXALAlIurjUJuZmU2AQwZDRDwFIOmzwCkRcW3zfEl/LGlJRHxynGo0M7Mcjebk87uAB4dofwB499iUY2ZmE200wXAE0DVE+1Zg0diUY2ZmE200wfA0Q38s9feAjWNTjpmZTbTRDIlxHfAlSRVg7xfazgSuBD431oWZmdnEGM2QGF+UtAC4iv1fdBsAvhwRnx+P4szMLH+jGkI7Ii6X9GlgBclHWNdERPe4VGZmZhNi1NdWiIg9wH3jUIuZmU0CvuazmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpaRazBIOkfSWknrJF12kH5vkxSSOvOsz8zMcgwGSUXgauBckm9OXyRpxRD9ZgL/DfhFXrWZmdl+eb5jOBVYFxHrI2IAuBW4YIh+fwV8HujLsTYzM0vlGQxLgA1N0xvTtn0knQQcHRG3H2xBki6RtErSqq6uoS4RYWZmL1SewaAh2mLfTKkAfAn46KEWFBHXR0RnRHQuXLhwDEs0M7M8g2EjcHTT9FHAM03TM4FXAj+V9CRwGrDSJ6DNzPKVZzDcByyXtCy92M+FwMq9MyNiZ0QsiIilEbEUuBc4PyJW5Vijmdm0l1swREQNuBS4E3gUuC0iVku6QtL5edVhZmYHN+rrMbwYEXEHcMegtk8N0/eMPGoyM7Msf/PZzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaWkWswSDpH0lpJ6yRdNsT8P5W0RtLDkn4i6Zg86zMzsxyDQVIRuBo4F1gBXCRpxaBuDwKdEfEq4FvA5/Oqz8zMEnm+YzgVWBcR6yNiALgVuKC5Q0TcHRE96eS9wFE51mdmZuQbDEuADU3TG9O24VwMfH+oGZIukbRK0qqurq4xLNHMzPIMBg3RFkN2lN4FdAJ/M9T8iLg+IjojonPhwoVjWKKZmZVy/F0bgaObpo8CnhncSdJZwCeA10dEf061mZlZKs93DPcByyUtk1QBLgRWNneQdBJwHXB+RGzJsTYzM0vlFgwRUQMuBe4EHgVui4jVkq6QdH7a7W+AGcA/S/qlpJXDLM7MzMZJnoeSiIg7gDsGtX2q6f5ZedZjZmYH8jefzcwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7OMXL/HYIenbXsGeGjDDma0llgyp43u/hodLSU27+qjvVJk6fwOWkoFeqt1brtvA7v6auzoqVKtN9jZW6WlVKBSKrBoVis9A3X6qnVmtZaY017hJXNa2dVbY2ZriUJBCGgpFwEoShw5u4XnuwfoaCkxs7XEnLYK3QM1IoL+WgMBxUIyTFd3f43dfTX29NdoLRepNYLegRpz2yvM66jQ0VKiICHB/I4KpeLkfd3UX6uzs6cKgu17quzuqxJAqSAGasnzuva53Ty3q49GBL0DdSQxt71CqShKheTW0VKiWm/QM1CnWBC7+2r01+r0DNTprzYoFkSxIHqrdRqNQILeap1qLdiyu4/u/hrtlRKVUoGOSpGWcpGiRLXeoBFBuVigWBD9tQYtpQKt6bZrrxQpFcSegTotpQLlYoGBeoOIQIidvVV6q3UE9NXq9FXTeUpq7q/W0zoaFAqiXCxQSbdXENTqSa191aQOAQP1oKVUYHZbmVqjQSFdVqkgZrYmu8Ln9wxQKoiCRKkoioUClaJoKRVpRNAzUKejpcjstjKVYoFdfTXKRREBreUiHS0ligUQoh5BvRH0V+sM1BtISb9SQbRVinT316jVG/TXGpTT2stF0bW7n129NQoFqJSKdO3up79Wp1JMnqdyUfRVG5SL4k/OeBnnvHLxmP99ORhydOUdj3Ldz9Zz9omLuOadr963wxrKQK2R/IEepM94aTSCX27cwZGzWnn02V3c8/jzlEsFFsxoob9W5/4nt7NpRy8vndfOmmd3sXF77wv6PW3lIpU0FHoH6nT314BkR15vDDm+Ym6K6Y6zvVKkrVykp1pnXkdlX8j1VeuUigV29lQpFcXCmUlAFQqitVSgEUFX9wDVWoPd/VXayyWOmNVCpVhgbkeFOW1lWstFigURad/1Xd3MbivTV0t2ggtntDCvo0Jfrc4TW/ewbU+VHT0D7Oyt0jNQH9F6zG0vUyoWKBeEJLbtGdi3w2p+jvc+5zNaSrSWi7RVCpQLBSSoNWLfTqkRQXulSCPgxJfMZlZbmT39Nar1BnsG6vRX69QayY6uUIBqPeir1mmrFOkdqLOzt5rsLGsNBmoNZqTBNFBvUCkmvw9gVls5+T0NmNFaor2SBEoE7O6rsXBGC22VIi2lAo1GUG0kv0eABKVCgSCS51giSHa6/dUkNEtF0QjY01+jVg827egDYGZriaJEI4K+aoN6o0613kiWLdFRKbJpR50dPVUGanVmtpb3PY99tTp7+ms0AiKCgpJQbU3/zpNgTZbd3VdjVlvyQqStUqS/2qCgJLwWzKgwp71MowE7ewZYNKslDcMG9UYSJPM6KmnQFcfsb76ZIib2H/DF6uzsjFWrVk10GQf1ms/8mM27hh4P8JwTj+SRZ3bu27m+93eWUi6K//1vTwDwllctZsncNp7Z0Ud/tc6py+bxmmXzufZnj/PhM5dz3KKZo6qla3c/X//3J+ivNXjDK47gqW17+PYDm7j/qe0AtJYL9FUbB13G3PYyvdU6c9srHDm7ldNftoDli2bQUiry7M5eIqAg9v3TPL2th+09A5SLBc46YREnLJ7JvI4K0v7Q2/sKf+8rzIFag827+tiwvWffq6l6I/ln66vWaUTQiGBnb5VCupz+WoNdvVUqpQIFJf+QAnoGarSUivTXG7xkdiuz28pU60Gt0aC9UmJ3X5Vnd/ZRTV/VVWsNtnb301ut019rsKe/xoyWElu7+3nq+R529VaZ2VoGkp1WvRFs7e5nXkdl3w6lKDF/RoWWUoGZrcnztWV3PwO1Bs9397OjN3nHtPffr61c5PjFM9nZU6Ul3ZFs3d3Ptj0DFATLFnYwv6OF+R0V5nZUmNlaYl5HBYA57UnQAFTrDVpKRWa2llg6v4PZ7eUht2Gk4dBbrVMuJq/kk+eXzHaxw5ek+yOic8h5Doax8+DT2zn+yFls2tHDLf+xga/9/IkD+nzsTcdx7/pt/Hzd1jH93ee+8kjaKkV+uHoz3f01jl3QwdHz2tneM8DmXX3Mbivz5NYeBurD7/SbQ+GUpXNZOLOFo+e184EzXk5rucD2PVV6BmocM7/joO927ND2Hhap1ZNDGgd7Z7i3r9lYcjCMoV8/t4sjZ7Uyp72yr219Vzdv/OK/HvRxN198KqcdO3/fq1+AHT0D/PX/fZSegTofeMPLaC0X+fGazVz5/V9zw3s7eePxi/jBI8/yx994gLNOWMRjW3ZTrTWolJJX1e2VEr/atHPU6/D5t72Ks1ccycqHNvH0th7OOmERpy6b552P2TTiYBgjjz67i3O//G8AfPatv8W5r1zMO792L49s2jXsY5648rxx3+E2GsGaZ3fxl99bzVtPPooLTzkaSezpr9FeKaYnvfyq08z2czCMgb5qneM/+YOD9rn54lN57bHzD3lowMxsoh0sGPyppGFc9ZPH+Pu7HqNaPzA4l8xpY9OO/Z/EOeuERXzlv5y076N4ZmZTmYOhyb3rn+fVx8ylv9bgb3/0myH7fPANL+PjZx+fc2VmZvlxMKRu+PkTXHH7Gs54xUI+ctZxQ/a59ZLT6Dxmbs6VmZnly8GQ+uZ9GwD46douPvTG5QB0HjOX952+jDe/auy/WWhmNllN+2D4yaObufim/Sev28pFdvYOAPA/zj2eU5bOm6jSzMwmxOQdDCYnzaEAyTgw778xaatM4rFyzMzGi/d8B/HyI2ZMdAlmZrmb1sHwwNPbM9Nv/q3suYSOlml/pM3MpqFpved76zX/b9/9E18yi4+f/Qq+9I7f5jsPbuKtJy+ZwMrMzCbOtA6GvU5+6Ry+/YHT902//ZSjJ7AaM7OJNa0PJe3Ve4hhps3MphMHA7B9z8BEl2BmNmlM62BoLServ73HwWBmtte0DoZFs1qB5MpfZmaWmNbB8NJ57RNdgpnZpDOtg6GlNK1X38xsSNN6z9hIL7Vw0/tPndhCzMwmkWkeDMF/Omo2rz9u4USXYmY2aUzzYMDXQTYzG2RaB0NE4Eszm5llTetgaERQ8DsGM7OMXINB0jmS1kpaJ+myIea3SPpmOv8XkpaOZz2NBg4GM7NBcgsGSUXgauBcYAVwkaQVg7pdDGyPiJcDXwI+N1719AzU6Orux7lgZpaV5zuGU4F1EbE+IgaAW4ELBvW5ALgpvf8t4EyN09nhm+95inVbuqn4uwxmZhl5Dru9BNjQNL0ReM1wfSKiJmknMB/Y2txJ0iXAJelkt6S1L7CmBU/B1m/84Qt89NS0gEHP5zTgdZ4evM6jc8xwM/IMhqFe+ccL6ENEXA9c/6ILklZFROeLXc5U4nWeHrzO08N4rXOex1E2As1XwDkKeGa4PpJKwGxgWy7VmZkZkG8w3Acsl7RMUgW4EFg5qM9K4D3p/bcBd0XEAe8YzMxs/OR2KCk9Z3ApcCdQBG6IiNWSrgBWRcRK4GvAzZLWkbxTuHCcy3rRh6OmIK/z9OB1nh7GZZ3lF+RmZtbMn9U0M7MMB4OZmWVM22A41PAcU4WkoyXdLelRSaslfThtnyfpR5IeS3/OTdsl6ap0vR+WdHLTst6T9n9M0nuG+52ThaSipAcl3Z5OL0uHUnksHVqlkrYPO9SKpMvT9rWSzp6YNRkZSXMkfUvSr9Pt/drDfTtL+u/p3/Ujkm6R1Hq4bWdJN0jaIumRprYx266SXi3pV+ljrhrRl4YjYtrdSE5+Pw4cC1SAh4AVE13XC1yXxcDJ6f2ZwG9Ihhz5PHBZ2n4Z8Ln0/nnA90m+M3Ia8Iu0fR6wPv05N70/d6LX7xDr/qfAPwG3p9O3ARem968F/iS9/wHg2vT+hcA30/sr0m3fAixL/yaKE71eB1nfm4A/TO9XgDmH83Ym+cLrE0Bb0/Z97+G2nYHfA04GHmlqG7PtCvwH8Nr0Md8Hzj1kTRP9pEzQhngtcGfT9OXA5RNd1xit23eB3wfWAovTtsXA2vT+dcBFTf3XpvMvAq5ras/0m2w3ku/B/AR4I3B7+ke/FSgN3sYkn4R7bXq/lPbT4O3e3G+y3YBZ6U5Sg9oP2+3M/pEQ5qXb7Xbg7MNxOwNLBwXDmGzXdN6vm9oz/Ya7TddDSUMNz7FkgmoZM+lb55OAXwCLIuJZgPTnEWm34dZ9qj0nfwf8GdBIp+cDOyKilk43158ZagXYO9TKVFrnY4Eu4Ovp4bOvSurgMN7OEbEJ+ALwNPAsyXa7n8N7O+81Vtt1SXp/cPtBTddgGNHQG1OJpBnAvwAfiYhdB+s6RFscpH3SkfQWYEtE3N/cPETXOMS8KbPOJK+ATwb+ISJOAvaQHGIYzpRf5/S4+gUkh39eAnSQjM482OG0nQ9ltOv4gtZ9ugbDSIbnmDIklUlC4R8j4ttp82ZJi9P5i4Etaftw6z6VnpPTgfMlPUkySu8bSd5BzFEylApk6x9uqJWptM4bgY0R8Yt0+lskQXE4b+ezgCcioisiqsC3gd/h8N7Oe43Vdt2Y3h/cflDTNRhGMjzHlJB+wuBrwKMR8bdNs5qHF3kPybmHve3vTj/dcBqwM32reifwJklz01dqb0rbJp2IuDwijoqIpSTb7q6IeCdwN8lQKnDgOg811MpK4ML00yzLgOUkJ+omnYh4Dtgg6RVp05nAGg7j7UxyCOk0Se3p3/nedT5st3OTMdmu6bzdkk5Ln8N3Ny1reBN90mUCT/acR/IJnseBT0x0PS9iPV5H8tbwYeCX6e08kmOrPwEeS3/OS/uL5IJJjwO/AjqblvV+YF16e99Er9sI1/8M9n8q6ViSf/h1wD8DLWl7azq9Lp1/bNPjP5E+F2sZwac1JnhdfxtYlW7r75B8+uSw3s7AXwK/Bh4Bbib5ZNFhtZ2BW0jOoVRJXuFfPJbbFehMn7/Hga8w6AMMQ908JIaZmWVM10NJZmY2DAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmB2CpKWSQtKkutD8ZK3Lpj4Hg9mhbSAZjOyXAJLOSHfIC/IqQNJPJX3lYHWZjZXcrvlsNlVFRB14bjyWLakcyXAPozaeddn05ncMNqmkr4yvkfQZSVvTC5h8QVIhnf+kpI8N8ZivNE0/KelTkm6UtFvSBknvUHKhm1sldacXM3nTCGvad8gmHcH27nRWV9p+Y9pPkv5M0uOSetOLo7xriOVcJOkuSb3Af5U0X8lFaDamj1st6X1Nj7sReD3wwfTxkS7rgENJkn5PyUVq+iRtlvSldNiXET2/aZ+3KrkITK+kbZL+VdKikTxXdnhwMNhk9E6gRjJg2qXAR4B3jHIZHyEZFuFkkgu73ERyUZ87SIaW+BnwDUmto1zuBuAP0vsnkhzK+XA6/WmS4Qw+SHJxmCuB6yS9edAyrgSuSft8h2QohweAt6TL/HL6uDPT/h8G7gG+nv6+xWSHWAZA0hKSC7E8SDL8+sUk4+9fOajrsM+vpCNJBia8CTiB5CIyN4/gebHDyUSPE+Kbb8034KfAPYPafgR8Nb3/JPCxIR7zlabpJ4FbmqZnkIwndVVT29K0rXMENWX6kozPFMCCpj4dQC/wu4Me+3fAHYOW89ER/M5b967zUOs4TF1/TTJOTqGpz3uBfqB9hM/vyekyj5novwXfJu7mcww2GT08aPoZ9l+oZNTLiIhuST0kg47ttTn9OdrlDmcFySv/H0hqHoCsTBJUzVY1T0gqklxb4R0kF1FpIbl0509HWcMJJDv9RlPbz9NlvZz9z8nBnt+HgB8Dj0j6YXr/WxHRNcpabApzMNhkNPhkbLD/sGeDAy8+Uh7hMqqDpmHsDqfuXc5/Jhku+mC17Bk0/THgoySHjH4FdAOfYfShJYa/CEtz+7DPb0TU03Mvp5EM3XwxcKWk10fEQ6Osx6YoB4NNNV0kx9gBSM8RHE9yXD0vA+nPYlPbGpJDNsdExF2jXN7rgO9FxM2w7xobxwE7Bv3O4hCPbbYGeLukQtO7htelj318pMVERJCc07hH0hXAapJ3Mw6GacLBYFPNXcD7Ja0kCYlPMPQ7hvH0FMmr7DdL+h7QGxG7JX0B+EK6Y/8ZybmN04BGRFx/kOX9BniHpNeRXMD+QySXs2wOuyeBU9NPRXWTXJlssGtITiRfI+nLJNct+CzJuYmekaxYevGXs0gu/LKZ5CT20SShY9OEP5VkU82VJOHwXeCHJMfQH8izgEguUv/nJCd7N5Nc/ATgk8BfkBwaWk1yUvcPgCcOschPk3yC6vskgbIH+MdBfb5A8sp/DUkgvnSYus4l2Zn/EriB5CIw/3MUq7eT5NKpt5NcJOaLwF9FxDdGsQyb4nyhHjMzy/A7BjMzy3Aw2LQn6dr029BD3a6d6PrM8uZDSTbtSToCmDXM7F0RsSXPeswmmoPBzMwyfCjJzMwyHAxmZpbhYDAzswwHg5mZZfx/kASadwlUyuwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Do not learn\n",
    "# use first reward function initially and then switch to second reward function\n",
    "\n",
    "# Initialize learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new1]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions, should_learn=False)\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 3\n",
    "In this scenario we are going to start rewarding actions that have never seen a reward previously when we change the cost function. \n",
    "\n",
    "### Tom\n",
    "| | `get_cost` | `get_cost_new2` |\n",
    "|---|---|---|\n",
    "| **Morning** | Politics |  Politics|\n",
    "| **Afternoon** | Music |   Food |\n",
    "\n",
    "### Anna\n",
    "| | `get_cost` | `get_cost_new2` |\n",
    "|---|---|---|\n",
    "| **Morning** | Sports | Food|\n",
    "| **Afternoon** | Politics |  Food |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost_new2(context,action):\n",
    "    if context['user'] == \"Tom\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'politics':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'food':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE\n",
    "    elif context['user'] == \"Anna\":\n",
    "        if context['time_of_day'] == \"morning\" and action == 'food':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        elif context['time_of_day'] == \"afternoon\" and action == 'food':\n",
    "            return USER_LIKED_ARTICLE\n",
    "        else:\n",
    "            return USER_DISLIKED_ARTICLE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With learning\n",
    "Let us now switch to the third reward function after a few samples (running the first reward function). Recall that this reward function changes the preferences of the users and is working with a **different** action space than before. We should see the learner pick up these changes and optimize towards the new preferences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcZZ3v8c+vtu5Od2ftANkgYVE2RTAiKAqKKIuCM+MojIoLXsYZ9+V6QR0XBsUFRbkuwHWBcVREnZdGjBdUUFzQIQIiBAMhBNIEsy+91vqbP86ppE7ldNIduqq6+3zfr1e9us5Sp59Tp7u+9TzPOc8xd0dERKQq1eoCiIjIxKJgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiWhaMJjZN8xso5ndP8JyM7OrzWy1md1nZic0q2wiIrJbM2sM1wNn7mX5WcAR4eNi4KtNKJOIiNRpWjC4+x3A1r2sch7wHx74AzDTzOY1p3QiIlKVaXUBaiwA1tVM94bznqxf0cwuJqhV0NnZ+ewjjzyyKQUUEZkq/vSnP21297lxyyZSMFjMvNjxOtz9OuA6gKVLl/qKFSsaWS4RkSnHzB4badlEOiupF1hUM70QWN+isoiIJNZECoZlwIXh2UknATvcfY9mJBERaaymNSWZ2XeB04AeM+sFPgpkAdz9GmA5cDawGhgE3tSssomIyG5NCwZ3v2Afyx14W5OKIyIiI5hITUkiIjIBKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg0wIA/kSA/kS7j7u265UnHypzHCxzP1P7GDrQGHcf4fIVJJpdQFk4qlUnHvWbeOaX69hRkeWFWu3Mlgok04Z2wYLDBcrdGTTdLVn2DpQ4NmHzKI9m2a4WOaUw3s4cHobc7vbOHb+DA6Y3r5ru9sGCqx8ciePbRmk4s7DG/r4yxM7WLtlcNeHdU9XjnOeMY/jD55FueJs6Bumd9sQOwaL9IfhYQbu0JZN0ZHNsGUgT09XWxgqRmdbmnyxwub+PL3bhtg+FJS51pKeTrJpY0ZHlnVbh0inDIAtA3ly6RSdbRkGC2WOXTCdxXM6KZYrpFNGf75MLp2iuz1DLpPisS0DDBcrtGdTDBbKFEoVtg0WGMiXmZZL88Gzj+JFRx7QtGMnMh6sEd/Qmmnp0qW+YsWKVhdjUhrIl/Dw592PbeNbf3iM3z+yZVSv7elqI18s05cv7XW9jmyaYrmCA+VK9G+tLZPiGQtmMH9mB/NndtCWSfHA+h384sGNkfVymRRtmRQzp2XpzGXIplOkUka+WKZQrjCnM8eTO4Zpy6QwMwbyJVJmHDSjnYNnT2N2Z45sOthGT1ew7sMb+ymWK+wYKjJ/RgcW5AJzu9voHy5RrjjDpTIPb+ind9sQ3e0ZKg7t2RTFcoXN/QXSZiya3UGhXCGbTtHdniWTMmZ35piWS3P349tYv32Yz/3jcbzy+AWjPSwiTWFmf3L3pXHLmlpjMLMzgS8CaeBr7v6puuUHAzcAM8N1LnH35c0s41S0pT/Plbc+xOb+POu3D/HA+p37fM0bn7eYQ+ZM4wVHzGXR7A5KZSdlRkcuHbt+KfyQdeDBJ3eyqS/Pbx/ezEChxLaBIocd0Mm8GR08c+EMFs/pxAwOnN5Oe3bP7a3dPMCm/jxdbRnmzWinuz276xv9ROHuuENqL+Xqz5d4yw138Z6b7mXrQIE3PX8xZhNrP0TiNK3GYGZp4CHgDKAXuAu4wN1X1qxzHXCPu3/VzI4Glrv74r1tN2k1hk19eUqVCrOm5dg5VOQ3D2/m9lUbOWredP7u+AXMn9kBwPbBAtf8eg3X//7RPZpRqtIpozOXJp0yTj/qQN566qEcfkB3M3dnyhsulnn7d+7hFw9u4I3PW8y/vfzoCRdyMnoD+RIVdwYLZbYOFHCHzrbgy03KjI19w+wcLjGYL9ORS1GpwPahIikLvigUSpVdtdpSOfi/7M+XmdvdxrRcmnLFcWBaLk2+WCYd1nRLZadQKjOnq43u9gzDxTIVh2Pnz+DgOdP2a18mSo3hRGC1u68JC3UjcB6wsmYdB6aHz2cA65tYvgml+o30Vw9t5M3X7zv4br7vST57y6rYZcfMn85zFs/m1KfPJZMyTjm8R99cm6Q9m+ba1z+by3+6km/+bi0PPrmTz7/mWSwIA1x2K5QqbOrPs3OoSLnipFOGO1TcWb99iM39BfrzRUoVZ0ZHFvcgeEsVJ2XBB7M7FMoVcukUw8UyxXKFQtnZ3J9nS3+e/nwJw+jLl8imjZRZ8AWpLUOpXGGgUGa4UGagUGKwUCZlMFQo05FLM5AvM1Qsj/t+d2TT+73dy195LK+bc8g4l6i5wbAAWFcz3Qs8t26djwG3mtk7gE7gJXEbMrOLgYsBDj744HEvaDO987v3sOzP6znvWfP58b1jz8FXPms+C2Z18LvVW7h33fbIss+/+jjOOPpAutuz41Vc2Q/plPHRVxzDkQd18/GfrOTsL/6Gy847hvOeNTH6Hdydigd9QBV3ShXnye1DlCpOWyZFueK7mgnzxQqdbeld84rlCn3DJTrbMvTnS7RlUhwyp5OKO5WKk0mn2NSXp1CqsGUgz47BIk/uHKZvuMSOoSIbw+d9w0X68iUa0YCRTRs9XW3M6crR1ZbBHRbMbKdUcUplx3G2DxbIplNMb89wQHcbXW0ZpuXSVNzDEysqdObS9HS3YUBnW4bZnTkA+odLYIDDAdPb6G7P0tmWZqhQxsyYNS1LxaGrLThhAYdpbWkyYc3RzOjPlxgulkmbYQZDxTJtmTSFUoVi2IeVy6TYsHOYwUKJ9mxQ0z+wu33kHX8KmtmU9I/Ay9z9LeH064ET3f0dNeu8NyzT58zsZODrwLHuHt8WwuRsSnp8yyAv/Ozto1r3wpMP4RXHzefxLYOcccyBTN/Lh3ztsVSNYGJau3mA99x0L/c8vp2LTlnCv5x2GD1dbft8Xbni9G4bZMPO4Bt1qRL06XS3ZylVnC39eTb25Xli2xBldwj7P4IPfd/1LblQKrNxZ56KO7lMioF8mS0D+RGbGxth1rQsM6fl6G7PcEB3O11taWZOyzGjI8u8Ge10tWfIpVPkS8FJC2kzFs7qYG53G13tGdJm9A0HZ6e1Z9Jk0kbFgyYYA7Lp4ASB9mzw4av/hXgTpSmpF1hUM72QPZuKLgLOBHD3O82sHegBNjJFPLShj5dedUfssrecsoQLT15MJm3Mm9Ee+YN+zuLZ+9y2/gEmvsU9nXzv4pO55If38fXfPsr1v1/LcQtnsHDWNOZ05Vi/fYhi2dk6UKBvuMiOoRKDhRLFcoViee9f4rLp4Ewsw0hZ8Pfg7rs6yNsyaXJp46j500mZUSiV6cxlmNWZozOXJptOYRZ8sB44vZ1cJsVQoUwqBbOm5XCCD+L+fIlM2pjZkSWXSdEVntrbmcuwc7jIkzuGyaQMJ2gO7elqI5tO0dMdfPi3ZeJPYBiLzra9f3TFndQgo9fMYLgLOMLMlgBPAOcD/1S3zuPA6cD1ZnYU0A5samIZG6JQqvDSq37NB848kn/99t0AvOrZC5nRkeXD5xylD/SEyWVSfO7Vx3Hh8xbzkz+vZ8Xarfxu9Wb68yV6uoJOyJnTshw2t4sZHdngG3QmxeI5nczpzDG3O/igndGRZcdQkZQZB05vY0ZHlky69desHrtgRquLIE9RU69jMLOzgS8QnIr6DXf/hJldBqxw92XhmUj/D+gi6Ij+gLvfurdtTuSmpFK5wptvWMEdD0Wz7cKTD+Gy845tUalERCZOUxLhNQnL6+Z9pOb5SuD5zSzTeLj6lw/z+0c2c+3rlzKjI+gDWLd1kBd8Zs9+hIWzOhQKIjKhaUiMp+jOR7bw+Z8/BMBxH7+VD59zFIvndPKW/9hdi3ntcw/mE3/3DNZs6mdJT2eriioiMioKhv3w85UbeGhDX+x1A5f/9MHI9JpPnr2r8+/QuV1NKZ+IyFOhYBiDvuEiz/vUbfQN7zk+0COfPJvDPri7lezpB3Zzy3te2MziiYiMi9afwjCJPONjt8aGwtpPnUM6Zaz91DmcuHg2C2Z28LN3vaAFJRQReepUY9hPt7//tNj+gpveenILSiMiMn4UDKPwf35wH99bEYzmcdaxB/HV1z27xSUSEWkcNSWNQjUUAF5x3PwWlkREpPEUDPtQHRq36sxjDmpRSUREmkPBsA/P/Hhw4fWLjzyAVZefudcbs4iITAUKhn0YLATjpH/83GPGZfAvEZGJTsGwF7XjSC2avX93SRIRmWwUDHuxqS8PwD89d3LfDEhEZCwUDHtx4id/CajDWUSSRcEwgpXrd+56vmCW7s8rIsmhYBjB2Vf/ZtfzwzT4nYgkiIJhH67RVc4ikjAKhhi1ZyMdu2B6C0siItJ8CoYYm/sLALzz9CNYOEunqYpIsigYQpWK8+jmAQA+uux+APKlciuLJCLSEgqG0MXfWsGLrvwVj20ZYFFYS7jw5MWtLZSISAsoGEK/eHAjEDQjXXvHGgAWzNRpqiKSPAqGOt/54+OtLoKISEvpRj11fnh3Lz1dOU45vKfVRRERaQnVGIiengqwfbDIfDUjiUhCKRiA4WL0ZjylitPT1dai0oiItJaCAejLF/eY90DNWEkiIkmiYAD6hkt7zHvBEepjEJFkUjAA2weDK51r77vwoqcf0KriiIi0lIIBeN3X/huA5y6ZvWve9A6dsCUiyaRgAIaKwdAXvduGds0zs1YVR0SkpRQMNc489iCes3gW82a0t7ooIiItk9j2EndnyaXL+fA5R/GGkw/hhjsf47C5XXz/rc9rddFERFoqsTWG6jVtl//0QQYLZdUSRERCyQ2GmueDxTIduXTLyiIiMpE0NRjM7EwzW2Vmq83skhHWebWZrTSzB8zsO40qSyWsMpjBYL7ENAWDiAjQxD4GM0sDXwbOAHqBu8xsmbuvrFnnCOBS4Pnuvs3MGnYxQe3wSIOFMtOyie1uERGJaGaN4URgtbuvcfcCcCNwXt06/wv4srtvA3D3jY0qjNc0Jg2pKUlEZJdmBsMCYF3NdG84r9bTgKeZ2e/M7A9mdmbchszsYjNbYWYrNm3atF+FqdYYjLDGoGAQEQGaGwxxV4x53XQGOAI4DbgA+JqZzdzjRe7XuftSd186d+7c/SpMbVPScLFMR1bBICICzQ2GXmBRzfRCYH3MOj9296K7PwqsIgiKcVfblJQvVWjLJvYELRGRiGZ+Gt4FHGFmS8wsB5wPLKtb50fAiwDMrIegaWlNIwpTqakx5Itl2jKqMYiIQBODwd1LwNuBW4AHgZvc/QEzu8zMzg1XuwXYYmYrgduB/+3uWxpUnl3Ph1VjEBHZpannaLr7cmB53byP1Dx34L3ho6GqNYaKQ6FUUY1BRCSU3K/Jdd3ej2zqb005REQmmMQGg9clw86hPW/vKSKSRMkNhroaQy6d2LdCRCQisZ+G9RdQ5DKJfStERCIS+2nodVUGBYOISCCxn4b1NYaDput+DCIikOBgqNTVGN5zxtNaVBIRkYklscFQX2Vo11hJIiJAgoOhvilJREQCowoGM8ua2d/M7JhGF6hZ6k9XFRGRwKiCwd2LQJEp9EW7/gI3EREJjKUp6f8Cl5rZlLgHpmoMIiLxxvIh/wLgVOAJM7sfGKhd6O7nxr5qglIuiIjEG0swbAZ+2KiCNFv9BW4iIhIYdTC4+5saWZBmUy6IiMQbdR+Dmd0Wd/9lM5tuZreNb7FERKRVxtL5fBqQi5nfTtD/MKmoxiAiEm+fTUlmdkLN5DPNbGvNdBp4GfDEeBes0XS6qohIvNH0MawgOInHgVtjlg8B7xjPQjVDbY3hn194aOsKIiIywYwmGJYABqwBnkNwdlJVAdjo7uUGlK2hagfRu/Tso1pYEhGRiWWfweDujwGY2aeA57j7NbXLzeytZrbA3f+tQWVsCDUkiYjEG0vn8+uAe2Lm3w1cOD7FaR51PouIxBtLMBwAbIqZvxk4cHyK00xKBhGROGMJhseJPy31hUDv+BSneVRjEBGJN5YhMa4FrjKzHFC9oO104Arg0+NdsEZTLoiIxBvLkBifM7Me4Gp2X+hWAL7o7p9pROEaqVpjeMlRk7AVTESkgcY0hLa7X2pmlwNHE5zCutLd+xtSsgarXuD29ycsaHFJREQmljHfW8HdB4C7GlCWpqrWGKy1xRARmXCSe8/najAoGUREIpIbDLu6n5UMIiK1khsMqjGIiMRKbDBUpZQMIiIRiQ2G6iB6igURkajEBoOakkRE4iU3GMKfCgYRkaimBoOZnWlmq8xstZldspf1XmVmbmZLG1UW39WUpGQQEanVtGAwszTwZeAsgiunLzCzo2PW6wbeCfyxkeXZNVaSckFEJKKZNYYTgdXuvsbdC8CNwHkx6/078BlguJGF0ZXPIiLxmhkMC4B1NdO94bxdzOx4YJG737y3DZnZxWa2wsxWbNoUd4uI0fDqtvbz9SIiU1MzgyHuE3h3i45ZCrgKeN++NuTu17n7UndfOnfu3P0qjGoMIiLxmhkMvcCimumFwPqa6W7gWOBXZrYWOAlY1qgOaJ2VJCISr5nBcBdwhJktCW/2cz6wrLrQ3Xe4e4+7L3b3xcAfgHPdfUUjCrO7xqBkEBGp1bRgcPcS8HbgFuBB4CZ3f8DMLjOzc5tVjpryAKoxiIjUG/P9GJ4Kd18OLK+b95ER1j2toWUJfyoXRESiEnvl8y5KBhGRiMQGg/u+1xERSaLEBkOVOp9FRKISGwy77+AmIiK1EhsMVTorSUQkKrnBoAqDiEis5AZDSBUGEZGoxAaDKgwiIvESGwxVGl1VRCQqscGg6xhEROIlNhiqVGEQEYlKbDDoOgYRkXiJDYYqVRhERKISGwzqYxARiZfYYKhSH4OISFRig0EVBhGReIkNht1UZRARqZXYYHB1MoiIxEpsMFSpj0FEJCrxwSAiIlGJDQY1JImIxEtsMFSpJUlEJCq5waAqg4hIrOQGQ0jDbouIRCU2GDSInohIvMQGQ5XqCyIiUYkNBl3fJiISL7HBUKUuBhGRqMQGg2oMIiLxEhsMVaZeBhGRiMQGgyoMIiLxEhsMVepjEBGJSmwwaNhtEZF4iQ0GERGJ19RgMLMzzWyVma02s0tilr/XzFaa2X1m9kszO6RRZVF9QUQkXtOCwczSwJeBs4CjgQvM7Oi61e4Blrr7M4EfAJ9pfLka/RtERCaXZtYYTgRWu/sady8ANwLn1a7g7re7+2A4+QdgYaMKoy4GEZF4zQyGBcC6munecN5ILgJ+FrfAzC42sxVmtmLTpk1PqVC6jkFEJKqZwRD3CRz7vd3MXgcsBT4bt9zdr3P3pe6+dO7cuftZHFUZRETiZJr4u3qBRTXTC4H19SuZ2UuADwGnunu+0YVSH4OISFQzawx3AUeY2RIzywHnA8tqVzCz44FrgXPdfWMjC6M+BhGReE0LBncvAW8HbgEeBG5y9wfM7DIzOzdc7bNAF/B9M7vXzJaNsLlxoxqDiEhUM5uScPflwPK6eR+pef6SZpZHRET2lNgrn9WSJCISL7HBUKXTVUVEohIbDOp8FhGJl9hgqFLns4hIVGKDwdXLICISK7HBUKUKg4hIVGKDQX0MIiLxEhsMVepjEBGJSmwwqMIgIhIvscGwm6oMIiK1EhsMrk4GEZFYiQ2GKvUxiIhEJT4YREQkKvHBoAqDiEhUYoNBXQwiIvESGwxVpk4GEZGIxAaDxkoSEYmX2GCoUn1BRCQqscGgPgYRkXiJDYYqdTGIiEQlNhhUYxARiZfYYKjSPZ9FRKISHwwiIhKV2GBQS5KISLzEBkOVOp9FRKISGwwadltEJF5ig0FEROIlNhhUXxARiZfYYKhSH4OISFRyg0FVBhGRWMkNhpCG3RYRiUpsMGjYbRGReIkNhirVF0REohIbDLqMQUQkXmKDoUpdDCIiUU0NBjM708xWmdlqM7skZnmbmX0vXP5HM1vcqLKowiAiEq9pwWBmaeDLwFnA0cAFZnZ03WoXAdvc/XDgKuDTjSrPYKEclEu9DCIiEc2sMZwIrHb3Ne5eAG4Ezqtb5zzghvD5D4DTrUHnk/6ldzsAmbSCQUSkVqaJv2sBsK5muhd47kjruHvJzHYAc4DNtSuZ2cXAxeFkv5mt2s8y9cz9dHTbCdAD2ucE0D4nw1PZ50NGWtDMYIj7al7f1D+adXD364DrnnKBzFa4+9Knup3JRPucDNrnZGjUPjezKakXWFQzvRBYP9I6ZpYBZgBbm1I6EREBmhsMdwFHmNkSM8sB5wPL6tZZBrwhfP4q4DbXjRNERJqqaU1JYZ/B24FbgDTwDXd/wMwuA1a4+zLg68C3zGw1QU3h/AYX6yk3R01C2udk0D4nQ0P22fSFXEREaiX+ymcREYlSMIiISERig2Ffw3NMFma2yMxuN7MHzewBM3tXOH+2mf3czB4Of84K55uZXR3u931mdkLNtt4Qrv+wmb1hpN85UZhZ2szuMbObw+kl4VAqD4dDq+TC+SMOtWJml4bzV5nZy1qzJ6NjZjPN7Adm9tfweJ881Y+zmb0n/Lu+38y+a2btU+04m9k3zGyjmd1fM2/cjquZPdvM/hK+5upRXTTs7ol7EHR+PwIcCuSAPwNHt7pc+7kv84ATwufdwEMEQ458BrgknH8J8Onw+dnAzwiuGTkJ+GM4fzawJvw5K3w+q9X7t499fy/wHeDmcPom4Pzw+TXAv4TP/xW4Jnx+PvC98PnR4bFvA5aEfxPpVu/XXvb3BuAt4fMcMHMqH2eCC14fBTpqju8bp9pxBl4InADcXzNv3I4r8N/AyeFrfgactc8ytfpNadGBOBm4pWb6UuDSVpdrnPbtx8AZwCpgXjhvHrAqfH4tcEHN+qvC5RcA19bMj6w30R4E18H8EngxcHP4R78ZyNQfY4Iz4U4On2fC9az+uNeuN9EewPTwQ9Lq5k/Z48zukRBmh8ftZuBlU/E4A4vrgmFcjmu47K818yPrjfRIalNS3PAcC1pUlnETVp2PB/4IHOjuTwKEPw8IVxtp3yfbe/IF4ANAJZyeA2x391I4XVv+yFArQHWolcm0z4cCm4Bvhs1nXzOzTqbwcXb3J4ArgceBJwmO25+Y2se5aryO64Lwef38vUpqMIxq6I3JxMy6gB8C73b3nXtbNWae72X+hGNmLwc2uvufamfHrOr7WDZp9pngG/AJwFfd/XhggKCJYSSTfp/DdvXzCJp/5gOdBKMz15tKx3lfxrqP+7XvSQ2G0QzPMWmYWZYgFL7t7v8Vzt5gZvPC5fOAjeH8kfZ9Mr0nzwfONbO1BKP0vpigBjHTgqFUIFr+kYZamUz73Av0uvsfw+kfEATFVD7OLwEedfdN7l4E/gt4HlP7OFeN13HtDZ/Xz9+rpAbDaIbnmBTCMwy+Djzo7p+vWVQ7vMgbCPoeqvMvDM9uOAnYEVZVbwFeamazwm9qLw3nTTjufqm7L3T3xQTH7jZ3fy1wO8FQKrDnPscNtbIMOD88m2UJcARBR92E4+5/A9aZ2dPDWacDK5nCx5mgCekkM5sW/p1X93nKHuca43Jcw2V9ZnZS+B5eWLOtkbW606WFnT1nE5zB8wjwoVaX5ynsxykEVcP7gHvDx9kEbau/BB4Of84O1zeCGyY9AvwFWFqzrTcDq8PHm1q9b6Pc/9PYfVbSoQT/8KuB7wNt4fz2cHp1uPzQmtd/KHwvVjGKszVavK/PAlaEx/pHBGefTOnjDHwc+CtwP/AtgjOLptRxBr5L0IdSJPiGf9F4Hldgafj+PQJ8iboTGOIeGhJDREQiktqUJCIiI1AwiIhIhIJBREQiFAwiIhKhYBARkQgFg8g+mNliM3Mzm1A3mp+o5ZLJT8Egsm/rCAYjuxfAzE4LP5B7mlUAM/uVmX1pb+USGS9Nu+ezyGTl7mXgb43YtpllPRjuYcwaWS5JNtUYZEIJvxl/xcw+aWabwxuYXGlmqXD5WjN7f8xrvlQzvdbMPmJm15tZn5mtM7PXWHCjmxvNrD+8mclLR1mmXU024Qi2t4eLNoXzrw/XMzP7gJk9YmZD4c1RXheznQvM7DYzGwL+2czmWHATmt7wdQ+Y2ZtqXnc9cCrwtvD1Hm5rj6YkM3uhBTepGTazDWZ2VTjsy6je33Cdv7fgJjBDZrbVzH5tZgeO5r2SqUHBIBPRa4ESwYBpbwfeDbxmjNt4N8GwCCcQ3NjlBoKb+iwnGFriDuA/zax9jNtdB/xD+PwYgqacd4XTlxMMZ/A2gpvDXAFca2bn1G3jCuAr4To/IhjK4W7g5eE2vxi+7vRw/XcBdwLfDH/fPKJDLANgZgsIbsRyD8Hw6xcRjL9/Rd2qI76/ZnYQwcCENwBHEdxE5lujeF9kKmn1OCF66FH7AH4F3Fk37+fA18Lna4H3x7zmSzXTa4Hv1kx3EYwndXXNvMXhvKWjKFNkXYLxmRzoqVmnExgCXlD32i8Ay+u2875R/M4bq/sct48jlOsTBOPkpGrWeSOQB6aN8v09IdzmIa3+W9CjdQ/1MchEdF/d9Hp236hkzNtw934zGyQYdKxqQ/hzrNsdydEE3/z/v5nVDkCWJQiqWitqJ8wsTXBvhdcQ3ESljeDWnb8aYxmOIvjQr9TM+224rcPZ/Z7s7f39M/AL4H4zuzV8/gN33zTGssgkpmCQiai+M9bZ3exZYc+bj2RHuY1i3TSMX3NqdTuvIBguem9lGaibfj/wPoImo78A/cAnGXtoGSPfhKV2/ojvr7uXw76XkwiGbr4IuMLMTnX3P4+xPDJJKRhkstlE0MYOQNhHcCRBu3qzFMKf6Zp5KwmabA5x99vGuL1TgJ+4+7dg1z02ngZsr/ud6ZjX1loJvNrMUjW1hlPC1z4y2sK4uxP0adxpZpcBDxDUZhQMCaFgkMnmNtmLuwgAAAEzSURBVODNZraMICQ+RHyNoZEeI/iWfY6Z/QQYcvc+M7sSuDL8YL+DoG/jJKDi7tftZXsPAa8xs1MIbmD/DoLbWdaG3VrgxPCsqH6CO5PV+wpBR/JXzOyLBPct+BRB38TgaHYsvPnLSwhu/LKBoBN7EUHoSELorCSZbK4gCIcfA7cStKHf3cwCeHCT+o8SdPZuILj5CcC/AR8jaBp6gKBT9x+AR/exycsJzqD6GUGgDADfrlvnSoJv/isJAvHgEcp1FsGH+b3ANwhuAvPBMezeDoJbp95McJOYzwH/7u7/OYZtyCSnG/WIiEiEagwiIhKhYJDEM7Nrwquh4x7XtLp8Is2mpiRJPDM7AJg+wuKd7r6xmeURaTUFg4iIRKgpSUREIhQMIiISoWAQEZEIBYOIiET8DzFqzaawHTXtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# use first reward function initially and then switch to third reward function\n",
    "\n",
    "# Initialize learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new2]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions)\n",
    "\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAd/ElEQVR4nO3de5xfdX3n8df7d5uZ3ANJIEI0UFCMrS7pFGO9oVAFVNhd+6hgXS91S3crrrZaF+qlFl1Rq+ulYpH1uraFUu1DI6J4AXW1igwiSIKRAIFELknMda6/22f/OGeS35n8JpmBmTMz+b2fj8fvMb9zzvd35nN+Z+b3/p3b9ygiMDMzG1WY6QLMzGx2cTCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZll5BYMkj4rabuku8aZLkkfl7RZ0p2S1uZVm5mZHZTnFsPngXMOM/1c4NT0cTHwDznUZGZmY+QWDBHxA2DXYZpcAPzfSPwEWCJpZT7VmZnZqNJMF9DiBGBry/C2dNzDYxtKuphkq4L58+f/7mmnnZZLgWZmR4vbbrttZ0QsbzdtNgWD2oxr219HRFwNXA3Q29sbfX1901mXmdlRR9ID402bTWclbQNWtQyfCDw0Q7WYmXWs2RQM64FXp2cnrQP2RsQhu5HMzGx65bYrSdI1wJnAMknbgL8BygARcRVwA3AesBkYBF6XV21mZnZQbsEQERcdYXoAb8ipHDMzG8ds2pVkZmazgIPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaWkWswSDpH0iZJmyVd2mb6EyXdLOl2SXdKOi/P+szMLMdgkFQErgTOBdYAF0laM6bZO4DrIuJ04ELgk3nVZ2ZmiTy3GM4ANkfEfRFRBa4FLhjTJoBF6fPFwEM51mdmZuQbDCcAW1uGt6XjWr0beJWkbcANwBvbzUjSxZL6JPXt2LFjOmo1M+tYeQaD2oyLMcMXAZ+PiBOB84AvSjqkxoi4OiJ6I6J3+fLl01CqmVnnyjMYtgGrWoZP5NBdRa8HrgOIiB8D3cCyXKozMzMg32C4FThV0kmSKiQHl9ePafMgcBaApKeSBIP3FZmZ5Si3YIiIOnAJcCNwN8nZRxskXS7p/LTZW4A/lXQHcA3w2ogYu7vJzMymUSnPXxYRN5AcVG4d966W5xuBZ+dZk5mZZfnKZzMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWUauwSDpHEmbJG2WdOk4bf5I0kZJGyT9c571mZkZlPL6RZKKwJXAHwDbgFslrY+IjS1tTgUuA54dEbslrcirPjMzS+S5xXAGsDki7ouIKnAtcMGYNn8KXBkRuwEiYnuO9ZmZGfkGwwnA1pbhbem4Vk8GnizpR5J+IumcdjOSdLGkPkl9O3bsmKZyzcw6U57BoDbjYsxwCTgVOBO4CPi0pCWHvCji6ojojYje5cuXT3mhZmadLM9g2Aasahk+EXioTZuvRkQtIu4HNpEEhZmZ5STPYLgVOFXSSZIqwIXA+jFtvgK8AEDSMpJdS/flWKOZWcfLLRgiog5cAtwI3A1cFxEbJF0u6fy02Y3AbyRtBG4G/ioifpNXjWZmBooYu5t/bunt7Y2+vr6ZLsPMbE6RdFtE9Lab5iufzcwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLGNCwSCpLOkRSU+b7oLMzGxmTSgYIqIG1Di0m2wzMzvKTGZX0t8Dl0nK7XagZmaWv8l8yD8XeD7wa0l3AQOtEyPi/LavMjOzOWUywbAT+PJ0FWJmZrPDhIMhIl43nYWYmdnsMOFjDJJuanf/ZUmLJN00tWVNv3se3c9/+cwtDNcaM12KmdmsMpmDz2cClTbju0mOP8wpf7N+A//vnp3c9sDumS7FzGxWOeKuJElrWwafLmlXy3AReDHw66kuzMzMZsZEjjH0kVy/EMC32kwfAt44lUXlaY7f2dTMbMpNJBhOAgTcB/weydlJo6rA9oiYczvqpZmuwMxsdjpiMETEAwCS3g/8XkRc1Tpd0n+TdEJEvHOaapwW3lIwM2tvMgefXwXc3mb8z4BXT005ZmY20yYTDCuAHW3G7wSOm5py8uNdSWZm7U0mGB6k/WmpzwO2TU05ZmY20ybTJcangI9IqgCjF7SdBVwBfGCqC8tLuMNYM7OMyXSJ8WFJy4CPc/BCtyrwsYj44HQUZ2Zm+ZtUF9oRcZmk9wJrSE5h3RgR/dNSmZmZzYhJ31shIgaAW6ehllwJH302M2vH93w2M7OMjg8GX+hmZpbV8cFgZmZZHR8MvtDNzCyr44PBu5LMzLI6Nhi8pWBm1l7HBoOZmbWXazBIOkfSJkmbJV16mHZ/KCkk9eZZn5mZ5RgMkorAlcC5JFdOXyRpTZt2C4H/AdySV21mZnZQnlsMZwCbI+K+iKgC1wIXtGn3HuCDwHAeRfnYs5lZVp7BcAKwtWV4WzruAEmnA6si4vrDzUjSxZL6JPXt2NHuFhETFz4tycwsI89gaHce0IFPZUkF4CPAW440o4i4OiJ6I6J3+fLlU1iimZnlGQzbgFUtwycCD7UMLwR+G/iepC3AOmC9D0CbmeUrz2C4FThV0knpzX4uBNaPToyIvRGxLCJWR8Rq4CfA+RHRNx3FKL2QwTuSzMyycguGiKgDlwA3AncD10XEBkmXSzo/rzpGHdiv5WQwM8uY9P0YHo+IuAG4Ycy4d43T9szprGX0ymff2tPMLMtXPpuZWYaDwczMMjo2GEaPMfgyBjOzrM4NhtGzkhwMZmYZnRsM6U/ngplZVscGg5mZtedgMDOzjI4NhtHrGPYMVme2EDOzWaZjg+GObXsBeNuX75zhSszMZpeODYZqvQn4rCQzs7E6NhjUrhNwMzPr4GCY6QLMzGapjg2GgjcZzMza6thgkIPBzKytDg6Gma7AzGx26thg2LF/ZKZLMDOblTo2GMzMrD0Hg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWUbHBsMTFnfPdAlmZrNSxwZDoeAr3MzM2uncYPClz2ZmbXVsMDgXzMza69hgMDOz9jo2GLzBYGbWXscGg5mZtedgMDOzDAeDmZllOBjMzCyjY4Nh3cnHznQJZmazUscGw1NXLprpEszMZqVcg0HSOZI2Sdos6dI20/9S0kZJd0r6rqQn5VmfmZnlGAySisCVwLnAGuAiSWvGNLsd6I2IpwNfAj6YV31mZpbIc4vhDGBzRNwXEVXgWuCC1gYRcXNEDKaDPwFOzLE+MzMj32A4AdjaMrwtHTee1wPfaDdB0sWS+iT17dixYwpLNDOzPIOhXS8U0bah9CqgF/i7dtMj4uqI6I2I3uXLlz+mYiLa/mozs45XyvF3bQNWtQyfCDw0tpGks4G3A8+PiJE8Cms0g6Lvz2BmBuS7xXArcKqkkyRVgAuB9a0NJJ0OfAo4PyK251VYrdHM61eZmc16uQVDRNSBS4AbgbuB6yJig6TLJZ2fNvs7YAHwr5J+Lmn9OLObUlUHg5nZAXnuSiIibgBuGDPuXS3Pz86znlG1uoPBzGxUx1753Hro2VsMZmYHdWwwtKrVDz1DabjWYNdANTNu/3CN1Zd+nT/7Yl9epZmZ5c7BQPsthtPe+U3WvufbbN83DMBIvcHvvPtbANy44VH6tuzKtUYzs7w4GDj0rKSH9w4deH7G+74LwLr056iPfuee6S/MzGwGdGwwtF7fNlxr8PDeIe769V4AXvl/bsm0XX3p19k9WAPgtOMX8qI1x/HDzTtZfenXuX/nALsGqnztjoe4/cHdrL706/z7vTsZrNZzW5a5ptEM7t3RT/9InU2P7OeB3wxw/84B9g3XJjyPdhcoRgSNpi9cNHu8cj0raba6Z3s//+mT/w7AN9/8XO7fOTBu22+++Xncu6Ofb218FIAXfOh7h7QZDZbXPXs173jJmjlx8VxEsG+ozuJ5ZZrNYMND+ygUYE3aPfnuwRpbdw2ydF6Fhd0lfr5tDzv2j7Bj/wg/2ryTUrHA8gVdLOop8dCeIQZGGozUG6xY2E25KHoqJW5/cDe/enQ/h/vsXtxTplIq0FMuMlhtsLinxMLuMvVmk1o9qDWbVIoFNm/vZ3FPmWMXVDhuUTcP7hpk+74RRuoNTlw6j1NWLOCUFQtY0FVi+cIumhEIMa9SpFgQQ9UGuwerLOwu01MpMFhtMFxrsnXXIMWCKBVF/3CdgpJ1t3JJNycvW0BXucBQtcHinjKlgiiXCiydV2FJT5mCxED6hWC41mDpvAr7hmtUSgUWdJUYqjUYrjYpFsXgSJ0d/SOsWNjFcYu6WdBVot4MSgXRaAb7hutU6016ykWCoKtUpBFBTzmpPyKQZv/flc1NDgbg0i/feeD5zx7YQzH95+x7x9n0vvc7AHzkFc/gxU87HoDfWr6A777l+Zz14e8fdr6f+9EWPvejLZlxC7tL7B9OPjz+4uwn8+cv+C0KEiP1Br98ZD/f3vgoTzluIS99+koAigXx0/t3sainnHzANYNj5lfYP1xn6fwKANfdupXF88q8aM1xmQ+LiOQDfvdgla/c/hDLFlR4xqolVIoF3v21DQyM1JHEnsHqYT+sj2Rhd/Lhe+e2PewZrHH8om56KkW6SgXu2b4TAQPVBisWdvGyZzyB5Qu6WDKvzN6hGscv7qFSKlCrN9k/XOeB3wzQiKDeCGqNJsP1JrV6k1pDrFhYpiAYqjU4/YmrqNabbN8/zCN7hzlxaQ/PPmUZi7rLbN01yObt/fzwnp2TPuOsVBDlYoHheoOecpFmBKVCgf6R6d0ClJKt2FJB1I+wMkoFIUGlWKARQQSUiwW6SgXqzaBab7JkXhlI1k2lVKDZhD2DVQZrjQPro1QQ3eUiAEvmlTlhSQ8rFnazdF6ZYxd0USyAEIVCEmQS7Buu8/DeYSKCSqlAKf3SUywU6C4X+E1/lXKxwM7+EQqCrlKRXYNVHt47xJ7BWvK/1QjmdRVZOq9CM4J5lRIFQamY7MAoCPYP15lfKbGgu3QgBKv1JhJ0l4sI6KkU6S4VWdBdotEMhmsNBqoNuksFeipFhmsNGs3kpJFysUAzgnoz+bvaNVClWBCLusss6imzdF6ZYkEM15L3bl65yL70dQGM1JrsGaqyf7hOUaJSGn3v48B71GwGCLpLxfSLCMn4CLpKBbrLRRZ2leiuFJlfSZarf6TOvuE6tUaTeZUiC7vLB/ZmjL5uXqVEo9kkgJ5ykYJEAM1msPZJSzllxYKp/3uc630G9fb2Rl/f5M8S+uwP7+fy6zceMn7JvDJ70t1GW97/EjZv388x87s4Jv0QHuv2B3ezZ6jG2lVLufPXe1izchHHLuji4b1DPOuKmyZd10w6ZcUCHtw1SLXe5LW/v5o9g1U2PdrP3Q/v47TjF7Lu5GNZubibkXqTE5b0sGxhF09Y3M1Jy+ZTKhYO7MoZ/QefaRFBrRE8tGeIgkS5JAZGGjQjqBQLLOop0z9cZ6BaZ0FX8h3pxKU9bb+J7x2qce+Ofqr1JvMrJfYO1ag3mzQj2DVQY/dAlWYE87tKjNSbzK8U6R+pUy4WKBbEYLVOqVBgfleRiCTwVy7uYWf/CI/sG2b3QJXucpFqo0l3qciinhLlYrJ1IsFQtUGpWGCk3qBab9JoJstWKiZbD/VmMFJvUkxrH6w2AOgfqVFLQ/bY+RXmd5UoSHSVkg+84VrSbvdglV/vHuKRfcPsHaoxXGsfqOWiOH5xN+VCgZF6k3qziRD1ZpOhaoMl6Yf9MfMrFCSGag2W9JQ5bnE3x86vHFj2gZE6uwaqFApiuJask2o9mVczgp5KkWq9yUC1fuCDsqtUoBkHax6qNRisNhgYqVMsiJ5ykZ5KkZFak+F6g3KxQLkgFnaXqTWbFCRK6dbg0nlJLfuGa+wbqrF7sJas264ie4eS96yShsno717ck4TIaK2FghAgJV8kC0pOgx+uNigWRUTy4S2JkXqTkVqD/pblGdVdLlAqFBiqNSa9K/S9//G3edW6x3bbGkm3RURvu2neYhhjNBRGnbJi4WHbn/7EpQeeP/fUgx36rVzcw5b3v4Tf9I9w5c338tkf3c/3/+pM3nP93Xzn7kf5wMt/h//55V9MWd0XnbGKa356sPPa0a0egOecsowXnraCZ6xawjU/fZD5lSK9q4/hZc94wpT9fkj+QUrF2bN7QxKVkli9bP64bcYL/LEW95RZ27Kuj2YRwf6WLaRGI+guFykUoFwoUJgDu0Yfj9GgLSrZMgOmbLddRBLgA+n7O69SoqdSPDBtoNo4EO6QnDE5+qUCklAcra1QgCXzJvb3O1kdu8XwmR/ez3vabDGMuvbidb4vtJkdtQ63xTA7tvlnIYeCmXUqB0OLv7/o9JkuwcxsxvkYQ4uXPeMJU77f3cxsrun4LYYXPOWx3QHOzOxo1bHBMHrQ/ZXPTE71uvKVa2eyHDOzWaPjdyU98+RjuOWvz+K4Rd0zXYqZ2azQsVsMrRwKZmYHORjMzCzDwWBmZhkOBjMzy+j4YDi6e30xM5u8jg8GMzPLcjCYmVlGxwbDHO9U1sxs2nRsMIzy7RHNzLI6PhjMzCzLwWBmZhkOBjMzy+jYYAh89NnMrJ2ODYZRPvRsZpbV8cFgZmZZDgYzM8twMJiZWUbHBoOvfDYza69jg2GUL3w2M8vKNRgknSNpk6TNki5tM71L0r+k02+RtDrP+szMLMdgkFQErgTOBdYAF0laM6bZ64HdEXEK8BHgA3nVZ2ZmiTy3GM4ANkfEfRFRBa4FLhjT5gLgC+nzLwFnyb3cmZnlqpTj7zoB2NoyvA145nhtIqIuaS9wLLCztZGki4GL08F+SZseY03L5n8gO+8OsAy8zB3Ay9wZHs8yP2m8CXkGQ7tv/mPPDZpIGyLiauDqx12Q1BcRvY93PnOJl7kzeJk7w3Qtc567krYBq1qGTwQeGq+NpBKwGNiVS3VmZgbkGwy3AqdKOklSBbgQWD+mzXrgNenzPwRuivAVB2ZmecptV1J6zOAS4EagCHw2IjZIuhzoi4j1wGeAL0raTLKlcOE0l/W4d0fNQV7mzuBl7gzTsszyF3IzM2vV8Vc+m5lZloPBzMwyOjYYjtQ9x1whaZWkmyXdLWmDpDel44+R9G1J96Q/l6bjJenj6XLfKWlty7xek7a/R9Jrxvuds4WkoqTbJV2fDp+UdqVyT9q1SiUdP25XK5IuS8dvkvTimVmSiZG0RNKXJP0yXd/POtrXs6S/SP+u75J0jaTuo209S/qspO2S7moZN2XrVdLvSvpF+pqPT+ii4YjouAfJwe97gZOBCnAHsGam63qMy7ISWJs+Xwj8iqTLkQ8Cl6bjLwU+kD4/D/gGyTUj64Bb0vHHAPelP5emz5fO9PIdYdn/Evhn4Pp0+DrgwvT5VcB/T5//OXBV+vxC4F/S52vSdd8FnJT+TRRnerkOs7xfAP5r+rwCLDma1zPJBa/3Az0t6/e1R9t6Bp4HrAXuahk3ZesV+CnwrPQ13wDOPWJNM/2mzNCKeBZwY8vwZcBlM13XFC3bV4E/ADYBK9NxK4FN6fNPARe1tN+UTr8I+FTL+Ey72fYguQ7mu8ALgevTP/qdQGnsOiY5E+5Z6fNS2k5j13tru9n2ABalH5IaM/6oXc8c7AnhmHS9XQ+8+Ghcz8DqMcEwJes1nfbLlvGZduM9OnVXUrvuOU6YoVqmTLrpfDpwC3BcRDwMkP5ckTYbb9nn2nvyUeBtQDMdPhbYExH1dLi1/kxXK8BoVytzaZlPBnYAn0t3n31a0nyO4vUcEb8GPgQ8CDxMst5u4+hez6Omar2ekD4fO/6wOjUYJtT1xlwiaQHwZeDNEbHvcE3bjIvDjJ91JL0U2B4Rt7WObtM0jjBtziwzyTfgtcA/RMTpwADJLobxzPllTverX0Cy++cJwHyS3pnHOprW85FMdhkf07J3ajBMpHuOOUNSmSQU/iki/i0d/aiklen0lcD2dPx4yz6X3pNnA+dL2kLSS+8LSbYglijpSgWy9Y/X1cpcWuZtwLaIuCUd/hJJUBzN6/ls4P6I2BERNeDfgN/n6F7Po6ZqvW5Ln48df1idGgwT6Z5jTkjPMPgMcHdE/O+WSa3di7yG5NjD6PhXp2c3rAP2ppuqNwIvkrQ0/ab2onTcrBMRl0XEiRGxmmTd3RQRfwzcTNKVChy6zO26WlkPXJiezXIScCrJgbpZJyIeAbZKeko66ixgI0fxeibZhbRO0rz073x0mY/a9dxiStZrOm2/pHXpe/jqlnmNb6YPuszgwZ7zSM7guRd4+0zX8ziW4zkkm4Z3Aj9PH+eR7Fv9LnBP+vOYtL1Ibph0L/ALoLdlXn8CbE4fr5vpZZvg8p/JwbOSTib5h98M/CvQlY7vToc3p9NPbnn929P3YhMTOFtjhpf1PwB96br+CsnZJ0f1egb+FvglcBfwRZIzi46q9QxcQ3IMpUbyDf/1U7legd70/bsX+ARjTmBo93CXGGZmltGpu5LMzGwcDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwOwJJqyWFpFl1o/nZWpfNfQ4GsyPbStIZ2c8BJJ2ZfiAvy6sASd+T9InD1WU2VXK757PZXBURDeCR6Zi3pHIk3T1M2nTWZZ3NWww2q6TfjD8p6X2SdqY3MPmQpEI6fYukt7Z5zSdahrdIepekz0vaL2mrpFcoudHNtZL605uZvGiCNR3YZZP2YHtzOmlHOv7zaTtJepukeyUNpTdHeVWb+Vwk6SZJQ8CfSTpWyU1otqWv2yDpdS2v+zzwfOAN6esjndchu5IkPU/JTWqGJT0q6SNpty8Ten/TNv9ZyU1ghiTtkvR9ScdN5L2yo4ODwWajPwbqJB2mXQK8GXjFJOfxZpJuEdaS3NjlCyQ39bmBpGuJHwD/KKl7kvPdCrw8ff40kl05b0qH30vSncEbSG4OcwXwKUkvGTOPK4BPpm2+QtKVw8+Al6bz/Fj6urPS9m8Cfgx8Lv19K8l2sQyApBNIbsRyO0n3668n6X//ijFNx31/JR1P0jHhF4CnktxE5osTeF/saDLT/YT44UfrA/ge8OMx474NfDp9vgV4a5vXfKJleAtwTcvwApL+pD7eMm51Oq53AjVl2pL0zxTAspY284Eh4LljXvtR4IYx83nLBH7ntaPL3G4Zx6nrf5H0k1NoafNaYASYN8H3d206zyfN9N+CHzP38DEGm43uHDP8EAdvVDLpeUREv6RBkk7HRj2a/pzsfMezhuSb/zcltXZAViYJqlZ9rQOSiiT3VngFyU1Uukhu3fm9SdbwVJIP/WbLuB+m8zqFg+/J4d7fO4DvAHdJ+lb6/EsRsWOStdgc5mCw2Wjswdjg4G7PJofefKQ8wXnUxgzD1O1OHZ3Py0i6iz5cLQNjht8KvIVkl9EvgH7gfUw+tMT4N2FpHT/u+xsRjfTYyzqSrptfD1wh6fkRccck67E5ysFgc80Okn3sAKTHCE4j2a+el2r6s9gybiPJLpsnRcRNk5zfc4CvRcQX4cA9Np4M7BnzO4ttXttqI/BHkgotWw3PSV9770SLiYggOabxY0mXAxtItmYcDB3CwWBzzU3An0haTxISb6f9FsN0eoDkW/ZLJH0NGIqI/ZI+BHwo/WD/AcmxjXVAMyKuPsz8fgW8QtJzSG5g/0aS21m2ht0W4Iz0rKh+kjuTjfVJkgPJn5T0MZL7Fryf5NjE4EQWLL35y9kkN355lOQg9iqS0LEO4bOSbK65giQcvgp8i2Qf+s/yLCCSm9T/DcnB3kdJbn4C8E7g3SS7hjaQHNR9OXD/EWb5XpIzqL5BEigDwD+NafMhkm/+G0kC8Ynj1HUuyYf5z4HPktwE5q8nsXh7SW6dej3JTWI+DLwnIv5xEvOwOc436jEzswxvMZiZWYaDwTqepKvSq6HbPa6a6frM8uZdSdbxJK0AFo0zeV9EbM+zHrOZ5mAwM7MM70oyM7MMB4OZmWU4GMzMLMPBYGZmGf8fKbXSco3ub9QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Do not learn\n",
    "# use first reward function initially and then switch to third reward function\n",
    "\n",
    "# Initialize learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n",
    "\n",
    "num_iterations_per_cost_func = 5000\n",
    "cost_functions = [get_cost, get_cost_new2]\n",
    "total_iterations = num_iterations_per_cost_func * len(cost_functions)\n",
    "\n",
    "ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions, should_learn=False)\n",
    "\n",
    "plot_ctr(total_iterations, ctr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "This tutorial aimed ad showcasing a real world scenario that contextual bandits can be applied to. We were able to take a context and list of actions and learn what actions worked best for which context. We saw that the learner was able to respond rapidly to changes in the world. As a baseline we showed the case without learning to prove we completely outperform it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# things to do\n",
    "# (1) change reward function and see how learning happens - DONE (PAVITHRA)\n",
    "# (2) polish content - good story telling - IN PROGRESS (PAVITHRA)\n",
    "# (3) add comments - IN PROGRESS (PAVITHRA)\n",
    "# (4) add plots to see how rewards change per action over time THINKING THROUGH THIS (PAVITHRA)\n",
    "# (5) Scenario 4: add some noise to reward function\n",
    "# (6) explain VW example format - DONE (Jack)\n",
    "# (7) label plots properly = DONE (Jack)\n",
    "# (8) Summarize results and wrap up story"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}