notebooks/challenges_with_gradient_boosters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenges using gradient boosters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*I only recommend to look into this notebook if you are truly interested in machine learning. In case you are more interested in the soccer part, you do not need to go through it. None of the things we learn in this notebook will be needed in any of the following tutorials.*\n",
    "\n",
    "Welcome back to another tutorial on soccer analytics! In this notebook we want to learn about some challenges that can arise when dealing with gradient boosting algorithms like [lightGBM](https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc) or [XGBoost](https://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/). Often times, these two algorithms are kind of seen as the holy grail when solving supervised machine-learning tasks on structured data. And rightfully though, as they often outperform other machine-learning approaches on these kind of tasks. While I'm also a big fan of these algorithms, they do come with some drawbacks that one should at least be aware of, before blindly using these algorithms in a real-life situation. Looking at what those drawbacks are and how to overcome some of those challenges, is the aim of this notebook.\n",
    "\n",
    "We are going to start the notebook with a performance comparison of logistic regression and lightGBM, before we dive into some challenges that we find when using gradient (tree) boosters. The notebook is therefore split into the following chapters:\n",
    "\n",
    "0. [Performance comparison](#performance) <br>We compare the performance of logistic regression and lightGBM on a simple expected goal model\n",
    "\n",
    "\n",
    "1. [Challenge: Overfitting](#overfitting) <br>We look into overfitting and why it not only harms you model error, but might also affect the feature interpretation. Moreover, we learn about ways to eliminate overfitting\n",
    "\n",
    "\n",
    "2. [Challenge: Feature interpretation](#feature_interpretation) <br>We get to know a great package called *shap* that allows us to do feature interpretation, even though gradient boosters are considered black-box models \n",
    "\n",
    "\n",
    "3. [Challenge: Missing monotonicity](#monotonicity) <br>We observe that gradient boosters are generally non-monotone, which might have a negative effect on the model credibility when presenting results to a non data scientist. We then learn how to overcome this problem by adding monotone constraints, but need to admit that enforcing \"smoothness\" to the predictions is an almost impossible task\n",
    "\n",
    "\n",
    "4. [Challenge: No extrapolation](#extrapolation) <br>We look into the fact that gradient (tree) boosters are incapable of extrapolation and elaborate on what that means in the context of an expected goal model. We then learn about ways on how to conceal this problem \n",
    "\n",
    "While this might look like a tutorial on why <b>not</b> to use gradient boosters, by no means it is. As we already said in the previous notebook, everything always depends on what exactly you want to do with your model and which tasks you want to use it. There are many great use cases out there gradient boosters are ideal for and I've used them in many different projects. Nevertheless, instead of blindly using those algorithms, it is useful to know their limitations, especially in the context of presentability of model results. And to show you these limitations and how to overcome some of them, is exactly the idea of this notebook. \n",
    "\n",
    "Notice that we build a dummy model that serves as a helper to identify and work through the challenges that come with gradient boosters. However, this notebook <b>does not intend to build a good expected goal model using lightGBM nor is it seen as an introduction to lightGBM</b>, i.e. \n",
    "- no outlier detection at the beginning of the notebook\n",
    "- no dedicated feature engineering nor error analysis\n",
    "- no extensive hyperparameter optimization\n",
    "- no introduction to lightGBM or XGBoost as there are a lot of great tutorials and notebooks out there (e.g. [here](https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc) and [here](https://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/))\n",
    "\n",
    "If you want to build a state-of-the-art goal model using lightGBM or XGBoost, recommend you to go through both the previous notebook as well as this notebook and try to plug things together. However, do not solely rely on this notebook!\n",
    "\n",
    "Notice furthermore, that we are going to touch more advanced topics in this notebook such as lightGBM and XGBoost itself, SHAP values, Baysian hyperparameter optimization or outlier detection using isolation forests. Describing these concepts in detail would not only bust the size of the notebook, but is also not the idea behind the soccer analytics project. I therefore often times only paste a link to another website where you can learn more about these concepts. I guess this makes way more sense than me trying to explain them here.\n",
    "\n",
    "Ok, enough talking for now, let's get started! :-)\n",
    "\n",
    "<b>Update</b>: In case you want to get a deeper understanding on how to use Random Forests for expected goal models, check out [this](https://github.com/andrewsimplebet/expected_goals_deep_dive) great series by Andrew Puopolo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import shap\n",
    "\n",
    "import sklearn.metrics as sk_metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression, LinearRegression\n",
    "from sklearn.ensemble import IsolationForest\n",
    "import lightgbm as lgb\n",
    "import plotly.graph_objects as go\n",
    "import plotly.express as px\n",
    "\n",
    "if os.getcwd().split(os.sep)[-1] == \"notebooks\":\n",
    "    os.chdir(\"../\")\n",
    "\n",
    "import helper.io as io\n",
    "import helper.expected_goal_model as goal_model\n",
    "\n",
    "# this is very useful as it makes sure that always all columns and rows of a data frame are displayed\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the event data of all the leagues and extract only the shots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "league = \"all\"\n",
    "df_events = io.read_event_data(league, notebook=\"expected_goal_model\")\n",
    "df_shots = df_events[df_events[\"eventName\"] == \"Shot\"].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"performance\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance comparison of logistic regression and lightGBM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start our analysis by comparing the model performance of a logistic regression with the performance of an out-of-the-box lightGBM algorithm. \n",
    "\n",
    "In order to do so, we will use similiar features like the ones we used in the previous notebook. In order to not be too biased towards the logistic regression, we will, however, disregard the features that were specifically and under lots of work built to fix the flaws of the logistic regression. I'm hereby talking about the following features:\n",
    "- *inFrontOfGoal* measuring whether a player was directly in front of the goal\n",
    "- *angle* measuring the angle between the player and the goal\n",
    "- *headerDistToGoalLine* measuring the distance to the goal line specifically for headers\n",
    "\n",
    "But before we choose the features, let's first split the data into train and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, df_test, _, _ = train_test_split(df_shots, df_shots[\"goal\"], test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the features the same way as in the previous notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "egm = goal_model.ExpectedGoalModelLogistic()\n",
    "\n",
    "# create the same features as used in logistic regression\n",
    "df_train = egm.create_features(df_train)\n",
    "df_test = egm.create_features(df_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned above, let's only keep the completely natural features and not the ones that were especially designed for the logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = [\"distToGoalLine\", \"distToCenter\", \"counterAttack\", \"corner\", \"smartPass\", \"duel\", \"shotBefore\", \n",
    "            \"head/body\", \"weakFoot\", \"strongFoot\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training of the logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#################\n",
      "# Logistic reg. #\n",
      "#################\n",
      "Log loss on test set: 0.28549\n",
      "AUC on test set: 78.36%\n"
     ]
    }
   ],
   "source": [
    "# training of the logistic regression on the train set\n",
    "\n",
    "model_lr = LogisticRegression(random_state=42)\n",
    "model_lr.fit(df_train[features], np.array(df_train[\"goal\"]).ravel())\n",
    "\n",
    "# prediction on the test set\n",
    "pred_probs = model_lr.predict_proba(df_test[features])[:,1]\n",
    "\n",
    "print(\"#################\\n# Logistic reg. #\\n#################\")\n",
    "print(f\"Log loss on test set: {sk_metrics.log_loss(df_test['goal'], pred_probs):.5f}\")\n",
    "print(f\"AUC on test set: {sk_metrics.roc_auc_score(df_test['goal'], pred_probs)*100:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, log loss of 0.28549. As expected, this is fundamentally worse than what we had in the previous notebook. However, this also makes sense given that we did not consider 3 important features that we built manually to improve the logistic regression prediction. Let's see what lightGBM can do..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training of the out-of-the-box lightGBM model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We build an out-of-the-box LGBM classifier below. In case you have not heard about lightGBM yet, I recommend you to go through this [explanation](https://medium.com/@pushkarmandot/https-medium-com-pushkarmandot-what-is-lightgbm-how-to-implement-it-how-to-fine-tune-the-parameters-60347819b7fc). By the way, even though we use lightGBM in this notebook, all the insights and challenges are identical when using XGBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "############\n",
      "# LightGBM #\n",
      "############\n",
      "Log loss on test set: 0.28351\n",
      "AUC on test set: 78.29%\n"
     ]
    }
   ],
   "source": [
    "# train an out-of-the-box lightGBM classifier\n",
    "model = lgb.LGBMClassifier(random_state=42, objective=\"binary\")\n",
    "model.fit(df_train[features], df_train[\"goal\"])\n",
    "\n",
    "# predict the probabilities\n",
    "pred_probs = model.predict_proba(df_test[features])[:,1]\n",
    "\n",
    "print(\"############\\n# LightGBM #\\n############\")\n",
    "print(f\"Log loss on test set: {sk_metrics.log_loss(df_test['goal'], pred_probs):.5f}\")\n",
    "print(f\"AUC on test set: {sk_metrics.roc_auc_score(df_test['goal'], pred_probs)*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm, this is somehow underwhelming, isn't it? We use this very fancy model and the log loss is only slightly better than with logistic regression. And the AUC is even worse... \n",
    "\n",
    "Let's look deeper into why this is the case and how we can fix this in the next section.\n",
    "\n",
    "<a id=\"overfitting\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge 1: Overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You noticed that above we only focused on the error on the test set? You might say now: \"Yes, of course, because that's what we are interested in. Being good on data, that the algorithm has seen, does not tell me anything.\" I agree, ultimately what counts is the performance on the test set. However, by not looking at the error on the training set, we are not able to identify one very common issue, namely <b>overfitting</b>. \n",
    "\n",
    "In very easy words, overfitting describes the behavior, that a model learns the training data too well and can therefore not distinguish between noise and actual signal any more. If you haven't heard about overfitting yet, you can quickly read about it [here](https://elitedatascience.com/overfitting-in-machine-learning) or watch [this](https://www.youtube.com/watch?v=Anq4PgdASsc) short video. So, while overfitting is usually not a big problem with \"simple\" models such as logistic regression, it is *very* often a problem with more complex models such as gradient boosters, random forests or neural networks. Hm, ok, but that raises three questions that we are going to answer in the remainder of this section. \n",
    "\n",
    "1. Why is overfitting bad?\n",
    "2. How can we detect overfitting?\n",
    "3. What can we do against overfitting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why is overfitting bad?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overfitting is bad for two reasons:\n",
    "\n",
    "- A model that overfits has often a worse performance on the test set than a non-overfitting model\n",
    "- An overfitting model brings us into huge trouble when interpreting features importances\n",
    "\n",
    "While the first point is hopefully obvious after you have read the link above or watched the video, the second might be a little bit more complex to understand. Let's therefore quickly come up with an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>distToGoalLine</th>\n",
       "      <th>bodyPart</th>\n",
       "      <th>goal</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9.0</td>\n",
       "      <td>header</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.1</td>\n",
       "      <td>foot</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.2</td>\n",
       "      <td>header</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.3</td>\n",
       "      <td>foot</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.4</td>\n",
       "      <td>header</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9.5</td>\n",
       "      <td>foot</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9.6</td>\n",
       "      <td>header</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9.7</td>\n",
       "      <td>foot</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9.8</td>\n",
       "      <td>foot</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9.9</td>\n",
       "      <td>header</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   distToGoalLine bodyPart  goal\n",
       "0             9.0   header     0\n",
       "1             9.1     foot     1\n",
       "2             9.2   header     0\n",
       "3             9.3     foot     1\n",
       "4             9.4   header     0\n",
       "5             9.5     foot     1\n",
       "6             9.6   header     0\n",
       "7             9.7     foot     1\n",
       "8             9.8     foot     0\n",
       "9             9.9   header     1"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\"distToGoalLine\": np.arange(9,10,0.1), \n",
    "                   \"bodyPart\": [\"header\",\"foot\"] * 4 + [\"foot\",\"header\"], \n",
    "                   \"goal\": [0,1] * 5})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so what is the situation? We have 10 shots, 5 of which resulted in a goal and 5 did not. If we look at the body part, 4/5 shots with the foot went in and 1/5 headers went in. Regarding the distance to the goal line, shots from 9.1m, 9.3m, 9.5m, 9.7m and 9.9m went in, while the others did not. \n",
    "\n",
    "If I asked you, which feature contains any information and which feature doesn't, your answer would be very clear, wouldn't it? Clearly, it is very important whether you shoot with the head or the foot, while the distance does not make any difference (at least not in our toy example :-)). So what would a highly overfitted model do? It would learn that shots from exactly 9.10m go in, while shots from exactly 9.00m or 9.20m don't etc. So what would be the most important feature for an overfitted model? Correct, the distance to the goal line. This feature can explain to 100% whether the shot results in a goal or not. So, you see while overfitting is a problem for feature interpretation? ;-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do we detect overfitting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gladly, this is pretty simple! :-) All we need to do is to compare the error between the training dataset and the test set. \n",
    "\n",
    "*If the error on the training data is significantly lower than on the test data, the model overfits.*\n",
    "\n",
    "So, let's check whether our models overfit. (Actually, we should have tested for overfitting of the logistic regression already in the previous notebook, but I did not want to overload you with yet another machine learning terminology)\n",
    "\n",
    "Let's start with the logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#################\n",
      "# Logistic reg. #\n",
      "#################\n",
      "\n",
      "Test set\n",
      "############\n",
      "Log loss on test set: 0.28549\n",
      "AUC on test set: 78.36%\n",
      "\n",
      "Training set\n",
      "############\n",
      "Log loss on train set: 0.28771\n",
      "AUC on train set: 77.52%\n"
     ]
    }
   ],
   "source": [
    "# training of the logistic regression on the train set\n",
    "model_lr = LogisticRegression(random_state=42)\n",
    "model_lr.fit(df_train[features], np.array(df_train[\"goal\"]).ravel())\n",
    "\n",
    "# prediction on training and test set\n",
    "pred_probs = model_lr.predict_proba(df_test[features])[:,1]\n",
    "pred_probs_train = model_lr.predict_proba(df_train[features])[:,1]\n",
    "\n",
    "df_train[\"pred_lr\"] = pred_probs_train\n",
    "df_test[\"pred_lr\"] = pred_probs\n",
    "\n",
    "print(\"#################\\n# Logistic reg. #\\n#################\")\n",
    "\n",
    "print(\"\\nTest set\\n############\")\n",
    "print(f\"Log loss on test set: {sk_metrics.log_loss(df_test['goal'], pred_probs):.5f}\")\n",
    "print(f\"AUC on test set: {sk_metrics.roc_auc_score(df_test['goal'], pred_probs)*100:.2f}%\")\n",
    "\n",
    "print(\"\\nTraining set\\n############\")\n",
    "print(f\"Log loss on train set: {sk_metrics.log_loss(df_train['goal'], pred_probs_train):.5f}\")\n",
    "print(f\"AUC on train set: {sk_metrics.roc_auc_score(df_train['goal'], pred_probs_train)*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, this looks very good, as the test set error is even slightly smaller than the training data set error. So, there is definitely no overfitting in place! :-)\n",
    "\n",
    "Let's see how the situation looks like for lightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "############\n",
      "# LightGBM #\n",
      "############\n",
      "\n",
      "Test set\n",
      "############\n",
      "Log loss on test set: 0.28351\n",
      "AUC on test set: 78.29%\n",
      "\n",
      "Training set\n",
      "############\n",
      "Log loss on train set: 0.26363\n",
      "AUC on train set: 82.62%\n"
     ]
    }
   ],
   "source": [
    "pred_probs = model.predict_proba(df_test[features])[:,1]\n",
    "pred_probs_train = model.predict_proba(df_train[features])[:,1]\n",
    "\n",
    "df_train[\"prediction\"] = pred_probs_train\n",
    "df_test[\"prediction\"] = pred_probs\n",
    "\n",
    "print(\"############\\n# LightGBM #\\n############\")\n",
    "\n",
    "print(\"\\nTest set\\n############\")\n",
    "print(f\"Log loss on test set: {sk_metrics.log_loss(df_test['goal'], pred_probs):.5f}\")\n",
    "print(f\"AUC on test set: {sk_metrics.roc_auc_score(df_test['goal'], pred_probs)*100:.2f}%\")\n",
    "\n",
    "print(\"\\nTraining set\\n############\")\n",
    "print(f\"Log loss on train set: {sk_metrics.log_loss(df_train['goal'], pred_probs_train):.5f}\")\n",
    "print(f\"AUC on train set: {sk_metrics.roc_auc_score(df_train['goal'], pred_probs_train)*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm, not that good any more, is it? While we have a log loss of 0.28351 on the test set, it is at 0.26363 on the training set. This is not a huge overfitting and you will definitely see much worse on some of your future projects, but it is still something we should try to fix. But how do we fix overfitting? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What can we do against overfitting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When having to deal with overfitting, there are generally speaking 3 strategies to deal with it:\n",
    "\n",
    "1. Increase the number of data points\n",
    "2. Reduce the number of features\n",
    "3. Adapt / simplify the model\n",
    "\n",
    "Unfortunately, increasing the number of data points is no option here as the data is what it is. So the first strategy does not work for us. Given that we only have a very small number of features at the moment, also strategy number two does not seem to be adequate here. So it seems like we are stuck with the third strategy. But, how the hell do we simplify our lightGBM model?\n",
    "\n",
    "Unlike the logistic regression, which we could not tweak a lot, more complex models come with a long list of hyperparameters (see [here](https://lightgbm.readthedocs.io/en/latest/Parameters.html) for all hyperparameters for lightGBM). A set of hyperparameters is basically a configuration of our model. And there are some configurations that are \"simpler\" and some are more \"complex\". Now, finding the right balance between \"simple\" and \"complex\" is the task of a so-called hyperparameter optimization. I will not go into detail regarding hyperparameter optimization as it is a huge topic itself and there are quite a lot of good resources already out there (e.g. [here](https://www.datacamp.com/community/tutorials/parameter-optimization-machine-learning-models) for a starter into hyperparameter optimization, [here](https://www.kaggle.com/willkoehrsen/intro-to-model-tuning-grid-and-random-search) for code on grid search and random seach or [here](https://towardsdatascience.com/automated-machine-learning-hyperparameter-tuning-in-python-dfda59b72f8a) for a more advanced Baysian approach).\n",
    "\n",
    "Let me add one sentence to overfitting in logistic regression. If this happens to you, you most probably have too many features that go into your model. From my experience, the most common reason for why this happens is, that categorical features with many different values are being converted into dummy variables which let's the number of features explode. So, when you find your logistic regression to be overfitting, I would start by looking into this. ;-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so here is the set of hyperparameters that I found using the Baysian approach. Notice that I will not go into detail on how I found them as this would definitely blow up the size of this notebook. But if interested, you can check out the links above to see how you can implement it yourself. Having said this, I did also not bother to try to optimize things a lot, but took the first solid hyperparameter combination ;-) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyp = {'boosting_type': 'gbdt', \n",
    "       'colsample_bytree': 0.9053563761947043, \n",
    "       'is_unbalance': False, \n",
    "       'learning_rate': 0.03811321514778682, \n",
    "       'min_child_samples': 395, \n",
    "       'n_estimators': 118, \n",
    "       'num_leaves': 20, \n",
    "       'reg_alpha': 0.5617832258471822, \n",
    "       'reg_lambda': 0.34491347534610917, \n",
    "       'subsample_for_bin': 140000, \n",
    "       'subsample': 0.5869782389570299, \n",
    "       'objective': 'binary'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if the overfitting was reduced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "############\n",
      "# LightGBM #\n",
      "############\n",
      "\n",
      "Test set\n",
      "############\n",
      "Log loss on test set: 0.28078\n",
      "AUC on test set: 78.92%\n",
      "\n",
      "Training set\n",
      "############\n",
      "Log loss on train set: 0.27695\n",
      "AUC on train set: 79.58%\n"
     ]
    }
   ],
   "source": [
    "model = lgb.LGBMClassifier(**hyp)\n",
    "model.fit(df_train[features], df_train[\"goal\"])\n",
    "\n",
    "pred_probs = model.predict_proba(df_test[features])[:,1]\n",
    "pred_probs_train = model.predict_proba(df_train[features])[:,1]\n",
    "\n",
    "df_train[\"prediction\"] = pred_probs_train\n",
    "df_test[\"prediction\"] = pred_probs\n",
    "\n",
    "print(\"############\\n# LightGBM #\\n############\")\n",
    "\n",
    "print(\"\\nTest set\\n############\")\n",
    "print(f\"Log loss on test set: {sk_metrics.log_loss(df_test['goal'], pred_probs):.5f}\")\n",
    "print(f\"AUC on test set: {sk_metrics.roc_auc_score(df_test['goal'], pred_probs)*100:.2f}%\")\n",
    "\n",
    "print(\"\\nTraining set\\n############\")\n",
    "print(f\"Log loss on train set: {sk_metrics.log_loss(df_train['goal'], pred_probs_train):.5f}\")\n",
    "print(f\"AUC on train set: {sk_metrics.roc_auc_score(df_train['goal'], pred_probs_train)*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! :-) Exactly the way we wanted it, overfitting was clearly reduced with the log loss on the training set now being 0.27695 instead of 0.26323. And notice that the error on the test set was also reduced from 0.28351 to now 0.28078. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool, we overcame the problem of overfitting and are now in a situation in which we outperform the logistic regression when using the same set of rudimentary features (log loss of 0.28078 vs 0.28549). From my experience this is almost always the case, especially when not building special features to overcome some of the flaws of a logistic regression (I'm speaking about features such as the *inFrontOfGoal* feature). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> In most real-world machine-learning tasks, a well-calibrated gradient booster such as lightGBM or XGBoost outperforms a logistic regression when using the same set of features </b>  \n",
    "\n",
    "Ok, so the situation is clear, isn't it? We wasted our time in the previous notebook trying to understand logistic regression, while we should have used lightGBM in the first place. Let me tell you that I'm a big fan of lightGBM and have used it in various projects. Nevertheless, unfortunately the situation is not quite as clear as it seems to be from the above statement as\n",
    "- often times smartly, hand-crafted features can bring a logistic regression to an (almost) similiar performance. This is especially true when dealing with rather simple problems with a relatively small number of features\n",
    "- lightGBM comes with some (potential) challenges that might not hurt the log loss but do might hurt the model credibility when presenting results in real life\n",
    "\n",
    "We have already learned about hand-crafted features for logistic regression in the previous notebook. So let's look into these challenges that might arise with lightGBM.\n",
    "\n",
    "<a id=\"feature_interpretation\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge 2: Difficult feature interpretation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arguably, the most common argument *against* using lightGBM or XGBoost is, that it is a black-box model and features can therefore not be interpreted. Funny enough, out of all the challenges that we consider in this notebook, this is in my opinion the one that is the simplest to solve thanks to a great package called *shap*. The package will help us to solve two questions we also had in the previous notebook: \n",
    "\n",
    "1. Does a variable have a positive or negative impact on the likelihood to score?\n",
    "2. What are the most important features?\n",
    "\n",
    "Again, I'm going to focus solely on how we can answer those questions and do not talk much about what SHAP values are in detail. Nevertheless, I highly recommend you to read about them [here](https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Does a variable have a positive or negative impact on the likelihood to score?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first compute the SHAP values and draw the graph and then look into the interpretation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\programs_clemens\\anaconda3\\envs\\soccer_analytics\\lib\\site-packages\\shap\\explainers\\tree.py:114: UserWarning:\n",
      "\n",
      "Setting feature_perturbation = \"tree_path_dependent\" because no background data was given.\n",
      "\n",
      "c:\\programs_clemens\\anaconda3\\envs\\soccer_analytics\\lib\\site-packages\\shap\\explainers\\tree.py:249: UserWarning:\n",
      "\n",
      "LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi4AAAFfCAYAAABp3EcmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOy9eZwdR3mv/1RVL2ebTSONVtvyirGNjUMDYTNhc8IFQm7CEhK4YHZyDSHADxJWhyUkhoAvcAMkBMJ6wSxx4hDCYjBgcAwNxuAF75K1azTrWXupqt8f1XPmzEiWhC3ZktzP5zPSOd3V1dXV3VXf875vVQlrLSUlJSUlJSUlRwPy/i5ASUlJSUlJScnBUgqXkpKSkpKSkqOGUriUlJSUlJSUHDWUwqWkpKSkpKTkqKEULiUlJSUlJSVHDaVwKSkpKSkpKTlqKIVLSUlJSUnJAxghxCYhxFnLtsVCiN8RQrxTCPHcg8jjIiHE+w9fKRfx7ouTlJSUlJSUlBx9WGvffn+XYTmlxaWkpKSkpKRknwgh/kUIcWHxeUQI8VUhxK+FEFcIIT6zzMqyXgjxn8X+rwshaoejTKXFpaSkpKSkpOQrQojewPfT9pHm7cCMtfZ0IcQK4GfAVwf2R8DDgTngm8CfAv90qAtaCpcHJuU6DyX3KZdffjkAz3jGM+7nkpSUHLGIw5PrHy5t7+3X7u48z7LWXt8/TIh4H2meALwawFo7LYS4bNn+b1prZ4vjrwFOvqfF3h+lq6ikpKSkpOSYRSz7u9eZ7e+H76DFRnOYjCOlcCkpKSkpKSk5GL4HvBBACDEGPPP+KEQpXEpKSkpKSo5ZDqnF5Z3AhBDiBuBzwI9w8Sz3KWWMS0lJSUlJyTHLgcWKtXbjPrZFxccrBza3gedZa3tCiGHgKuDTRfqLlh2/5PuhpBQuJSUlJSUlxyyHNOZ3DPiGEEIBFeAL1trvHMoTHAylcCkpKSkpKSk5INba3cDD7u9ylMKlpKSkpKTkmOXwjLK+PymFS0lJSUlJyTHLsSdcylFFJSUlJSUlJUcNpcWlpKSkpKTkmOXYs7iUwqWkpKSkpOSYpRQuJSUlJSUlJUcNpXApKSkpKSkpOWoohUtJSUlJSUnJUYJdJlyOBRlTCpeSkpKS+4nT37KHqZ4kDz06gSLoZfzsdUOcNqHu76KVlByxlMKlpKSkZD/ccluX975/FzOJJTdQkYJzz6nwypdN8NAPdtBzGSfOdwktdCS86OkNXvn04QPmu+ZN0wRdS15VzAU+Xq4RCB7y3nmSD47dB1dWUnJ0UgqXkpICc+ckk2d+gFa3Qo86Pikr2c4QUwgEyfhqqje/GzleX3JcJ96BGg7w1g+R3jFH5ayVCHEsGGTve6Z29BCeYGxlcK/qMMsM73nfTn5yW44CKnnOqgmfD/3tBjzv4PNttXLe8a6dpEKQKI/cV0xJwbYbNd/4ix00Ao8MwW7fYyLTDBnLF/+9xSX/1UEZMEphjWXaU5z3IMXnXjpCJXDnb+cgpGS2GkCqsUlOLc8ZMpbwdTMgBf/+0hq/e3pwj+uhpOTYcA4tpRQuJUcktpfRe9Y/Yb5+PQKNwYfzTqf21QuQK+sHzuA3PqFl7qR3sZOT8dEoLBLLFKvxsdTYjZ3qMb3yHcwzisBiUEg0OZKEKoqUlCoGOPP6PyY4c/WhL+cxijWWNz3zWnYND4G1rGp38Y1BWsM7//VclL/vuTJbzZxvfHWSzbd3AMnDHj3ME546zoUvuoVtYY3M88iAXErkroSXvmEr/3LJcQddrh9c1cQAobWEeU7LWqzn0RaCtpJoBAiBBXb7iq6SrM4yTu0k7Al8bq4EzBbi5ds3a/78c00uecEQV92ZE+aGXAiGZzvMK8WaLCe0FoDVec6WIOD3PtHh53+hOHd96ToquWeUMS4lJYcZe80ttF9+KfbXO/FSg8Ki8TBU4AebaK15O0Pp+xDynk/6rOe76Jf8C/pnW8hHRsnOOoXwKScxy0o8DD4GgUGiWM3tWBRN1lKnTYUUjaLHMBZBThUwrGMTGRUyAnrU2Hr2P3KSftuhq5jfkGQ+46rX/jfbr9iBQaAyg58ZeqEiDRU69Hjqxx7JCeetud/KOMjnLvo1u4caYC2jvQTPGARgheRdz7qWs5+wgkc9c4JVG6p84zPb+P6/7yEXisxTsPAsCMGdt0zypX+ZRAOmsviM5FIylmZsnfvNmryhumJQMoTG0LWWldrQNZZ2uHgOAQzlmqZyAubOIGBGKrCubPO+xzd+0ePTNxmGMo2VksxaVmhDSyn8QrQAeHYx14u+2eVRJ3lccZvhMessp69W/O6DfcZqpZgpORiOBamyFGEHXpaSBwxH1E23kzPkr/0idssuZn+YILEE9PDoUWGKDuucxaWg9sNX4D321APmm946Tffq7XjbttH8l+sRt+zCxzDLGE2G8MgZoYVFARZDQpdhfAwNWghAkOMzT5UMhXblBTZxKoO63yCQWMAyR5UmNap0GSLh5F/8KZVz7jvri7WWL538Fbp6oQO0qNwgEMhco6Uk9yW9aoD2Pdq1kOf9n7NZ/+jDV8bLL78cgGc84xl77evM53zg+deyqdKgUwlBCDytWTfXRBqDyjVSQOp51EYEnXnXFFsgF4JOEIBaFBAWENayo1qlGTo3Sy3LWNfpsrVW4/JPHd9Pa4xlZk4z3JAoJZDSNfIf/cRO/uuHXXIpGTKmvzZKT0qUtfjWooG7KhX2+E5AjOeaad9jRzWgpSRkZh81YQmBJHTP8/pml4q1tKUgE4JR7Y5p+5Kd1RByDZ4AlLvoQOIZi840v71B8sKHKZ53dsBwpVy95RjgsCgMIy5Y0t5L+6mjXsmUFpeS+4ZmB171cdiyB1oJ7JiBSy4gfdXn8Kfn8NDMsg5oENDGJ0fRQ6Lx6JAyAuDcRj+9Cwrh0r1qM7t+/6sw32FI78ajixUh2cgqstk2o8yRUkfToIJAImgyhMSwgj1Ui3O0GUWS0GAzXcYBS5MaCSF1AkJ294WLQZHi4bPY0si+FhRUSZilTo+QEMMvH3opEugSoDB4VUkwGmBqHv7D13P8a85k5FGHTjTc+dnbaAuBhy0KKNC+wk80QkjSqoc0lmqSQZKhtOFLF17L6EwLZSz1DTX+4Du/i1e9581Dnhpu+uEerr9qltnb5+lldY5/epss0Vz8+z8hsRJVCAItBakQyNBCEdeSK0Xb9xhvtcn8AC0gyHLa8x4SQSYlvcDHCNE/ZgF3yQIETLTaTNWqdHyfO4cW7qDjBz9t8d6PzeBZixGCRp4jgazIow70gFnfR1oL1uIZQ6X4sefkrmVtplHWsj3wMMBENyWvBPTs4vWgBFQVdPIlsTumUGB1Y7FYtvoetuaT1oq4Fm2gV5TaAokhryiQgqsnBVd/A175nwlffZbkqaf5VIP9C5gkM4xfnNLOF/uy1z9C8P6nVg7yzpYcbSx3FR0L3COLSxRFG4AtwInAnwCPiuN4759SxwBRFFngcXEcX3WAdMcDNwKnxXG8/T4p3D3nvrW45Br8Z++12QACCSgyamh8bP81C4EMSYJHG0MFQ0CXBr5I6NkGLeqkNFiQD1WaCAyjzCCxePT6r2yPCls4gQn2MMMow8wyxlR/v0bRpMY4d9FlHEsFiaVJgx2sZYxZxtmNBSaZYJ4x54Ip3EqVgS5xjiqzNPDICcgIyGhTJSusRj4Ztjg2JKVDgIfhMclLad08y6+eeyW6mzMareT4N53DSLTyoKr5e0/6BjPXzpArwdxYSJDoJauoCm1pN0IqvRQ58ATkUjC7cgQvy9FKMjbd5OTnbKQ9VOWmK3ZhlWTkhCGiZ2/g7N9bjTGWb31xB7d+f5Kwrrhlp2CkIXjoQyr88r92Y7qarqfwEPhGIxdq2Ri6SoLnU82ypY8IcNfKFXTDxUDUIE1ZP9/CFq4gay1bhodY3e7QC4O9BEufQjC0PMWOag0rF9NVsoyVnS5GCjLPCbhQa7qBv1d+uRDM+IvbpTFUtaZmnFXEFuXeVK2ggD3+otCrG0OOYLvvkShBt+LDsM/KPV3CVNMMFPOBj5/lrOumyCI/A2ybGEIvWJAUMNUDY0EK8CSMhu4zQDeD1DqBkxbPoMSlM5ZKIHjqgz2e+SCP55+j8N7ZdWrJk05MWQsGnniy5Irnl0HA9zOHRWFo8eIl7b2ynzzqlcy9Fi5xHG/6DY5bIgKiKGoN7A6L/5OFDXEcNw6Q37OBC4Fzce/8LuA7wCVxHN96sOU62DJHUfQ7wHfiOD7aLVX3jXCxFrxnuUZ3P+S4oaOKecC9vR3WkFMrvmfU2YqmgqZGShUIMECTEXqEVEhQZChShmkCFn/xUUIjabMCD+1cCWTUmGdwgfQES51ZEsYxxeNogW2sJ8f90l9IbRBkSAwCj3zAkQXbGCVEUyEFoEbCNsaLShdYBBV69Fj4lWtR5DiZIVCFeMsRBGQYBBoFCs7b/Sf0dnWZv2EGO59y4w92s+nHU4zt6aIEfUHSrvukvsTTA+6KovOzLG0h20MVevUKw1MtVJYzMz7C1OoxhuZb+MbQqYSkgY+fZlSSlG7gk9SqCAuVbhc/y+kFPihFvdN11gkgCZwrahDDvpekt8BdY8O0arX+tjDLWDvXBOnsWZP1Gp0wAGupZzmBWeaKGbRwFOzxfaZr1f731Z0uw1lGLgTtwMcKsc/jAFIhmAsGOvPiuhpZRnWg3WwrxYznsT3wQAiUtRyf5v39Gkik4JZqyEi+KHB3BgptBaHWrCzeEQtMNQLmhooyZzk0B0SeL2F88XqwFqZ7rlIT4wSJHLBCBRIqyrnSMgPtzAmWcFkTZi35XwW87+qURih5+bkegdq7Tnq54UVf13x3k2WyYzlrBfz0JQGBAnmEjaSb7mpGK/KIK9d+OEzC5SXLhMs/HzUVcnfcrx3woDCJougTgBfH8YsO5tgoit4BvBZ4A/CsOI4noyhaCzwL+D3gkAiXknvBmgsOKFpcCKYEcjQjOBtMDzMQEmnxyRlCIvBJ8UhoMkGbEUBSp4NPwhib8EjQ+DRZiyLtu3A0AV5hFRGAxkeSARKDh0VgCIr9ujhG0mWIEToYBE0aLHS7Ck21sOhYICXAFkcKoFaIFoA2FTwyvCJthgIEHlkRfCzJUYTkWHAuHsBDoJEEGCw5iVZcNf5ZAiytesCuVQ2kthw31aYdBkusKLV2hq8EaUVhBcjcYIpYjIUyW6BTD8k9RWNyHr8QOeNTc7SHqiRhgMlz2g03iksrDz/P6dZr/YDYXq1KbWYOlaSkYdAXLQBK672Ey905MgSwfq7FzdWq63StZbTbW3QdSelEC4AQ9DxFkLryilxj1aCzbhFPANbiG0MtyxnOsv5+F/5z98+nZy3SGIyUS8TN4BVNe4rtQUAgYCLXtKVEYJeIw0zA5sDf610QViAkNLLF7akQdBIDugtVzym9QSxLhZYQTqik2v2vltVwapzYUbj/BXuLFiXAWLx3J5AbsDmv/o/cnWtBvEicCLIU1h4BnuD6OUv14qXWs0evgwsfDm/5Ptw5t/RUUoKVUPMloYLpjlk8jy1cm2Lvp2RFAN98Frz5Kvj2XcvyBLa8HC79NbwvhjPH4dtbFvbqJWnHAmilzh24wKMm4F3nwYPGBdbCRVdZPnnjXkXgGSfAO88TnDLqYqG+f5fL+1HrBYGEWnBkBksfUQGNh4iDEi5RFK0B/hF4PM6qcfHAvouAx8Zx/OTi+2uAvwBWAvPAp+M4fnMURdcVh3wriiIDfDGO45fu55xnA5fgrCkzwCeB98ZxrKMo2gi8DXhRHMefWzgmjuMdwIeX5XMC8CHgMUAX+CrwV3Ecd4v9fwP8MTBRXNuH4zi+5GDqZdl5NgJ3AsfFcby1qJfHAdcAC9f50TiO3zFwzFnA3wMPAzrA54G3x3G8tCU40plqwh/8LcS3w3MeDZ+6EF7/edg9v9/DFjpPJ1QqLDT1hmpfPFBsEYWQsYXIqdCkziwpVVqMUGUaj4SUGm1WYfCYZRU1WhgULUao00EUr3GFTiFqNBq4k7PQKIYYZSVbEXgkjBXnczEsPhlZ3xLjyupklI8iR2FRCFYyS943IFJYZRwC8N0g2sJm5ESQARKCvpVm4ZwLzY4APAxdFdCqS5LAQ0tJp+5jLdQ72V4tVKAtQduJIS2g66l+byqAbi2k0svw2olze0iBZyzdWkgtTfv3aOWe6aVWmoFfsH2xYG2/tAt7zbJOdHCfBUxhnVhAGcP6uSY9TxFmGUExGscW+4SxfbfPoECySoIQpFKSKMlQlvfFWdtzVpBMSjLVd1rRXu4a2ofVpel5TrQU1yyMoZq7+txUcSJqIs3oeAqlDXVjqBv33KZFVhrnQqpYixbuu8JZYNpCIKQglYLQuPrb5nvOCmRwE7340qkvIRbdRb0cfFUE/xa1uj+rwqA4q6jF76KoJV18NgZ0sU8Wd8mKwkIjl55D20WhsYwfb4cf/9u+i2IKa1A7d5fXd3ntI1ZpkOkUHv6Fu8kTWP+Pi9+3d+42G2bSvbddvRue/BU4UBd/+Wa4/LN2H+ncd0mOfsORaIw/6g0se3Gwtfx5nAg5HqgCX9lXoiiKTgP+Fnh4HMc3RFE0CpwOEMfxOYXb5fyDiBcZAb4NfAR4KnAS8HWcG+l9wPm4u3HpAfLxiuN+BJwAjAKXAe8H/neR7EbgscAO4AnA16MouimO42/uL++D5Dzgy8A6nDi5Koqib8Vx/KMoiiaA7wNvBp4BrAL+DSeu3nkIzn3f8f5/g6tucp8/cyWccTxc8l8HPEz0/+3hHsUFh4sloIPGYpFIkiWvniHAx5niK3TI8PtxI3McX4wSAoqg24wAi6SFoEoXiSZkzpnl2UCXYQJy5gmZYj3TrGUDW5dYfQBG2MMsq8gJcOONFC2qgCjsRk6IBLgHVbLgUlqwtSxe8ULTtzgaSRYllsiB0UsLsQ9JMbNMUlMYJfG0ZWy6C3mXXl0tsbYs1u3iZ2kXclzcU+0s1utCQKtWkFSDJccuz6/RatMqrDCNdqe/PxGCPRPjNDo9lLV7NS6DoiX1FLnvEyYpXuHy0UoSZhl+ljsx0q8jV7aJVov5SgVlDENpQqp8pDV4xpJ4yllkhGBeCCq5JhfQ9otnqugUO0o5g86ysnl5Tu4vOvyc2Cvul7Uoa9FC0PF9dvmKjnLPxpznUTGG4cIFtKMasLlRwVrLymZvSR1IIbirEeKlmjDXbMgyslyw2/PQuQvwtcs7b21hvOIsKcbCXOKuZU93MU2gnIso39coJpzIWXJhdvGLJxYfyIXtYuGvuAf+MlGhXXSai6O5m3PuC4G7mYOv1dHjxjkgBnjddzUfeOKRZXk5FoNzDyhcoihaDzwROCWO4zlgLoqivwa+tY/kOe7xPDOKos1xHM8C/30PyvU0IAXeHcexBW6KoujvgNfhhMsqYDKO475+jqLoYuDluNfi6jiOzwceAZwKPDKO4zbQjqLorcBlURRdGMexHbTYAN+NoujrwJOAQyFcbonj+GPF52uiKPoFEOGE1P8Crovj+OPF/m1RFL0X+DsOs3BpNpsMDQ0d0s+DJEkyYG/YPxafLsMockQhXAyycOCkA+ksorCXmEIotBgrLCAGQ5WMbEC0uJBfjcYWsSQGnw4QoOnSQCDo9uNrLBVSulSwSGYYIyTpz+niwoLn8cjZyQlFOVLc2JO9URh6BEsCeJ2LyHUGAsiKmBmLQKIxSLIitSjmk1lI62Gd42uwoRcCIS2Vnl785Xo3GLX3L1qjBFIvNmtWQFIJ0N7+R6ZUk5RKkvaPSX2PxPfZs3IFAEnF1feK+SaVLN/reAFIbcCHJPDJF4RLEctihFjStylrsUZTNZJq3nbnBSw5lUIwBHnOXBiilcBISaosUwtCpLCmKCBZcF1Z298utekHAA+WMdSanhCMpikK1zHNBAHdgbSJEJyQutFIPSXZ3KjwkJkWoTZsrQQYY/u6YC5wFpy8JjhuNnPX4EvkyiozUjA81XXiYfA+LcSngLvHodp7mLWxLo21kOTOQuIVNRjKpc/G/nRGX6UuFyosFRuWwgeHewDuzlIxuEsM/B3DtAq336FqSw8Nx16lH4zFZUPx/+aBbXfuK2Ecx3dEUfSnwKuAT0RR9EvgnXEc70vk7I/jgE2FaFng9mI7wB5gVRRFwYJ4ieP4jcAbC2Hy5IF8dheiZTCfCk787C5cWy8rrlPgLEp3Y5T8jdmx7HsbWHg6TwQeE0XR7MB+ARx2uT74ghySz294prO4/Ox2eM5jCN/6bLjsWrj2lv2WQ+Mxw4lYJA0Wq8FN7BagWIhHyPvOFtf2CZqM0WZxPZeQLgEj+PTIqGCx+HRoMM8kNSwekpw13FDEIFRI70Z0yGKm3i4BXWCCrdRpFuc3pCg0glO4hU2cS0ZAjiSgR05AXgiUkByBJcNjjgqVwkW0gOo7osDHIsjI8NAoBKLvSnLnda9C0NUkdTUQ40Bh+l/M2eBESeYrKj3nNhHWoqXAKInSBpVpjOeTViRBmmOlIA29vkvCKhdXsjAR3HIEuPlTPFeWdnXv4bSp5+FneRFyvOz4hdAJYwiKEUap7zNTq9L1fVa33CsrjKGSuFE3RkAvDOl6yr0oAzEjAqhnGfMqdBaSXCPCsH+y4STt1+HCNmsM855PGio8YxhN0yVDlevGkJpFu5sEKlozrCVznnseB+d4yYTg1PkOWgjualQYTjU3jdWgnaJ9RV5YaWSimRyuMF8LEJ5w1qVUM1e4xpRx0tXD0tPLREGul1pQFipTm0WRoK0z+1U954/xtLOOWJzIWSJgF1xNOLeU1ktEXb9ywW3Tdv9xa4NmNcGAK8gOxMcMnn7QbXV0I4B/eLK7jnvTrh5qHpAWF2Bb8f8JuE4fXKe7T+I4/hrwtSiKAuCVwL9FUTQex3GHg48T2gKcEEWRGBAvJxXbwVl7LPAc4HP7OH4wn4koimrF+Rfy6QF7oih6DM7C8STgmiJ+5ivcNxJ1M26E0tPug3MdXsaH4IfvWbrtxefBq/cvXHKqRVTI0s5R4ERNSoUaO7F4GBYHmLkufmlHKcnx6NBmjHlGWMUkIU261AnpoAloUWeGNVTpAYo5xvHo9YNznQspGYgqccywgjpNDIJdrCXHKyJbBOvYSY8AjQ/k3MkYIbYfCOxm2fUIl4mW5S+CAHwMBo0uyqNxI4wsCwG94Gc5qpPTqblO2etpsurSjqxbVXRqAcNzi8PBc0/RqwX9wNewm2GkcMOCQ88NvxWCXClmR0fQvofSOZU0I0z2DgwwOOtMWAT0hpmbDn+wA2rVqnQDn4m55l7HLxCmWb/jD7KM6doKEIJdjTorWm18FgN6pQWpNShFLwzwtcZL3bNjhMCzlpFe4oSLMWTdHrsKl1ZPSVZ3e/R8r18HHd8nLSwTuZT0lKI64PqYV2qvYGIDnNzpMeN7dJXXn9MFoJFrJpXHdauG3fwygKc1eT2ARgDGEsz2sNYyX3f3T2qLMJq8vRijpKVkQ5qRCkFPG2imzvKSWResVJEw7NY3cn5EzV5Yu2ipaRdWr6q3t0AwgDXOgrJg0Um0czvJYltmQBuGK4J5jdumi/1i2ZNs4axxeMW5gg//1HLLvF14oQHwpWFtw1lV93QtRlvSftjIQnzN3ha/iQpc8kSY7sAbr4LOgCFPAO3XCG6dtbz7ajfI6oote2WBAuq+G0E+GET4W6vgC0+HsYpkumt454/gy7fCclvhY9bAex4Hv71eEXqCr96cU1HwlI2SucSwqn4kxrccmxywpotA0yuBi6MougBnkdjnXOZRFD0IJ2p+gIvVmGNxegKAnTjXzX5jXHBxKZcAb46i6H1Fnm8CPl6U6c7CrfLhKIpC4LI4jqeiKFoFnDWQz0+A24C/j6Lo9bgYl3cBn4rj2ERRNIx7pSYBG0XR03AxNV/eX+GiKFr+83Jve/iB+Qzw+iiKXoyz8KTARtw8MAcOEDnS+bPfg5kWvP2Ld5vEo4sgx+KREhCQQhHXokhR9EgZRhejcpz1IsQUadMioNcjoc4cCRWmWYl1YXKEtJhjJTtZBzirxRCjzCNpDcz/0mAGTcgITWwxxLnJEAu27oyAX3MWBsHCLLs5AV2qyP5n10GupMM0DQI0SRGyC0tH0yz8GHUznJh+u25xbiyNQRd2oYU23WAI65Kzr3wG3miAkDB70yy3/fNt7P7hbkTh8pGAtALjqyUWiTQctNIIktpiHEu/2zOGufFRssAFiNbnu+CpfnntQDmFcAGyC/uCLKPRbNEaavSFgbSWeqfHsq6y7+6DZb8QFjK2lnqv56bEX9bRZkK4jk24eViMEEhryZXCFp8RAqMUo1kGrTbTlZChLCNVikaaoaUL1t3LClT8v/AyK1xMTBMIjCETAlP8VYxleWBRSwqmfK8vWsAJRqpF3UuBqvus2d1ia66pJjkr5139TIceMwtPibVoCXvG604cdHJoLWtiar776+R7CxeBE0q93A2nDovRSd3cxcMsT5saEJLszSG3Tmu+cYdgw5DgOWfsPa+LNoY/+7bhpzsMv5i0rKgKrnuRx9rG3sOhL3zEXocfEv53tO/tZ0/Apc9c/J5pS6KhEQistQe1aOdEXfKFZx6cyf2PHrTYfa46gHv1/uWBaXEBN8ncP+EsGAujih63j3QB8A7gjOL7bcAfxXHcK76/BXhnFEUfAC6N4/gV+zpZHMdzURSdD3wQN9x5DvgU8IGBNG+LougGXJDtJcVIpR3Ad4G3F2nyKIqejhtVdBfO0vI14C+LbL4JfBYncCwuOPZfD1AXCifKBvk4Lij5oInjeGcURU8ojvsbnCDcVOR19CMlvO057g9g48th854lSQSaBrvpMQ6AoofA0mOosDiMsvDSmWIkkcBiZQ5GUmeGHiGWnF1sYA+rsUiqzLOGO7BAiDO0WSRBEeRrljkvOgwXwb1F0bEM0SzmbtFoJB1qS46p0WEHJzHGNCmLDbwqpI/ppxb9mMSFpk30z2Po4hWiwPIk82LueMtP2fHPtzFy3moaJzVY8T+OY+zx6/ZZxbWTRlj3tBP637902lfp5FBtpzTme3gD8QxBqun7SwctIwufjaU91KDe6dKrVDj/BRnHt4oAACAASURBVBOMT6znux++naylWXl8lce+4hROfLSbDG/zz2f41bd2Mb+zy6Y7MmqjilM2Vtm2JWHHDCQIhFK0a1Wq8xnKLgwbdyHYvjHYNCPxPcIiDqbn+9R7CaEx6CCgZwzVIo3SxlmFlELZxQvTSqHSDLsQczE44klKGlnGkNZFoDRMBz7VLCcXmiDP0cJNtR8YQ2gWY4oUILSmp1T/b6G+UqVQWqOMIZeCWq5pKcUez5WzP/rJWmeZEIvNrBXuOai1UkaTxaHZY0nOjO8hhcC3lh2rhty15M7a0beeCJzbZ4Gsb8ooBIp1rXonQ2RuKYPzThU883SPp50sWfuBxAXdiqIguZPGZ09YPCV48CqPB6+6+25BScnHf/dI7qQX8ZXoe9UeyKu1H4uuonKtogcm989N/8518Ndfghu3QTWEL/w5zf/xaWzbDev0aGIJ0AQsn/FDYAhpognxXvtI/A++oL9Pv/Kf6Xz8p8yzghyf4ZedhX/rDoIr/xuQTLOeJmN4aGo4Dd2iUYwasoQkGAwBpu/i8ehSZ5qEKrvYSFKsGQ1u5tshuiT4DDHHLlZRLX6nZ0jmqeDhFohMCrePLWJdXIyORfpQe8rxpLt7rH3Jgzj+lQ8+JFWcziXc+enbuO6vr8NfNlo1CRVpoMg9SVJ1gatekiO0oTVax3huQrQ1jxrn+R86996XJTF882N3cvuPp0lkl/WPbnPOmQ/ny39zKyhFLgXG8xB20TmXKeXWKyrws4zaYICvcYG0qafIpcLPc5S1tIo1jpaTCfAHnvYdlZCvfdp5ulttzctfNRC6ZwxWLQy7py92WkqReItBvaNZhl9Ym3YFHiO5IQduarjJ81IlafuS6ZGqs5hYC4FCacvayRZBMcOtLoafg7PybB6YNXi4WLzRZE4UIYth0aqYUE4KJ1qmeiDgU39c4YW/FZBpCLy776i0MTzl0yk/3G7JM6cmn3CS4LsvrN7tMSX3GYdFYSTiVUva+9B+9KhXMqVweWByRN307PJfYTZNIbdP0vzbq4sZak3R3buhxG76fgFkeO99Ov5fHniFCTvbRV9zG9lHf0Dr8juwJsBiyVFUSDAEqGIm3RTJHlYywSyKjHG29FuRDlW2cjoGi4clKIKGDYYJdjHDCNtZi0AQklAhRwMdQmqPWMODrnnuYau7u+NfJ76AzcEU03YIu7jisAWaQwFaugDX3JOc8fqH8Nt/dtphK8/+FlnstXI+8KobaM0acilpDwiXIM2opmkxc5nlnKeu5MdXzOEbg7AutiXUmlxKMqXcXCmFyNDAVBgykST9695cr/H1Tx6/VxmMsXS6BqNheFjxhS9Ncdk35kksICWdYoRSqDXD+aKQMsDWMMSzhjmlaPoexlpaK6pMDhdiwFrqezo8dLbDdl+RFSOcdg1XWd3sIa2lIwRdJZlX7oadlGZ999wdvoKqG4Lft5B5gpedbfnTh/o8dJ3PSPWo74tKDpNw6Yk/W9LeV+w/HPUPSylcHpgcsTfddlLmX/gl7H/8HL+XFkOBExQJGSE5Po3WB6F+LxaF05rOBZ8h+94dMNJAnLWa2pueyObf+iQVBHVmGGVyMTlwB+eigbCYjbdHSECPNews0kja1GgzWriiMk6xb7k3VXGvSCZ7XPXs77HnhhmwkIeSXCiCXKM9QRp4nPfRR3LK7244cGaHgP0JF4Bf/nCa/3fxZgTQ81R/HaFakiCE4PwL1vHwp6ygPuSz6aYWf//Xd2E0eLmmot0cKEZAKwxJpERLybZ6HSMFI0lKPcuYDUM6SvLtTxz8Nd9wY4f3/N0uJgMX2LxcuKRC8IuhgdFp1o0U2zPRYGYgjujUbXOs6SRcM1QlRVLThjGjyZSkkS0Gbu/2FB0h2FhYmQyQjSqe/5gqv9pp+KMzJA87PuTkFYJ6cNT3PyVLOUzC5X8vEy7/96h/cMow6JIjClELGPnyC4AX0Pyjf4KvXetWEKaGOH01jR+8/t6JFgClqH3mgr02T4zuIZ/V5ITYYmo4sCRUUOhieHOOQSHJSQjo4VMhwyBpMUqOh0+LNT988b0r470kXFXhSVc+9X4tw2/C2Y9bwQ2/aPKTK+aQuWa0l2CFm5PlzZ87k8bIogjY+OAGH/6iC6Obm0z57yumaDZzdu3UCAXPedla/vLPNzGcpTR9n0xKWr5Hx1P89um/WZPXaChSIRjOMuZ9n0RKEiEIrEUDv64um7FICFpSMDHfY7bq1kKqd1PmtWVzo8p7/meds8fg5knNWy53AchD2WJwbWAtTaVIhbOQtaTkzedVeO1TSldOyT2jjHEpOVYob/q+MIbWhjcQ7NiOxu8Pje6wgpQh8CSjN/wF6rQ1Sw/bPIUJfUSuYcsM8pEbEfsY0vlA5kAWl0F2bO4wNOrTGPEPmPbu6HY0l7z3Lm6/NSGRiqav+J3HDfPalx7cStsL5Lnl2a+4C63drLa9gUntOlhmAydmlmJYk6Ro3ydVii3KMteo84bzPN7zlEWhs+510+wJfda1e3jWWVd2+B49KRnPMlIledhJHt+7cPge10PJUcVhURhdceGS9r5qP3LUK5lSuDwwKW96yX3KbyJcjjTm5jTv+OBObt3q5k0RFkZWeXz6PWv4xFdm+MhVOT2psNYykmU8+hFVPvLisQPme9Z7Z2nvyNg9UiVMcjq5JSkCjDe/eYjjx4+sqeNLDjuHSbi8eplw+XApXEqOSsqbXnKfcjQLl8PJH3yuw7dvM5DmmK4hFYJ//MOAlzx237M6lxzTHBZB0RGvWdLe1+yHjnrhUsa4lJSUlNxPXPb82v1dhJKSo45SuJSUlJSUlByjHIvBuaVwKSkpKSkpOWYphUtJSUnJEcnsbM7575/m5tQj0aBCyea/qrNy6J6PTiopOdopLS4lJSUlRyiPevc0k6lg3i+GJyew+u1NKhWPTj106/h0Uy56SsA7nngv5wIqKSm53yiFS0lJyRHLhW+9kyt2SKaqFR5h2rzjFeO87pIZpJAM5zm7KgG+zjljvsNjpeBrEwPztAgwgaJXD93aPlJA4HHRN1Pe8vgAT5Vz7ZQc+5QWl5KSkpL7iJf9/s+YEh6/Pvl4jmt1uDpo8OL/20TUqpzU7uIj2NhNWNPpuSU5jeU5Oyf52PHFatq+hKEKZjBT5VZcXvfGGbSnsBbOWAufeVGd0z+c4iU5p9cMX3phg1M3lC6mkpIjkfInR0lJyRFFczblo2/9NWtmO9y0dhVP37GbV1x/C2/9yXWcOTnDTiW5uVpBAbfXa0sasdBYPE+4eERtIV+ULdJYHjLdZswTKG1ZPd9hTZJw+244+UM5mfLo+j6/aPs87kMtfnZj776+9JKSQ45bnX7x71igtLiUlDwA6Nw0zS0XfBdz0xRBr406Y5Tweeey7rmnUDlh6LCcM21mfP1V17Dzxjl0VTD2hzkXP/q7hL0UmWt0PeR5X3gkleEKt/94F1f/3fW0UkUa+MyPjvCj00/C8zw2jY8xW63wnOt+zTmTUwhPcdtwgxnfoz6w4CFAJgR5bp1byFhop4xLi+7ltIWgmRse1m7xnZXD7KyGkBtWdVKoVZzYCRRWCHYFij//6Aw/mhgB44TQWi9n+5sPT12VlBw+jg2xMkg5c+4Dk/Km34+YNGfTaR/D2zyJCXxWf/cFVDc2YP04sxN/iZ3sYfHRSCw5Yz+9ED868aDzn37jN7j9fTdTo0eNjO7KMab2+CgBLVXBYDGeW+k4yFJOfvWpNL/6K2Z3e/hP3MCZX3ka7Xg7P/mTq9CdjJFhxYa3Pow1f7iRyqqDmzCtuaXFvz72v/C0pVMPsMD0ymGGZ5oEqRMb3UpAa7gKQYAwhqSX0xmpM5ZlGCH40oNPQRfrAEljeMk113HrmpVsXTUOWCrdBIlg1YB42V0N+H+nrIdGANZCZgAB2sBcDywEQz5pOvAKaM0KrZle1XBpjYVUu2MQUPWciynT+ElG1cD6TkoSSK556xArG+Xvv5JDwmFRGHPi/1vS3o/Y9x31SqYULg9Mypt+iDGtFJHnIEEM1+i86TJal1xNN5XU6SHw8cnoelW25ytZyxRt6hg8fBIkCQLLEG0khg5DKDQBPXJSslPWs/rGN7H7U9fTe+1/QLeHT4sWq8gJaNCiTUBT1NFCscFsoYFFIMhQ3MLxNFUN32iUNaTKQwu3+nXVJlS0wcPQlT4d6dNuBOQVhQVaNR+spdFNqXRTNj7vRM756HkAxH/1M+78t01kwMTDV3HWy0/jx++4jrnbWtSSnN2rh6EIgrVAda6NVZLW2BBGCmSuyUOfTCl2eD6nzMyR1CsgJVcev46tw87CcfrOSVbPNrlt43pksZ5PJc146LYdbBtfgWcMWkh+eNxKblw7OnBjLOTF495OoZfDSAC9gcgXa/EaPnno9b/TyV2BAwnB4ppBItdsmOvyoOkOoTbctKLO7RcfeF2ikpKD4LAIilnxxiXt/ai9uBQuB0sURRuALcCJwJ8Aj4rjuFy45P6hFC4Hy1yH3rv+Hf/vL0NgSRnF4pMD3gqf9E8eQesj11IjR6FpUWdaVBBCMWI6jLCbaU7AhZMZqswxRAufjBzFXZyIIcDdEoFAs5rtCAwVumg8trKaUabpMEyPOquY7sd1GMCgCZkjpE1IwlZxPJNiPT1RQ2HItI8AchQCQVf5fUvGAsoaxvM2TRnSVc5C0h71SUNJpiQjrS6rpzs0ayGpr2j5AmEl3UaFai8rrg5yCSiJ1JZeKOkM11hoj62SBO0erbEh8nBxlWWhNUGao7TBCEGvVgEpyIXgo6efTMfzmAmceDq/2WZBRvhaE921nc+fewo3rx4Da3nYXZP8YsNK9MKIoUHhMp+AAoYDmMuW3ueRwN0CJRjpZM61FHjYigJ/ULgY7J4OUgqePN3k2rXDzAce9W7GlouGqAVl2GDJPaYULgfJ/WLjjOP4bw42bRRFFnhcHMdXFd9bA7sX1ohPBvJuHCC/5wCvAc4BMmAz8HngQ3Ecpwdbrv3kfxHw2DiOn3xv8yrZNybLkb4H1qL3tDDb52Byjua3tyGPH0L885XoW2cIbAdpU3qdEInCoKnQxaeNICgC1RISakWXbrHkWMBSQyKRpPi0SKiSM4RHikcbaMB0G/8j36POKOAhSBhjmp3iVHapCU6ztzFrN6BRTIoRpNCcbGbxybBAh1ohWqDfuaPYyQZA4JFyHHfiQyF+DGNME5KiUVgUClAoUnyGi9dgnd3CDeqhGOE63CHanKK3M8UQU4xhCotFTadUTYoWkjlVJUcS2pwugYttlYJe6KEyg9+COa+GyaFX95HS5VFNnGixOGkWGMAU1gyhsGqx01dphrAWKwfaTSFASpQ2Lg9rSYVg59gI9STlpG7CNaNhP+2NlZCHdHsI4LjZJpnvs67ZdcJFCLaNNXjuL+/km6etp+l7GG2xFh6/ZTdT9QrXrV/h8qp60M1BwOo0ZVfXlXM4zdgw7/Jve5I7Rc1ZjIoyW08ixyqYmR5XbRij03BlS0Kf1e9s0xkKXf3mBrSmEQpaWoABJSxYkJ7gf54qOHcCHrle8NOtlpoPD10nufT6jE/83GKN5fdPk/zTH1QIlODD16RceWfO639b8sRTq/3q29UyfOWmnG1tOHeVYOMo3DJt+dE2wV0twddvy0BD6AmwFqngmSdKXvUwwXknBpQ8EDjqdcpeHHXO2UFhEkXRJwAvjuMXHcyxURS9A/iL4u+yOI5noig6E3gTsBYnYu53oijy4zjODpzyvke/6rPYz/43jNeR33sD8qSJpQm2z8Aj3gbbZrCAwcPiARb5iNWweRYzmYLJULRZsDQAGBSaEQQaRYZAAxqDjy2EhiVH4mHIUXRJqLKbk7BIKnSp0SJAAxJTyJEKhoQATRWfFhIfi4dAAIoaTSxVV0ZaCMDQIWcUny4GnxYrEUh8MjQ+Hh3AMMcEqpBAAotPlzPNzYybeSwBBoHC4oscCteNQTDPSlrcncZ29ZET0KHOHEMYfIZpUaODQqNRS44YZa7/OcPrixaAVDjrRoMuswxTyZ1YGbJO6HhWM5q3USQktg7WYoWg1szwUoMRoAxoT5D7klonZW44RFhQdrHEzbrPUHvxsbXL5kmRxiK1YWhqntmJsUUxIAS9aojxFBjDzzduIAlcmYeMYdxodhR5tYTglD0zDCdp3+q0ptntn2Osm3LydJPn//w2/s85p0LVo2IND98+xZ5ayHUThVvHl4wZwZm75/nFyOJ9WNHN+s18PTdUsPTs4jMKYJREjFbpDFpXpKCtJDa3rlX1nIupBWANWNBFRenEcOmNhkt/rUAUcTjVBTeVD9KCNVz6y5xLf9mBUEHNAwL+88sGum1QEuWDDjyXaa4hGbgZVffO0XPfE4DAA2354vWaL14P0EW/vdIXoSXHJseief2wCZcoitYA/wg8HtgFXDyw7yIGrBJRFL0GJyZWAvPAp+M4fnMURdcVh3wriiIDfDGO45fu55xnA5cA5wIzwCeB98ZxrKMo2gi8DXhxHMefWTgmjuMbgP81kMd4UdbzgQrwPeDVcRzvKvZvKq7rScAjgU3Ay+M4/nEURc8F3gzIAcvQ2XEc3xFF0eOA9wJnFGX7B+ADcRzbKIp+B/gOcAHw18Aq4IgbvmCuuBH7se+7L+0E87QPIW9699JEF3wcts0Arv2UaDQ+IDA/2Y575ASKDqL/Srn/NXUkFkmGxAVGWhSWEAqZIYruSpEjyNnO6eSF4U3jM84UurBiKFK8Qvz4pNzFSeRIVrN1YFigwFlLMsD0t0o0Pp1C2mhGmGYnJ5NSZYItaDxu4Aw61KnQ40x+QY3p4tguq/DoUGGEHeQEtM0ZVEROhwYaUYgf3a+2Kk3WshmBYQcb6TAMWGYYoUuNNUwisEW9WEKm8UjIqJMwhjMeQo5HjzFWmFmm5aiLTTFdphimQwWNRAG+XToaZxU7OZlfMc0qNuWnsccbR2QKP8vRErQSdIY9EIIUxar5NrtHG6jcoiV0A0WY5BhR9LuAwSKMwUoJxiATV0Yv1zRmm7TG3CMutHGiBUBKJlpttqxwcSpWSh7STViR5qRSckKWsXV0mBNn5lHWEGQ5N9Tr0M04e2ae82/dzrynuLNagURDZugNB3zt9OM4b8skj9w2ybWrV1DNNY/fMuWe0eHFpj0f6MS1gKzq92N0sP0Lw0rh3EYD8S+2UVgwcgv+gBiQwg3NHvyeW/At9LQTGYUVzBWosF95RbrawH5fQuo+a08tbs8G8rc4MRQql34hzbLh4QBP/kyP776oSsmxy7EyBHqQw+mQ/TzuR8bxwHnAi/aVKIqi04C/BZ4ex/EQcCbw7wBxHJ9TJDs/juPGAUTLCPBtnNBYAzwNeDHwuoU8cM3CF/eThwAuw736ZwEnAE3gC8uSvhjnblo456eL8n4J+BvgyqK8jUK0nAn8J/A+nCh5GnAh8IKBPBXwVJzoWn13ZTwUNJvNe/TZbppako+Zbu+d/q49y842qPcFi79c9/4dYJdZEfY+Zvl20APa26CW5CsGph4TWAIS5liBeywHRVOGxV9y/uWlWxBZaeFGmWQVHeoA9Kjgs1gXkhxJCliaTOCRMsHttAnJ8egWmtQnp0abCk02cBsBCT4ZG7idCl1GmWWIHhvZyhhNRmmR4qPoUGEOjx5VplC0abOaPZzKFCcxxzrGdIvxrMmqvEliQqYYpUulX28GQVa8/pKcddwBwAom8cjQAxYbZcB4YrEDBLSQJL4iF9AJPZSFILeuzwX+f/buO06uqmzg+O/MzPaSuimEkNASMEiRhyJNQEBRAggIWKgCFoqAebGALwgI0l9eIdJBsIA0QxSU4hsEkcADSO+phPS2fXfKff84d7OzLdlJdnZmZ5/v5zOfvXPLuWdmdu8+c85zzm2NOSKJBMOW11K5qp6yukZSRbF1n2RpYwvlq+shkSKIdfzcXVre3ejaerZYU8eU+kYmtcYpTyQZV99IMhaltaiIt4YP4Y2qSoYtq+eAect5r6qSX287kcfGjfEFlMSgJcWHVRXcuf0EZpdWEKxu4tB5yxgST1AdT7DP0jWUtCYY39DMlg0tVCWSpCKO+SMr23NlwI80SrTXbWR9p3leXPgeRX23TI+/TOnrk6nuf72DoP397nx8W3CV9nl0nGUvvU6sd593lvmfG3tNsOW+X+57rtNj4MtK4CIi44ADgWmqulZVl+BbErqTwL+bU0SkUlXXqOqLG3HarwKtwOWq2qKq7wJXAW3BTg2wYgN5LLuGjzPDejcCFwAHhsnFbW5V1bdVNQncAWwTBk49+T7woKrOUNWkqr4H3ERaS0/oJ2nnzZqqqqqNWo6c+HkY0/4yY784ouv+P/9ah3MF637FAqAJh3/7fSN8R1Hq07qX2ktwxNPK8FffOEUkKaKGueH6gArWhmeMh9kqRQRhSb4lopwYrUTCMpM4EgQ0MYJmKkhRmpZz4s8SEJAkyirG4kOrRNj902n+ENrvfRMQYSVjiVPGMD6hiBZGsILRweLwtUd9dw6OEpppJtqh68eRYihrKaWFIpKUkv4rG+3y/ycFNDOcJCVAMVFSRAkYz1LGs5QxrKTtv1/6e94YiTF3xFAqixdSFgZeAT44i6bC1qDwH3A0nvKJruG6luIogYOm8iKIuA7BRpvypiRFSahoTlDRmCARi5AIg5QAaKwsI1Ec6/J7MLS5hZqGRrZZvopxtfWUJhIMa25mVcSxIhqhOfzHXReNsDYWY59Vazl82UqSzvHysOp1OTwk0uqMw9X797EqmaIk1X7W6kSCzVrjbFnfTHEQ0FwS4/2RlTQm21+/b20Jh1UTQBSSRVGK4u2tZmkfoG9Jiacg3jHYIZVqb/Vw+Nac+taw/KA9mTiRav+wGuJ+fSrsVkqlldV2XCy9hQc/GqqthShdp26he74W5kFt5DXBlvt+ua/ZBHS91/ZPPj1nZG53O4YtEt/C/3O/Q0TeAC5V1SczPOd4YJ6qpv+pfhyuB1gOjBSR4vUEL1viE36Xikj6+mZ8y9En4fPFadvavmpXQVqiQddyDxSRo9LWRfCjrNqkOj3PO64oRvSTawj+9SFsPYrIuG6GgX5zb9j/M3DLUwRNcZLDhhDc8zyRC75M7IhdoKGZ1H8+IYgnCVqaCOauJFi0CqdziF50NI3XP4d7aQ5u780obW2GQz5LYvZ8ks/NJRpvIoiUE8SKCCIRWpoqSDXHqGEOERyNDKGJIopJEBAnSoK1VBMQpYFKimlgNB/TArQwFN/91AIUU0RT2EFVQTEJHCkCEiTD4cR+v2jYMtNKDZ9STzlrGEY1DUB5OE6oiWaGkKCMISxa11LjgCGspomhYe5NhChJIEotw/iYSUziXQBaSJ8rJcCRAHzORwpHC0Mopo4ISRIUs5qxlKUFN0W0ECdKJDx3NY2sppoWiml1UUqDOCUkGJdazEvJyaxt3YKVNFBKI0sZRxPVlCYTxFItpIhQFy0hknKU1cdpLY/ikgENpSVE4ilSYVdEPBahKBEQTQV+pFPE5/a0iQbgYkWsGVJFUWsryeIiUrEosW4CnihQ09BMWWsr79cMp76kmMWxGP8pKwHneL2inK8uX8mfR49k8+YWpi5bte7YnWobeLK0pL0wh883iTiCYeUQgbUtCT5sbWX+iApGNLVS1Rxn3yWr113S3x1RSdI5/xfakICKWJh97PzP8DWvqiqFeJLipjjOQUtxeDltTbUHTeEoK6KuPfhoi+WTKf9inYO6Vp/vUhz1X+WSKUgG1JTBmqYU8fpWiEXBpfiBOPbcPMLeW8Q46MEUc9cG/jWmkv4qEsF3CyWT/g1YF7uF37jDIWC3fAW+vK0l6Ba6QglW0mUrcFkU/pyADx7A//Pulqo+AjwiIsXA94AZIjIibHnobW7RQmCCiLi04GUr2oOBJ8OyjgPu66GM+fhAZLiq9tT4uiHdHTcfuEtVz1zPcUGnoCsvuWgEt9/k9e+02TC49Fgc4b/bnx3Wvq1mCJGJPfeEVRy5a5d1Jd3s1yb9e0p3s2m0tYPUAEFrguSjr8Hjr5BcBrEvTqbyxM/Rcv2TNM98n+h2NZRcehjxuStJPfoSbkglqZufpKk1RpQkFTRTTB1JqkjRRA2LKaWOauqJ0UCEZlooIUEpjiQtacm3AVDPcIKwtcYHR145zaxgFCuooYwGduIlIrSQooQi6mmkGEc5EWL+fxIlrGUCUVppoDwcEh3+v8Ixl4nUsIaysKUqAJpcjJaIn7ulKtFCFCgmwajaWhxR5rMDECaQpiVMlwdxihMJ6ouLqauqCLtCAhIxR+Ag2pokWexzLRpLIhQ3JykOfMJtZ9F4gtJmP19NPBrp2twbBKRwBKmAsqYmWoti1Jf4f6xLi9rzORpiUV6trqI1EqE41fFPZvvaej4tKWJpaQlNpcVEWhPUts3PEnGQSpGMRnhh6xo/CmloOaPXNrFNbXuC78jGZj4t7nSfIodPgI107DIjGiFoTVLUmmCsa6GssojRIx0/3D3KEVOKcd28D31pzne76141prBlJXBR1U9EZBZwtYicApThE2O7EJHJ+KDmn0AThO397QHAEmBb4PkNnPav+MTcn4nINWGZPwZuDes0T0QuA24UkQgwQ1XXiMh24X6XAAr8J9znElVdKSI1wBdVtcfcmE6WAFt0atmZDjwrIn8D/ha+vklAjao+28tyzSZyxTFix+0Gx+3WIcgp/dXXKf1V+/PolHFw2I7+yfXHUwbEZ71H8wm/xdXFiR67CyXTT6YsFmMYsPiYP1H78LuUla4lus/WVN1yLME2F9FEFcuZQDFNtFJFI0NwtFJKHckw+RYCSmmhikYCHDWspJUhVLB0XX0qSDCfzSn70la4v79HSRiQpHCU00QKRysljGABMRpZRiUrqaaIBMXEWUkVKaIUBUmqU00UkWI4a0jhKE358VmxsFOtyRVRSoL5w4ZQ1JqioXDuwAAAIABJREFULB4nlgxYW1GalkDqOHHB12lY3MDDez9BKpHEpSAVhd2uF2Zf+BpNxcUkilJUNrQQCXxrRLw4SiTs2ogkkqSK0iZ8A0rrm4gAlfWNBMDSUSMoSiSJx6IMTaZYHMYSkSBgdFMzH1WU8VFZKXtFI1QlfbdJZWsrBy9dybPjRrMskeST0o6XuCGNrQQOasuL1r2WleXF6wK/FPBJVVqyajwJa5IQdZSkUrQ0JWFYKZTG1s3Me+OhRZy4cykVNoeLMf0im8Ohvwncjm/xaBtVtG83+xUDF+NH2wB8BBytqm2ZbxcCl4rI9cCfVPW73Z1MVdeKyCHADcA0fAB0N3B92j6/EJH38Im1N4tIK7415HfAYlVNiciRwGXAK+EIo+X41preBi4P4lt1loQB0i6q+paIHAZcHtYpEr7Oq3suxuSTov23o2jhld1uG/vQsV3WDQluZQhAXSO1R95O4t16SkYXU3XhESw5+ylKlywlRQmtlFJEHSOLGyg9aALRx2tJUEKcUmI0E+BoPv94tr7ua13O0X6u9kTOxLtLGPqZX1MedpU1UeoDkyCFCwJiBJTRSDMxFjKBFhdj+GatJBe10OLKiUaS1E0YwpCFzaQCR1GQwMUcpQ0tNBWVUrF1JQc9cygAFWMrOHHOMV3qs93xW/PHQ55i5dI4TdVl4SRzKRIRRywAnKOkqYVkPEFDaQnRiB8xtu7+QvgWm5RzNAUpmiMxauJxFkYj1BcVkYo4Xh01lMm1DYxqifugBV/uq6OG82lFOYlIhDXh8N+SxlYi0QjDWuLstKKOwAU8XV1KIky8LW5JMmPsCLZpbmGNc6xK+QHxRNy6biaKorREgGSL79ZpTUIswrTdInx/d7uLtMlfhdhVZFP+D072oeepIJmi+c7Z1D8+l6I9xzH0gn0hsmnf5OMrmnlx7O8IEikq8V0iRcSpp5ixV+7LxJ/s0hdV7yBIBTx0xDPUvlsLLoJLJGiZHKVxdRWRIKCxtISVI4YxsXEVNMVpbUz6YdTRCOBorCzn9TEjeX94+/T9TakUa2NRDlhTR0kq4D9V5QxvbmGX2vYRXUvLfCrz5rX1FAUp/rz5aPasayIGrC2KEY9GqW6NU1cSZd7QCoY0xxlZ18yMzWvCrqD29zoaBBSXRWguLiJwEEmmaPlJCTGb98RkR1Z+sZa5n3e43o8KLhvwv8AWuAxO9qGbfjVz5kwApk71d/lobUpQXNZ9g++TT63k/umL+LS6ikVVFevWj21ooigIqEm0j+R5ckglU5etpCwVEHeORZXl7LDMz89CEFBXXMyasIwAqAuHZFfGfY5RczTCquIYT207GurjftSO88OZvycRfnpgCcvrklSVxZhUY/kkJquyFLj8d6fA5dIBH7gMuJlzjTEDX09BC8AhB4/gkINHMPWEjyhOpmiNRihNJkmkkoxIpNZ1iwXAquIifjtuDBOamti9tqHLPZjSc2MdfoK5t2qqGd7QRHVrkrKWBHOKiihOpWitKPLdQPEkB20b5TdH+4Bni2F2mTQDVyF+S7W/SGNMXjr5hBoeuHspMWB1UYyPK8sJXCsj6xKUpgKKW1o4+pMl1EejJEuLqC8pIh6NsGBoFZvVNlBXFKUk6XN72kY5vTF2CMuKY7xxSTWVpd11wdnwYFNYCjHHxQIXY0xeOvqQIRx9SNd5HVOpgJYENLSkGFkVZdmaBBf8vp6HFwREklBVXU7dtuOoLSmipq6RQ+cswUUifDyskrK6FnbYIqCytKKbMxpjBgILXIwxA0ok4igrhrLwPkGjhsa458yhuPvquG9ujIZUis3qm6mub+bgPYu59qJJtLYmWfRpnPGbFzFqiI0CMoOHtbgYY0yeuvuEKu5e96xTi0pFhDHDLGAxg5EFLsYYY4wZIKzFxRhjjDEDRiGOKrI5qo0x/SqRSnHCjAT3v57AXesfF81KbPhAY0zG7O7QxhizCR55J8HRj/vl333Yvv6XCqsaE0z/il2SjOlbhRGspLMWF2NMv2kLWrrzm3f6rx7GmIHLvt4YY4wxBapQuofSWeBijDHGFChLzjXGmCxaVGtJusb0pUJMzrXAxRiTNza/Dfa604IXY/pKIQYu1lVkjMkr/16d6xoYU0gKI1hJlzeBi4jMAy5S1d/10/k+Ai5X1Xt62B4A+6rq8314zgRwkKrO6qsyjSlE7toNt7rEgLa95p4KE4fnzeXMmLxRiDku9peeRkROAL6pqofmui7GbLSqr0N9svf7H7cn3H8BBAG4gfPtLD202fIuaPphitIi6/02ptBZ4NLRkcCfc10JY9juTHh/cf+c64EX4YGjNu7YmIMUkApgVDV8OB2qy/u0er1VdmOKIlK0TrPLmjFtCiWvJV2+/YVvISLPAHsA84AzVPUFABE5HfghMB6YA/xYVZ8Mt+0E/C8wBYgCLwJnqerH4fYi4Crg2/jL7A2dTywipcDBwJkdV8tNwNaAAqer6kfhhnLgSuAooAx4HjhHVReE26uAm4CpQB3w32mFDgM+BfZS1dfS1v8TeEpVL9uI984UisWL+y9o2VSJtIboZbWw8/kw55acVScOvLk8wWdr8u3SZkxuFGLgkm/tqqcC5wBDgKeA3wKIyBnAj4FvAcOAC4FHRGSb8LgAuAQYB0wE6oH0XJmfAIcBewFbhvtM6HTuQ4C3VHVJ2rozgGOAUcDbwGMiEg233QDsGT4mACuAmWnb/wfYFvgMsCNwBD6oQlVXAw8Cp7WdSEQmAZ8H7trgu7SJ6urqbDmfl6/4KwPWghVAz6+xP3rc//hu1/Pasi0PlOW+VoijilwQ5EfqTpice7OqXhM+nwK8BQwF/gVcrar3pu0/E5itqpd3U9YOwJtApao2iMiHwK9U9c5wewWwGt+ic0+47i7gPVW9OnweAKelHVMeHnMAvkWnAThcVZ8Kt1cCq4AvALOBRuCrqvpMuH1b4APgAFWdJSL7ADOBsaraLCJXAZ9R1amb/m5uUH586KZnbiO7bnLt2L3ggWldVs+cOROAw9/PfvrYsu9BTaW1uJgBJytRxYfumg7X+22D/xrw0Uu+tbikt483hD+r8K0kN4vImrYHPoAYByAiW4vIIyKySERq8YEOwMjw5+b4ricAVLUBWNb2PGwlOQx4tFN90o9pBJaHZdUApfguq7bt9WGZ48PtJenHA3PTCw5HKy0CjhGRGHAScHv3b4sZdGb/Mr9HMdZUwzZj4MjdYY9tYGgFXHxMt0FLf/rb1yxoMaYj1+kx8A2Uv/D5wMWq+mAP22/B54zsqKor01pc2j6lRfjuIWBdi8uotOP3AZaratr9aqHTMeX4gOQTfADTgg+o2vJoKsMyF4bbW8PjPw6L2LKbet8GfAfftZUEBnAfgelTu28PqUc27thEAr5/K5TG4OQDYFUdPPxvuPUffntNBXx6N8QGyp9/7wSWlGtMF4XSPZRuoPyl3wBcEnb5vI5v7dgVWKGq7wHVwIfAGhEZCVza6fj7gP8SkVn4AOdqOoaeR9K1tQXgvPCYRcCv8C0ss1U1JSL3ApeJyDvAGuA64D3gpXD7H4BfiMhbQBM+kbeze8P1FwN3q2oGY1iN6UEsBref2XHdwbvCLWflpj5Z8s7JcMur8NkRcNquA+VSZozZVAPir11VbxeRVuBufMtFHHgVaGuXPg+4FagFFgDXAF9LK+JKYDg+NyWJD4Tmp20/Evh6N6e+A3gE2Co83xFpwcV5+GDmZXy30Av4nJe27T8EbsYHM7X4UUVHdnpda0TkIeAE/OgkYwy9bz258ZAsV8SYAa4QW1zyJjk3V0RkZ2Cmqo7P0fkvwQ+L7s9L8OD+0E2/yyQ599XjYZfNB8R3KmP6UlYijPfc9R2u99sF5w/4SMauDlCMbz3pdyIyGjgdP+zamEHP8lSM6VuF2OIy6K8SqvoS8FJ/n1dErge+C9ynqpaUa4wxps9Z4GL6jKqeD5yf63oYY4wpXIWYF2CBizHGGFOgCrHFJd8moDPGFLD15bAsPK3HTcYYs461uBhj+lV68DL9pQTNAZy3exTnCu+boTG5VogtLha4GGNy5ge72yXImGyyHBdjjDHGDBjW4mKMMRlw1yYAOI4qvjW5Lse1MWbwKcTAxZJzjTFZ0Ra0ADzA3pz4/vY5rI0xg1PQ6VEIrMXFGNPnnnw/0WmNYw1b5KQuxgxm1uJijDG98CfNdQ2MMYXKWlyMMX0uUZzrGhhjvMJrcbHAxRjT54J4rmtgjIHC7CqywMUY0+caWnJdA2MMFE5CbjoLXIwxfe7NlbmugTEGCrPFxZJzc0hEThaRj3JdD2P62ge5roAxBrDh0DklIhOBucB4Vf2kj8rcIizzWVU9sNO2WcDTqnp52rpLgH1U9aC+OL8xxhiTTakCbHEZMIFLXxGRIlVtSx08DVgDHCAik1TVvigas4n2vqPzHC5t+uAC+vgrMHcZHLYr/PkluHYGfNLP/VLDKuDYveCak6GqrH/PbYzJPHARkUrgEuAooAZYAHwXeBW4MlxfBjwPnKOqC8LjZtG1BSMA9lXV58PWjH2B2fiAAuA3qnpxuPx6+PP98LirVPUyERkBXA0cApQC/wecrapLw3PMA+4CDgB2B74D3C8iUeDUsM4nAWcA08Jjbgrr8nkR+QmwCPhv4GdARETqw7rsCLQCdwC7AsXAG8C5qvpK2us8Kjx2W6AJuFNVL+zmvf0ycDdwuqr+pafPwJh89sKaLBV88xNw1u1+edo90JyjoUurG+DWp+DJ1+Gj6RCxHneTvyzHxbsT2AP4IlANHAksAW4A9gwfE4AVwMwwQOit/fCB0GbAVOBnIrJ3uG2n8OdkVa0MgxYH/BnfdbdDeN464A+dyj0dOB+oBGaE66YCo4H78IHNSSJSAqCqZwHPAZeF55qsqg8AVwCzwnWVqjoH/x5OD889Bh/APSIiRQAicijwW3ywNwKYBDzR+YWLyHeB24HDLGgxphuPv9K+nKugJd3cZbCiNte1MGa9CjHHJaPARURGAccC31PVuaoaqOqHwBzgROAiVV2kqg3AucD2+FaO3vpAVW9R1YSqzgb+A8h69t81fJypqmtVtRG4ADhQRDZP2+92VX0trG9TuO4M4K9hy8x9+CDsqAzqCoCqLlDVx1S1MSz7ImALfOsKwNnALar6l/B11arq82lFOBG5CjgHnz/zCllWV1dny7ac1eWebEqZLbtvvW45iOW+lSM1egiMqALy4z235cJY7msBrsOjEGTaVTQx/Nk5F6QG300zp22FqtaLyDJgPPDvXpa/uNPzBqBqPftvCZQAS0U6xDfN+OChLYl3XvpGEZkAfAn4WljXFSLyGL7L64+9rGtbWSOB64H9gaFAKtxUE/6cCDy6niJGAWcBZ6nq/EzOvbGqqqps2ZaztjwpCh8k6damlF9y8TdgyzEwdxnuyN3hD8/Bb/4OdU30q5IYHLQTkbvPgmh0vXW2ZVvOdLmv5Vuw4pw7GDgeGBUEwVTnnADVQRD8o7dlZBq4zAt/bgu8k7Z+OdCCDyQ+hnW5MKOAheE+9UBF2wEislmG5051s24+PrgZrqrdbe/p2NPxrU13iEjbtnKgSkQmq+r7PZyvu3VXAmOBPVR1sYhUAbW0ZyLOo731pTtL8Tk2M0Qkoar3rWdfY/Le++fFOtwZul0fNFSfeED78k5bwlUnbnqZxhSwfOoecs6dDfwQnxd6TLi6CfhfYK/elpNRe6uqLgMeAqaLyEQRcSKyDbAVcC9wmYhsJiLlwHXAe8BLbYcDR4hITfjP/ZeZnBsfHKXoGAQovjvpxjBJl7D843sqRERiwCnAr/DJtTuHj0lhfc8Id10CbNPp8CXAFiKSfieWaqARWB0Ga1d1OuZm4HsicqiIxESkOi1vx78I1X/hk4uvFZEfrOc9MMYYY3otz7qKzgUOCoLgV7Q3BLwHTM6kkI3pKD4VHyw8i0+EnYFPSj0PH0i8jE+wHQscrqptjcY3hBX8ODz+r5mcNMwf+TnwRxFZIyIXhq0sR4av4xURqcOPStp/PUVNBYYDN6jqkvRHWMe2JN0bAAnP9XZ47IP4FqQl4fotgYvxLUsr8SOKXgDWNZSr6l/xo6SuAFYB7wNf7ub1vYof+XRhOJLJGGOMKSRVtPfCtDUGFeFH5/aaC4J8akgy/cQ+dJNVX7o1wZNd8g1TBNPsttHG9CArzSHPurs6XO+/EJyas2YX59xDwGtBEPzSObcqCILhzrkLgJ2DIPhmb8sZdBPQGWOyr6E51zUwxkDefUs9G5jpnDsdqHLOvY/PCZ2aSSEWuBhj+txWI+BfS3JdC2NMHuS1rBMEwWLn3G74aVK2wHcbvRQEwfoG13RhgYsxps9V2kz4xuSFPGtxIfD5KbPDx0axwMUY0+eO3xl+MzfXtTDG5FOLi3NuIT3EUkEQbNHbcixwMcb0uf22jgHpc7kE+CmXLDnXmP6UT4EL8O1Oz8fi53W5P5NCLHAxxmRFMC19IrqlPDb5NTLMwTPGFJAgCJ7tvM45Nwv4G3Bjb8uxwMUYkzXBNH+JmTnztRzXxJjBKaOs19xom3W/1yxwMcYYYwpUEMmfriLn3KWdVpUDXwGeyKQcC1yMMf2iLp77OzobM9gE+RO3gL/pcroG/E2KM7pHnwUuxpisendZgpseHUVVvJU//fl5/jplJ+I/zt7dcI0x7fKpxSUIglP6ohwLXIwxWfXdC//D85N3JohGIQjY96M3gc/lulrGDApBjhs6nXMH9ma/IAj+0dsyLXAxxmTVwuGjOOWVZzn8beWFiZO4ej8bWWRMfwmiOW9xubMX+wTAVr0t0AIXY0xWbbVqOXc+eAsAR7yjLKuo5vV3d2en7YfmuGbGmGwLgiCjEUO9Ydlyxpis+syyRby22USe3Wp7Us6xzcqlrP14Za6rZcygkIq4Do9CYC0uxpisWVPXyj27foGb9v0KAIe//TLzqofz9MML+L/Dts5x7YwpfLnOcUnnnKsGLgG+AIyE9ml9M5nyP49ekjGm0Jx7ztM0lrbfcfGxKbvxxuZbMaY+nsNaGTN4BBHX4ZFj0/GZ+ZcCw4GzgQXADZkUYi0um0BENsfflntLVZ23iWVNBOYC41X1k02vnTG5tfzuWURWJdli9XLmjRjdvsE5Xpk4ie1/NIeFFUOov3RE7ippTIHLs3lcDgG2D4JgpXMuGQTBDOecAjPJIHixFhdjTN9pboGPF5ByR3H631Nsu3YFC4eO7LLbnKEjeW/MeIoiji9981nufXZNDiprTOHLsxaXCLA2XK53zg0FFgPbZFKItbgYYzbsmCvh4Zd7vbsDkqVlfO0dZXRDHdP3+hKvjG/PaSlyjqRzrC2v5MMttua8/0ty0uxWcM4/Ovnn12Dfre1yZUymUjmPVTp4HZ/f8gzwHHAzUA98kEkhdiXIgIiMAW7Dv/FLgavTtt0DJFT1tLR184CLVPV34fN9gSuBzwCr8f1916tq0E8vwZjM3fSXjIIW8IHL1Y//nu2WL2a75Ys55s3ZTL7gf1hSPQyA5uISJqxYyuZ1q/hg5FhWVVR3G7C02e9RmHdyggkj7ZJlzAB2Ou0Juefg/x8OBU7MpBC7CmTm90AtsAVQBjzU2wNFZArwOPBt4C/AtvgbSy0H7u3zmhrTV86+K+NDVpRXsmDoSLZf/ikA1S1NbL1y6brABWD+yNHMHzm6pyK62OH3UPfDjKtizKCWB91D6eYHQZAECIJgOXDaBvbvluW49JKIjAMOBKap6lpVXQL8IoMivg88qKozVDWpqu8BN5FhpNkX6urqbNmWe788pIhMmwQ/rR5OPBpd9/zdUeN4bdzEDEvp6Itjm7rWzZZtucCW+1rgOj5ybIlzbrpzbp9NKcQFgfVS9IaI7AG8CBSpaiJctw3wIbAlfmx6j11FIvJXfODTklZsBFioqlP6eVSRfegmM+6ojHaPR6I8usNuHPvGiyyqHsZnz7uG1ZXVPe4fS8RJRMMG4B66jIJp1kBsClpWwoo/jX2gw/X+2MXH5Sx8cc7tAnwDOB5IAX8E/hAEwZuZlGNXgt5bFP6cAHwcLqdPZVwPrBvXKSIxYFTa9vnAXap6ZjYraUxWBI90XTfy27CyESYMgflrO2yKpZIc/uZsPh4+ipqGWpqKSzqVF3DU6y8ypmENQSrgwxFjeG1YDWcePp5ffCHGnFUJthpulydjNlU+JecGQfAa8BpwgXPuC/gg5hnn3JIgCHbsbTl2ZeglVf1ERGYBV4vIKfgcl5+n7xJu2xL4FD/BTlHa9unAsyLyN+Bv+FaPSUCNqj7bDy/BmL614nc9bnLAK2N+yLzR45h2+Ek0pwUuLgggkeCfE7fh+Nf+zRW/O5SqYWUdjregxZi+kWc5LuneB97Fz4W2bSYHWo5LZr4JlODf6OfomFT7e+Ax4FV8i8wC2ltpUNW3gMOAc/Hj1pcB9wA1/VBvY/rd3ktu5JPyahpKOgYl+8x9l8PeUapbWvjlfV2DFmNM38mnHBfn3FDn3Hecc8/g/0/uD1xFx96JDZdjOS6Dkn3opl+8+Ooqjvxzki9++CYvTJzM1quWcuOjd3HLjnvz66e/kevqGZNPshJW/GH8nzpc77+58Nhc5rg0Ai/gc1seCoJg7QYO6Za1xxpjsmbPzw1n6TNxHtlxT6bNeowRjfVc9OXj+NFPds511YwZFIL1zI+UA1sHQbB4UwuxwMUYk11BQHNRMZcffAwAe895l6a1LUBVbutlzCCQZ8m5mxy0gAUuxphs69Qdvaq8ki/sNjRHlTFmcMmzFpc+YYGLMSa70gKXaDLJeyPHUByzS48x/SHXCbnZYKOKjDFZtdnq5euCl2QkQkVLY45rZMzgkXKuw6MQWOBijMmqRVeNZ+TapRS3NFHWUE/dxSM2fJAxpk/k2XBo55w73Tn3D+fcG+G6/Zxzx2ZSjgUuxpisu2uP13hox1k0XjJswzsbYwrVpcB3gNvwNysG+AT4cSaFWOBijDHGFKjAuQ6PHDsZOCwIgvtpn09sLrBVJoVYhpwxxhhToPIgWEkXxd/XD9oDl8q0db1igYsxJut2+d4fGPdpC3C3X9HdTRuNMX0u13ktnTwBXO+cOw98zgtwGTAzk0Ksq8gYk3XjPm3pOJ+5OypXVTFmUAkirsMjx84DxgJrgSH4lpYJZJjjYi0uxhhjTIHKl64i51wUOAb4BlCND1gWBkGwJNOyrMXFGJNVLe7oXFfBmEErX1pcgiBIAtcHQdAcBMGyIAhe3pigBSxwMcZk2dpoLDu3vTXGDDQznXNTN7UQ6yoyxmTVkGS8y7oALJgxpj/kSVdRqBR4yDn3b2Ah7SOLCILgxN4WYoGLMSZr3LUJZg8fw+6rumkRbmmBkpL+r5Qxg0geJOSmeyt8bBIXdLpzqxkU7EM3WeeuTaxbTvzXcUQ7/9rtMwGeu6Gfa2VM3spKhDF9p792+MP7wetfzatIZmPkfYuLiATAvqr6fI7rMRE/w18jHf/xH6aqszax7P2Bp1U17z8PY9ZnzsoEW9/ddf3jk3dm6vuvdVz5/Hw/LNrmdDEmawKXP6mszrkDe9oWBME/eltOQfyjFJEiVe3akZ4dk1X1k346lzF55/bZCc54LrNjjjl5Gi0//Vb3G7ub08WCGWP6RJ51Fd3Z6XkNUIy/X1Gvp/3Pm8BFRM7BT04zEqgFfgt8Ndz8pIikgPtV9TQRmQfcBRwA7A58R0QeA64EjgLKgOeBc1R1QVj+LOAVYCJwCLAMOF9VZ4TbHfBT4AdAeXj+HYHnVPWSXtT/aOC/w/LnAZeo6qMb2i4im+FnE4yKSNu0x2eq6m97874Z05/Su38y0RrN8FJjLTHG9Il8mccFIAiCLdOfh3O7XATUZVJOXrQhicgk4Ff4bpcqYArwmKruFO5yiKpWquppaYedDpyPv8/BDOAGYM/wMQFYAcwUkWjaMScB1+Nn7LsJ+K2IlIfbTgB+CEwFRgOLgf16Wf/PA78HfgKMAH4G/FFE9tjQdlX9FDgUSIavsdKCFlOIGjMNXowxBS2c2+WXwAWZHJcXgQuQwCcmTRGRSlVdo6ovbuCY21X1NVUNgBbgROAiVV2kqg3AucD2+BaZNg+o6r9UNYW/rfYQYNtw24nArWGZceAa4NNuzvu2iKwJH6+G604BHlbVJ1Q1oap/BR4FTu3l9n5VV1dny7a80csbJ6AkmXlrTT68Xlu25fz/+1oP1+mRfw4GUpkckBeBi6rOAb6Fb0X5VESeF5FDNnDYvLTlGvz48DlpZdbju4PGp+23OG17Q7hYFf4cB8xP2x7gx5l3NkVVh4aPz4XrxqefO/Rx2rk3tL1fVVVV2bItb9TyHjVslBP0n0Q3vFsXuX69tmzL/b3c1wLnOjxyyTm30Dm3IO2xAngQn6bRa3nTdquqjwCPiEgx8D1ghoiMoOehu+kR2nJ8q8uW+IAAEakERtF98NGdRfguJsLjHb0PLBaG5063Vdq5N7Q9o2jTmFx58SR/yWiJJxhzIxw1Ce48PMZ1/04w7V89H3fVE3/o/Ula7ofi4k2sqTEG8i4599udnjcAHwRBUJtJIXkRuIjIZPw/9n8CTfg7Rwb4f+hL8N05PQ6HVtWUiNwLXCYi7wBrgOuA94CXelmN+4CrRORh4B3gHGCzXh57D/CMiNwHPI1P/j0K2L+X25fgk3O3VNW5vTynMTlTUhRj9bT25z/6fIwffd4vd5fAO7yxvss6AN66HqZM7PsKGmOA/ErOBXYLguDaziudc+cHQXB9bwvJi64i/HCoi/FdOWvwQcPRqtoMXAhcKiKrReTW9ZRxHqDAy8AC/K2zD1fVZC/rcC9wM36Ez1Jgc+BFfEvOeqnqC/jE32uB1cDVwLfb8nR6sf0DYDrwUpg7c0Iv62xM3gmmxQimtX8n+tLbL3ef3/K3CyxoMSbL8qmrCD+ytjsXZVKIzZzbAxGJ4AOgC1RnXqbTAAAgAElEQVQ1g3buAcE+dNMvdrgjwT0/PhtZtbTrxtTD+XYfFWNyKSt/DNfv9Y8O1/vzXziw3//o0iaemwkcRsfXuhXw8yAIJnQ5sAd50VWUL0TkOPzQ6gg+WagC3wJjjNkIb50WI356N0ELWNBizODRNvFcKX4OtjYBPlXi7EwKs8Clo7Pxw6TB3wjqK6q6Oof1McYYYzZaHnQPrZt4zjl3byZ3ge6JBS5pVHWfXNfBmEIT3XEzgjc+zdMpJIwpbPkQuLTpi6AF8ic51xhToCKv35TrKhgzaOVTcq5zrto5d71z7hXn3Pz0OV0yKccCF2OMMaZABRHX4ZFj04HPAZcCw/HpGQvwt+zpNQtcjDFZ98y1h3YcymY3UDSmX+RTiwt+DrOjgyCYASTDn8fh7xXYaxa4GGOyrmnSGP7y2Ck+YLGgxZjBKoKfYBag3jk3FD9/2zaZFGLJucYYY0yByoNWlnSvA18AngGew0/6Wg98kEkh1uJijDHGFKg86yo6nfYbJJ+Dv8XPUCCj0UbW4mKMyZr2+xZ9GUgRTM1lbYwZfPIgWFknCII5acvLgdM2phxrcTHG9AMHRLq9AaMxJnvyqcXFeac75/7hnHsjXLefc+7YTMqxwMUYkxUz/tM5SMmfb37GDBb5FLjgh0F/Bz9D/Rbhuk+AH2dSiAUuxpisWFWf6xoYYwLX8ZFjJwOHBUFwP+03+52Lv9Fir1ngYozJiqGlua6BMSbPRPGjiKA9cKlMW9crFrgYY7JiSFGua2CMybOuoseB651zJeBzXoDLgJmZFGKBizEmK774VK5rYIzJs8DlfGAz/CR0Q/AtLRPIMMfFhkMbY4wxBSqV+2AF59yYIAiWBEFQCxzpnBuFD1gWBkGwJNPyLHDphogEwL6q+nzaulnA54E4kATmAJer6sM5qaQxxhizAUF+jOb7AKhOe35LEARHbWxh1lWURkQ21Ct/mapWAiOAPwIPiMik7NfMmIHlHrX5WozJB3nSVdT5xPtvSmF52eIiIucA5wEjgVrgt/hx33Pxw6l+jG9mehb4Vvj8VCCFDy5uDsvZHLgD2BUoBt4AzlXVV8LtlwD7Aa/i7075qohsFlbjSRFJAferaofZ/VQ1ISLTgauAz4rIeOAKYBKQwN+H4RxVXRae53jgYmBzoBF4QlVPFhEHXA6cAlQBK4HrVPXXm/4uGpM7p8zKYOfmVvjqL+Efb2arOr13xsFw6/dzXQtj+kwe5LUAHW8Ov6nyrsUlbMH4FXCYqlYBU4DH0nY5GtgHP3nNRGA28DE+4ecU4H9EpG1imwgwHR/kjMEHKI90alnZD393yvHA0aq6U7j+EFWt7By0hHUsBs7Edxu9DrQAZwE1wGfDutwY7lsO3AecGb6erYA7w6IOBk4C9gi37QH8K4O3y5iB7+Yn8iNoAbjtKVhjE9AY08dizrkDnHMHOucO7Pw8XNdreRe44FssHDBFRCpVdY2qvpi2/TJVXaWqK4G/AHFVvV1VE6r6BLAa2AVAVReo6mOq2qiqTcBF+IBn27TyFqjqdaraqqqNG6jbhSKyBj/T3xH4QOcjVX1eVV8O67AEuBr4YtpxcWA7ERmuqg2q+ly4vhUoDV9rqaouVdVXM3/LMlNXV2fLtpzV5Z6/YLWvX7d/Y0sP++ZIczwv3kNbHpzLfS1PuoqWAXfhv7Tfie9dSH9+RyaFuSDo0xacPiEiRwHfx7dAvIGfJvgDfFfReFX9JNzvEmAfVT0o7dh5wEWq+jsRGQlcj+9PG4rvShoC7K+qz7Z1Falqh2hvPcm5T6vq5d3Ud1d8V9FOQDk+8KpUVRdu3x8/DGxffFLvdar6h3DbGfg7Y+4I/Bu4UFV1I962TOTfh24KyvruSRRM69RDvaIWdp0GC1ZkuVa9sM928NwVua6FGZyyElX8bOprHa73V8zcJS/6jjZFXua4qOoj+C6dYuB7wAx8nkqmrgTG4rtiFotIFT5nJv2DS3VzXKb/2O8HHgK+rqq1InIYaRPqqOosYJaIRIHDgYdFZLaqfqyqtwG3hV1KlwCP0H4PB2MGpD2qYHZvv0SOrIb5t/kumsYWuPxB2KwKnnkXJo+Gf74O767q/ckd8MMvw/sL4dOVUNsCW20Oz74JI8qgogL22AYa4lAWgcV18MCPoLQEhlZuzMs1Jm/lw3DovpZ3gYuITAa2BP4JNOEnqgnoPsDYkGp8MuxqEanEJ9P2xhJ8d9LzG9ox7Txrgbowv+YnbRtEZDQ+J+dpVV0bdjUBJEVkN6AEeBmfJ1OH7yozZkB78buxzO8EPbTSP6Z/zz+/qO/rZcxgkyfJuX0qH3NcivEjcBYDa4Bz8Am5zRtR1sXAKHx/2hvAC/g5WDbkQuBSEVktIrf2Yv8zgNPwgccjwINp2yL4RN55IlIH3AycpKrz8COJ/hdYEdbxEOD4XpzPGGOM2aA8yXHpU3mZ42Kyzj50k3U9tbh0yXExxkCWclymfe3NDtf7ax/97ICPXvKxxcUYUwCu3iHXNTDGFCL76mOMyYqxw3JdA2NMMODbV7qywMUYkxXl0VzXwBiTJ/cq6lMWuBhjsiLVmzR4Y0xWFeJwaMtxMcZkxTF72PciY3KtEEcV2ZXFGNMPAiAgmFac64oYM6gUSrCSzgIXY0zWtA19njmzbSLpqbmrjDGDUKrw4hbrKjLGGGPMwGEtLsYYY0yBsq4iY4zZCIe/fzAQgfcT1J8FFaV26TGmP6QKcDi0dRUZY7LKT/0fpe1yU3kTJFMbc89UY0ymCnFUkQUuxph+0PGCGbveAhdj+kPKdXwUAmuvNcYYYwpUIU5AZ4GLMcYYU6AKpXsonXUVGWOMMWbAsBYXY4wxpkAVSl5LOgtcjDHGmAJld4c2xhhjzIBhybnGGGOMGTAscDFdiEiRqsbzvUxTYP48G35wGyxeneuawJG7waM/zXUtjDHdsByXAiIilcAlwFFADbAA+C7wKnBluL4MeB44R1UXhMfNAv4DTAQOBK4Qke3wU4M2A18HGoBLVfXWtPPtG5b7GWA1MB24XlUDEdkfeBo4BfhFWJ+qbL12M8AtWwPHXAPJPJnE7c8vw9sLYMoWua6JMaYTm/K/sNwJ7AF8EagGjgSWADcAe4aPCcAKYKaIRNOOPRX4X2BI+BPgGGAmMBw4G7hJRCYAiMgU4HHgGnxQ8lXgLOCEtDKjwKHALsDovn2pHdXV1dnyQF5eujZ/gpY2r8/rsc4QdHtI3ryftmzLebRsNswFQfcXlUImIqOApcAOqvp22voIvrXkcFV9KlxXCawCvqCq/w5bXOao6qlpx90D1KjqV9PWLQdOU9UZInITUN7pmB8Bh6rqQWGLy/8BE9padrJs8H3ohSSZhJ1/BG/1x69KL8X/BLHuG3D9vYq6CqYN2gZfY7qTlaaR406e3+F6/8A9EwZ8E8xgvXJMDH9+0Gl9DVAKzGlboar1IrIMGA/8O1w9r5syF3d63kB7d8+WwIEiclTa9giwMO15qtNzY7oXjcJr18GMl2DWGzD9SUhlIRYtikDEQUuyfd2ISqhrgNbwfHtPhmd+0WPQYozJLctxKRzzwp/bAu+krV8OtOADjY9hXYvLKLoGGZmYD9ylqmeuZ59AVa0lxPROLApHf94/fv3dXNfGGJOnbFRRgVDVZSLyEDBdRE7GBxZbh5vvBS4TkXeANcB1wHvAS5twyunAsyLyN+Bv+K6aSfjupWc3oVxjjDGmR5acW1hOxY8OehaoA2YAY4DzAAVexo80GovPeUn2UM4GqepbwGHAufgupWXAPfiuKWOMMSYrkq7joxAMyuRcY8m5pv9Ycq4xvZKVsOKw0z7pcL3/yx2bD/jwZTC3uBhjjDFmgLGvPMYYY0yBslFFxhhjjBkwCjE51wIXY4wxpkAlbTi0McZkpgRoWZcP7i+ilphrTP+wriJjjMlQ87QYM2fOZHktnPqtqbmujjGDStK6iowxZuPUVOe6BsYMPoUyd0s6Gw5tjDHGmAHDWlyMMVnlJ6A7BGgisJ4iY/pVId6ryFpcjDFZ0z5rbhSo6HEWXWNMdiSd6/AoBNbiYozpJ4Vx0TRmICnErwoWuBhjjDEFqlBaWdJZ4GKMyQrrFjIm9xKFF7dY4GKMMcYUqkQBdtFacq4xxhhjBgxrcTHGGGMKVLzwGlzyN3ARkYnAXGC8qn6S4+p0ISK/Br4BlAJbqeqyHFfJGGOM6SBuybkDi4jMAp5W1cvT1l0CXAQ0h6vqgMeAc1W1qZfl7gWcCkxU1eV9WWdjjDGmr8RzXYEsKOjAZT1mqepBACIyDngC+Dnws14evxWweGODFhFxQFRVbdiFMd1picMBP4eXPoLiIjjlALjkOBh1Svf7Rx0kg+63ZWJIGdx8BnzrC5teljF5oLEAW1xcEPTBH/smEpFzgPOAkUAt8FvgNnxX0YnAT4HxwL+Bk1R1cXjcCOAG4GD87FZ/B85T1VUichPwffz8O3FgkapODltc9mkLXMJyrgEmq+rh4fMYcAFwMjAKeBs4R1VfEZELgEuBYqAReElVDxSRCcD/AnsDTcDDwE/bWnFEJADOBU4ApgAHAG+EZR0NDAFeAs5S1Y/65p3tUe4/dFPwehoOHUzrxfelXc6H/8zr2wpl4p0bYfvxuTu/GYyyEmGUnLuyw/W+5X9GDPhIJuejikRkEvAr4DBVrcL/U38sbZfjgP2AcUAF/h99m98Dw4DPANvjA5/7AFT1LOA54DJVrVTVyT2cfwJwKPB82upLgSOALwMjgLuAv4vIMFW9GvgeMCcs98Aw0PkrsASYAOyJD2Cu7XS674SvpxJ4DbgD2C7cfwwwG/iLiBRt4G3bJHV1dbZsy/223FHQ7T5dlt/tmtbWn9F28yMvdKyPLdtyPy33tVZch0chyHmLi4hshW/ROAl4XFXrw/UT8S0uu6vqy+G6M4HTVHUXEdkMWARMUtUPw+2TgfeAzVR18XpyXC4EGvCBWxXwIjBVVVeE3Ti1wFdV9Z9px70JXKWqvxORk4GLVHWbcNtewP8Bw1W1IVz3JeDPQLmqBmGLy0mqem+4fSSwHJigqgvCdRFgdXju9ECqr1mLi8m6TWpxOeQSeOqNjusiQGqTq9U7C26F8TX9dDJjgCy1uLhzV3W43gf/M3zARy85b3FR1TnAt4DTgU9F5HkROSRtl8Vpyw34QAN81xH44KbNx5229eRZVR2qqtVANb6L5l8iUoJvtakEZorImrYHPq9l8x7KGw8sawta0upSCqRf/ealLW8Z/nwj7RyrgKJe1N+Ywvb3i+H0g6G6DCbUwI3fgaYHYFhF9/uPru6b807ZHF66yoIWUzhcp0cByIvkXFV9BHhERIrx3TAzgF03cNjC8OdEoC0nZKtO2zb4/UxV60TkduAcYAfgVXyAdFBbS08vLARGiUi5qjam1aUZWJG2X3p95oc/t7WRScZ04hzc9n3/SLfqvtzUx5iBqgCTc3Pe4iIik0XkyyJSjk+iXYvvylhv0KGqnwJPAteJyFARGQZcBzzRlryLzznZZgPnL8fnnjQAH6tqANwIXCsi24b7VIrIl8Luqe68hA+erhOR8nC/y4C7VbXb1xHO+/IHYHo4sonwdXxNRCrXV2djjDFmsMp54IIfnXMxvktoDb7l42ja51lZn2/j52F5L3yswY9CanMDIGFXzNtp6/cXkXoRqcfnyXwO+Iqqrgm3X4xv9ZkhIrXAh/iWoG7fr3BY82H4rqQF+EBmNjBtA/U/HXgfmCUidcCbwNexHBRjjDF9wbmOjwKQ8+RckxP2oZus26TkXGMGn+wk505b0zE599qhAz56sSuIMcYYU7AGfJzSRT50FRljjDHG9IoFLsaYrLAuIWPygA2HNsYYY8yAUSDBSjoLXIwx/SSgIK+ixuS1wvubs64iY0zWtHcXxfksb1n3kTH9zbqKjDEmM8G0GDNnPhE+2yWndTFm8CmQaCWNBS7GGGNMoSq8uMUCF2NM9l333kSWNxczdWqua2KMGegscDHGZJW7qgUi20GZw10TJ/ivolxXyZjBowBbXCw51xiTNUEQQCTSfo8U5xhyhd0M3Zj+U3jZuRa4GGOy5pNVSVz6/dCCgNqWaO4qZMxgU3hxi3UVGWOyZ3gZBJ3uSOtSyRzVxphBqEDuCJ3OWlyMMVnTHO+6rjrezUpjjOkla3ExxmRNU0vQcYVzrK2ozE1ljBmMCq/BxVpcjDHZM+VXi7quLC7u/4oYYwqGBS4hEdlfRBK5rocxhaR21JgufexFra05qo0xg1HhZedaV1Evicg9wLeAlrTVL6jqIX1Q9izgaVW9fFPLMiafVLU0URer6rAu5iJsfcZ/+Pi2nXNUK2MGkcKIVTqwwCUzv1XV03JdCWNyZuQJsLKhV7u2Ak1X/r7L+paiGD97ZRZJdynrBkbHgJW/g+ryvqqpMQYscMkFETkauEJVJ4fPLwMuArZW1TkisgfwJDAC2A64DtgVaAR+D/y3qsbDY+8GDgKGAguBy1X1Dz2cV4BHgUtV9fYN1HEEcANwMP7X5O/Aeaq6akPbReQmYF/g8yLyE2BR22s1Jq8UHwPxVK93LwIS0a6XmFQkSnNJKR1mc0kAQ74NyYf8hHXGmD5SeJHLQLhC/APYRkS2CJ8fBHwU/mx7PgsYDjwLPAJsBnweHyj8NK2s54Gd8YHLpcA9IvKZzicUkcOBvwCnbyhoCf0eGAZ8BtgeGAnc15vtqnoW8BxwmapWWtBi8lYGQQv4y+XIhtpuNjiu+OJR3R/0zsLM62WM6Vnhpbjkf+CiqquBV4GDRKQamAL8Eh+UgA9cngZOBF5X1VtVtVVVFwFXhuvbyrpTVVeqalJV7wfeAPZPP5+InAPcBHxZVf/WqToniMiatMexIrIZ8CXgfFVdHdb3/9u78zC5qjKP499f0mwBwiarLEkElEXBeBBZVEBlIAiCGQczIksQcYkOzrAN+yITZHNEhieACgR8UBk2lwRJGOMDzKCcgQnGCBIgLIGYhSUhhEAn7/xxTsHtSld3ddO19vt5nnpSde+pe99zqjr3veecW/efgTEhhC17Wz9gDdUHS5cu9ef+vN/P++Leay9k76cfY5uXFnRZvvWri7t/w6jNm6KO/tyft8Lf12AlM+u9VIOFECYC2wE/B74BfBmYDYwAFgEfBiYAXwGWF94qYGiMcb0QwhDgPOBIYAvAgHVJw0XnhxD2I/XuLAQmxRjPLYvhBqCzfI5LHqp6EFgjxtiZl3UAbwF75hgqro8x/rEBk3Ob/0N3zccMhoytuvgK4Kp9xzBn06249mOfZlUeAtp42RJ+MfkK9n9qdtczp7tOg8P2HNCQnWshNekP0bnLu/x/b+ev0/L9Lk0/xyWbThpueQmYFmNcEEKYB5wELI4x/iWE8Azp4H9IhW2MIyU2BwKzY4yrQgiRrl+WVcAngOkhhBUxxn+rIrZS3/YI0hAWwKjCOvWyvrRf55qbBHZ71cXXXLmSs7+7lOVlPzj36prr8A+fP4HFl2430BE658q1fJqyuqYfKsruB4aTelqm5WX3AqeQkhqAyaQ5teNDCGuHEIaEEEaFEA7K64eTpgAuBIaEEMYDu5XvKMb4OGmy7PG5p6dHMcYXSJODLw8hbBhC2Ig0QXhqjPHF3tbnzcwHtu9TizjX5DR0KMvXXCv11BQMMfOkxbl6kbo+2kBLJC4xxhWk5OUN0rwUSAnL8PwvMcb5wP7A4cBc4GXSVUGl3o0bgT+Qej3mkSbK3ldhf3NJycthIYSrQwi9fdpHAUuBx/LjFQpza6pY/31S0vVKCOHPvezLudaxatVq/1m+1dEqHb3OuWbUEnNc3IDzD93VxTMvdjLiZuuavCx/HTt7eOOCcq451WaOywVvdJ3jcs7aLd/t4qc+zrmaUXmfrhm8saLbss65Wmj5PGU1nrg452pm+DrddO51DF19mXOuNtovb2mNOS7Ouda0cmXZgjaZHOicaxxPXJxzNbPGWt0kKp1+E3bnXP954uKcq5nhwzq6Xg5txhvnbdy4gJwbbPwn/51zrm+WTBgCb66AzreYephYaw2fWuec6z//H8Q5V1PrD+vglx+cCsBB7z+0wdE4N8i04bwyT1ycc865dtV+eYsPFTnnnHOudXji4pxzzrmW4UNFzjnnXLtqw6EiT1ycc865ttV+mYsnLs4551y7ar+8xee4OOecc651eI+Lc8451668x8U555xzrnE8cXHOOedcy/DExTnnnGtXVdxkUdJcSbvWMap3xRMX55xzzrUMT1ycc865diV1fVT9Nh0t6U+SHpV0h6TN8vL/kbRHfn61pD/n5x2SFklatyb1KPDExTnnnGtXVQwVrfaWNGx0MXCgmX0ImAX8MK++F/hUfr4vsFzSlsAewF/MbNmAxV6BXw49CEn6LfCeeu6zo6PjPZ2dnYvquc/+8lhrw2OtDY+1NhoQ691mdtBAb9RO7ujPBdH7A1PM7MX8+hpgZn7+X8AZkn4KLAZ+T0pkRpKSmprzxGUQqsUfR29CCDHGGOq93/7wWGvDY60Nj7U2WinWGhBgZctKrx8ARgOHkBKV3wPjSYnLOfUIzoeKnHPOOVd0LzBG0hb59QnAdAAzWwE8DJyelz0I7AN8KD+vOe9xcc4559x0SZ2F12cA0yQZ8BRwYmHdvaQ5LdHMOiXNAZ42szfrEagnLq5erm10AH3gsdaGx1obHmtttFKs74qZjaiw6sYK5ScCEwuvx9QgrIpkVj6M5ZxzzjnXnHyOi3POOedahg8VuZoIIRwFnArsDJwUY7yqQrn9gCnAX/OiFTHGPesS5DsxVBVrLnsCcBpp1v1U4NsxxlV1CTTtfxhwPfARoBM4Ocb4627K7UcD2jWEsCOpe3kT0qWSR8cYnygrMxS4EjiIdKXCxTHGH9U6tn7Geh7wDeCFvOiBGOM36xlnjuMyYCwwAvhgjHFWN2WapV2rifU8GtyuIYRNgJuA9wErgDnAiTHGhWXlmqJd3Ts8cXG18n/AF0kzz3szu8GXHVYVawhhJHAu8GHSgW4qcBQwudYBFpwMLI0xbh9C2AG4L4SwfYzxtW7KNqJdJwH/EWO8OSeE1wAHlJX5ErA9sAMpaXgkhDA9xji3rpFWFyvA5BjjyfUNbTV3Aj8A7uuhTLO0azWxQuPb1YBLYowzAEIIl5J+dO34snLN0q4u86EiVxMxxlkxxtlA3Xoj+qsPsf49cGeMcWHuZbkOOLLmAXZ1JOmAS+4diMDBdY6hWyGEzUi/73BLXnQLMDqEsGlZ0SOB62KMq/LZ7Z3AF+oXaZ9ibQoxxvtjjM/1Uqzh7QpVx9pwMcaXSklL9iCwXTdFm6Jd3Ts8cXHNYMcQwsMhhD+EEI5pdDA92BZ4pvD6WWCbJo6h3u26DTAvxrgSIP/7QjfxNUM7VhsrwBdDCI+GEO4JIexVzyD7qBnatS+apl1DCEOArwO/7GZ1q7Vr2/OhItcvIYSHSX/Q3dm8dECowsPANjHGV/NQzPQQwrwY4/QBCZQBjbXmeou1D5uqebsOEpOAi2KMb4UQPgPcFULYKca4uNGBtbhma9cfAq8BFee3uebhiYvrlxjj6AHazpLC86dDCHeSfoVxwA6wAxUr6Uyr2JW8LTCgXeK9xRpCKMVQmkC4LfC7brZT83btxnPAe0MIQ2OMK/Okxq1YvY1KdXgovy4/o62HqmKNMc4vPJ8WQngO2JX0M+fNphnatSrN1K55MvEOwKEVJtq3TLsOFj5U5BoqhLBlCEH5+cbAgaTJss3oNuDwEMKmuWv5BOAXdY7hVvIvWObJuXsAd5cXakS7xhgX5H2My4vGAY+UX6VBqsMJIYQheU7J4aS2rZtqYw0hvLfwfHfSlTKP1ynMvmp4u1arWdo1hHAR6Qq9w2OMKyoUa5l2HSz8B+hcTYQQxgGXAhsBbwLLgANjjLNDCBcAL8QYJ4UQJpDGlt8i9QBOjjFe0oyx5rInki6dBrgHmFDPoaYQwrrADaQrm1YCp8YY78rrGt6uIYQPkC4x3gh4mXSJ8eMhhCnAOTHGmHs3riIlUwDfizHW/VdKq4z1RtKBbSXpu3FujHFKA2K9Evg8sAWwCFgcY9ylSdu1mlgb3q4hhF2AWaSfDFieFz8dYzyiGdvVvcMTF+ecc861DB8qcs4551zL8MTFOeeccy3DExfnnHPOtQxPXJxzzjnXMjxxcc4551zL8MTFuRYiaYQkk7R1jffzNUk3FV5PlXRqT+9xtSFpjqRjqyxbl+9HPUhaS9ITkj7Q6Fhcc/HExbUlSaMk3SppvqTXJD0n6Q5Ja+b1x0qa0837Ki0/Kh8Qzulm3QxJK/J+XpX0iKSxtalZ7UlaF7gAOK+0zMwONrO6/r5OX+TPZt9GxzEY1KKtJe0nqbO4zMxWAJeRfmPJubd54uLa1RTgReD9wPrAXsBvAfVze18FXgK+ImloN+svNLP1SLe9vwX4uaQd+7mvRjsK+JOZPdnoQNygdwtwgKTtGx2Iax6euLi2I2kTUsIyycxeteR5M5uUz+L6ur2dgI8DxwBbAgdXKmtmncDVwFDgg91sa4KkR8qWjZS0UtKI/Pr63EO0VNJsSf/YQ2znSZpetmyGpLMKr3eV9FtJiyQ9K2mipDV6qPLhwLRK2ywMRxyT41smaYqkjSRdLGlB7un6ZuH9x+Yhj9MkvZjLXF6Mo7d6S/qQpLslLZT0kqRpefnMXOSe3Ov1owptNUzSD/I+Fkm6U9K2hfUzcky35RielPS5So1UqNN3JD2f33OZpE3yNpZIeqzYOyGpQ9I5kp7KdbhX0q6F9WtIuqLQhqd1s9+PS7o/v/9JSf8iqeqEXNJYSTNz7+BMSUeU16ms/A2lNq3U1pLm5nrdn5dHSXt0t43CsrlKPZlbAVOBofm9r0k6BsDMlpDuEXRYtfVz7c8TF9d2zGwx8GfgR5KOlrRzX/5j78aJpB6IX5N6ckCUVKQAAAasSURBVL5aqaDSUNQ3ST+1P7ObIj8FdpK0e2HZscAMM5ubX98P7A5sSBqyuUHSzv0JXNJmpBvX3U66ieBewGeAf+3hbaOB2VVsfiywL+mmcyOAPwBP5v0cB/x7MTEg3ahuW2BUjuNQ4OTC+or1lrRlrsfv8762AL4HYGa75fcfaGbrmdlXKsT7feBj+bEd6efof6WuPWjHAFcAG5B+5v1GScN6aIPtcryjclt8i3QQLt1C4nbg+kL5U4CjgTGkJPg+YJqk4Xn96cBngb2Bkbmub9/YU9IupO/gpcCmwCHABODLPcT4Nkl7kb6Dp5N6B88AbpG0ZzXv76Wtvwb8E7Ax8J/AlEK9etrmC6STgZV5m+uZ2Y2FIn8ifSedAzxxce1rP2AGcBLpZnp/k3R2WQIzUtIrxQept+RtktYmHRR+khf9GBij1Sc/npnf/zzwOWCsma02V8bMXgbuIh3YyfEcU9g+ZvZjM1tsZivN7GfAo7k+/XE0MNPMrjGzN81sHjAxL69kI2BJD+tLLjSzl3Ki+GvgLTO7zsw6zWwq6f4/Hy6UXwWcYmbL8zDUJeR2gF7r/WVgjplNNLNluS5V3+la0hBSnc8ys3lmtoz03dgJ+Gih6M/N7AEzWwVcS0pgduhh08uB83M8M0nJ6kNm9qCZrQRuBraXtEEufxzwPTN7LPf+XUC6X88hef3Ref0cM1tOSuyK92X5OnCrmd2V2+kxUoLV0+dZdBxwm5lNzZ/Tb4A7gPFVvr8nPzaz/zWzN0lJ5XJSEvZuLSElQ84Bnri4NmVmi8zsDDMbTTojPhU4h8KBEnjazDYsPoBvlG3qC8B6pAMQpLPdBUD5Wf1FeRubmdneZvarHsK7HvhS7p05IMd3O6QDrKQLJD2eu/JfAXYjnV33x0hgn7Lk7CekHotKXgZ6PVMmzSEqeb3sdWnZ+oXXC8zs9cLrucDWUFW9R5BuhtdfmwJrA0+VFpjZa6TPcptCuRcL65flp8U6lFuQk5yS8nYo1be0jW3KYlhFaodSDFvn18UYFhS2NxIYV/Z5nkvqvalGl/1nT9K1DfprbumJpZvgPUv+fN+l4aT5Zc4Bnri4QcDMXjezG0hn8Lv3UrzciaT5KrMkzSf1qGwMHK/uJ+lW4x7gDdLZ6LHAz/LZNcA4UlI0FtgoJ1MzqTyp+DVg3bJlWxWePwNML0vQNsgTiSt5BOjX0FQvNisbdhlBak/ovd5z6bnno7e7xS4EVpAO/ABIWg/YDHiuuvAHxHNlMQwhtUMphnn5dWn9uqQYS54BflL2eQ43s136s/9sVGH/vX2foHJbF+MWaViw9Pl22a6kDrrWq5j8lduV9J10DvDExbUhpUmiE5Umpa6RJ0SOJf0HeF8ftrMzsA9wBCnhKT0+SuqxGNOf+PJZ9mTg28DnKQwTkc4uO0kH2iGSxpN6HiqJwGhJH8n1nEDXA9NkIEgaL2nt3LMxStJBPWzzTuDTfa9Zr4YAF0taR9Io0jBIaS5Db/W+GXi/0uTeYflz/VRh/Xx6SGwKbX6hpK1yAnU58BjwxwGqXzVuAE6VtGPucTsT6AB+k9ffBJwi6X2S1iENpxWT1quBL0o6tPDd3lnSJ/uw/7GS/k7SUEkHk76DpXk4j5ASzM/m78oRwCfKtlGprcdLGq004foUYFihXhH4lNJE9LWAi4DiBPH5pMm5XZIqSeuT/t5+WWX93CDgiYtrR2+SzuZuJ3UxLwTOAr5lZrf2YTsnAg+b2a/MbH7h8Shwa17fX9cDnyQNVxUPnDeSJrnOIZ1970wPyZaZzSAdgO8mDVFsDjxQWD8f2J90pdBc0jDQHaSz7EpuAnbLycVAeoZUp6dJdbybdGCGXuqdJ3DuR5pY/DzwN6B4xc2ZwAWSXpZ0TYX9f4d0AH2INIyxJXBYnotSL5eSLvG9h1SHA0gTXUtziiaSLtt/kNROz5LaDQAzm0XqqTuJ9HkvICUjVQ0lmtl/k+ZUXUb6LlwCHGVmD+b1T5Im2F5L+ts5CLitbDOV2vpa4Mq83SOBQ8zs1bzup6Tk42HS0NSzpM+5FNdfSUnZH/MQWGmy8Tjgd2b2RDX1c4OD0lCkc869Q9LXgH3MrKqrVarY3rGkibH+exxtSNJc0ud7c29l+7DNtYBZpOTyLwO1Xdf6OhodgHOu+ZjZJGBSo+Nwg1e+6qqneU1ukPKhIuecc861DB8qcs4551zL8B4X55xzzrUMT1ycc8451zI8cXHOOedcy/DExTnnnHMtwxMX55xzzrUMT1ycc8451zL+Hzo7UtTWmagzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x396 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "explainer = shap.TreeExplainer(model)\n",
    "shap_values = explainer.shap_values(df_train[features])\n",
    "\n",
    "shap.summary_plot(shap_values[1], df_train[features], plot_type=\"dot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a nice looking chart, isn't it? So how do we interpret it? In the end, if we just want to understand whether a feature has a positive or negative impact, the idea is pretty simple: For each feature you need to only focus whether the red and blue dots appear on the left side of the y-axis or the right side of the y-axis. If the red dots are on the right and the blue dots on the left, this means the higher the value of the feature the more likely to score. And vice versa, if the red dots are on the left and the blue dots on the right, the higher the value of the feature the less likely to score.\n",
    "\n",
    "So, let's look at the *distToGoalLine* feature for example. Red dots are on the left and blue dots on the right. This means the bigger the distance to the goal line the less likely a player is to score. That makes perfect sense, doesn't it? Overall, if you go through the list, all of the features make a lot of sense, don't they? And also when comparing the feature influences according to the SHAP values to the influences in the last notebook, you will see that they are completely aligned :-)\n",
    "\n",
    "So that was easy, wasn't it? \n",
    "\n",
    "Again, this is only thought of as an example to show you the existence of SHAP values and how to use them. I highly urge you to read more about them before using them in any real-world project!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are the most important features?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, we can again use the SHAP values to answer this question. This can be done by looking at the mean absolute SHAP value for each feature, which can be observed in the chart below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAFbCAYAAADP32RWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdebxVVd3H8c9iUumCmkAOjKZQYmr5K7XUcCJNr5na5Cw5VBqpD5U54vA4opaZT86KOZSlghpOFaZW2i9NSzMTuIAIAspFQEWB/fyx1onN8Z57z71c7r2b+32/Xud1ztlr77XX2sM5v7PW2meHLMsQERERKaIu7V0AERERkZZSICMiIiKFpUBGRERECkuBjIiIiBSWAhkREREpLAUyIiIiUljd2rsA0nz33XdfVltb297FEBERaSuhUoJaZERERKSwFMiIiIhIYSmQERERkcJSICMiIiKFpUBGRERECkuBjIiIiBSWAhkREREpLAUyIiIiUlgKZERERKSwFMiIiIhIYSmQERERkcJSICMiIiKFpUBGRERECkuBjIiIiBSWAhkREREpLAUyIiIiUlghy7L2LoM0Uxi3TDtNREQ6pGxMtzWRbaiUoBYZERERKSwFMiIiIlJYCmRERESksBTIiIiISGG1aESOmfUHZgJDgEOAndy9tjUL1lGYWQbs4u5PNDHfQOBFYKi7v9YmhRMREenkVntosbtfUO285UGBmS3OJa+Tnpfm8q5pIr+vACcCnwRWAK8DjwI/dvf/VFuuapnZCOBRd//AdnP3GUCj5RUREZHWtUaukapWPlAxs+uBbu5+VDXLmtnZwEnAGOBgd59nZpsABwN7A60eyIiIiEjHUlUgY2YbA9cCnye2elySSxsL7Ozue6b3o4GTgT7AW8At7n6amT2XFnnYzFYAd7r7MY2scxvgx8TWlgXAjcCF7r7czAYDZwJHufsvSsu4+2zgp2X5DAKuBD4HvAP8BviRu7+T0i8Avg70S3X7qbv/uJrtUraewcA0YIC7v5q2yy7AU0Cpnv/n7mfnltkauAzYHngbuA04y93fb+76RUREOqNqB/veBiwHBgK7Akc1NJOZDQUuAvZz917AcGAigLtvm2Yb6e41TQQx6wOPAH8ANgb2BUYBp5TyIP45zq8aK7SZdQMeAOYAg4AdiQHNuNxsLwI7A72AY4ELzewLjeXbDLsCM4BNgVrgNDP7XCpbP+Ax4O6UvhOwF/CjVlq3iIjIWq/JFhkz2wzYHdjC3RcCC83sHODhBmZfRgwwhpvZdHevB/7SgnLtC7wHnO/uGfAvM7uYGMhcCvQF5rn7e7lyXgIcB3QF/uzuI4HPAFsCO7j7EmCJmZ0B3GtmJ7p7lm/RAX5vZg8AewAPtaDc5V5295+n10+Z2d8BA54EjgCec/drUvosM7sQuBg4txXWLSIistarpmupf3qenps2raEZ3X2qmR0KfBu43syeB85194aCnsYMAOpSEFMyJU0HmA/0NbMepWDG3X8A/CAFKnvm8pmbgph8PusSg6G5qSvs2FTPAKwH3N7M8lYyu+z9EmLLD8Qrvj5nZvW59EAMxERERKQK1XQtzUrPg3LThlSa2d3vdve9iGNkfgVMMLOeKbnaewTNBAaZWf7eCpun6RBbgzLgq1Xk0y+3/lI+7wLzUzfPxcDxQB933wC4j0bu6dCKphOvgNog91i/qSu1REREZKUmW2TSwNXJwCVmdjSxxeLMhuY1s2HEIOePxIG1C4kBx4o0yxxiV0+j/8lCHNfyY+KYkktTnj8Erkllmpa6YX5qZusA97r7G2bWF9g6l8/TwCvAZWb2P8AGwHnATe6+wsx6E8f+zAMyM9sX2Ae4q7HCmdm6ZZOWNVGfhowH/sfMRhFbgN4DBhP/h+bBFuQnIiLS6VQ72PcQ4v+8zAQeJ34JN6QHcDaxS6UeGA0c5O7vpvTTgXPNbIGZXVMhD9JYnJHELqLXieNVxgOX5+Y5k9iFdRRQZ2YLU9neJI6Vwd2XAfsRu41mEAObp4iXbJPyvTVNn0+8dPueJrZFV2KQln9c1cQyDdVxDrAbcABQR7wy6x5ii5GIiIhUIWRZtb090lGEccu000REpEPKxqyRv6irOORD91oSERGRwlIgIyIiIoWlQEZEREQKS4GMiIiIFFa73jRSWmbisEnU1ta2dzFERETanVpkREREpLAUyIiIiEhhKZARERGRwlIgIyIiIoWlQEZEREQKS4GMiIiIFJYCGRERESks3TSygHTTSJHiWEM30BPpbHTTSBEREVn7KJARERGRwlIgIyIiIoWlQEZEREQKq81GoZlZf2AmMAQ4BNjJ3XXnQxEREWmxdhlO7+4XVDuvmWXALu7+RHq/OJe8Tnpemsu7pon8vgqMBrYF3gemA7cBV7r7e9WWq5H8xwI7u/ueq5uXiIiINK5w1wXmAxUzux7o5u5HVbOsmZ0NnJwe97r7AjMbDvwQ2IQY1LQ7M+vu7u+3dzlEREQ6ujUWyJjZxsC1wOeB14FLcmljybVamNloYnDRB3gLuMXdTzOz59IiD5vZCuBOdz+mkXVuA/wY+CSwALgRuNDdl5vZYOBMYJS7jy8t4+4vAEfk8tgolXUksC7wB+C77v56Sq9L9doD2AGoA45z9z+Z2deA04AuuZajbdx9qpntAlwIbJXKdjVwubtnZjYCeBQ4GjgH6Av0amITi4iIdHprcrDvbcByYCCwK3BUQzOZ2VDgImA/d+8FDAcmArj7tmm2ke5e00QQsz7wCDHw2BjYFxgFnFLKg/iHOnc2kkcA7gUyYGtgELAIuL1s1lHE7qnSOm9J5f0lcAEwOZW3JgUxw4HfApcSg5R9gROBw3N5dgX2IQZhH6lURhEREVlpjQQyZrYZsDswxt0XuvscYktDQ5YRA4zhZlbj7vXu/pcWrHZf4D3gfHdf6u7/Ai4GSsFPX2B+E+Ngtk+PE1K53wZ+AOyeBiuXXOPuL7j7cuB6YIsUSFXybeAud5/g7svd/SXgKnItQcmpufWKiIhIE9ZU11LpSz8/5mRaQzOmFotDiV/215vZ88C57v5wM9c5AKhz9/zf909J0wHmAX3MrEcjwcwQ4gDi180sP/1dYsvSq+n97FzakvTcC1jYSL67m9mBuWldiFdxlawoey8iIiJNWFOBzKz0PIgYTED8Mm+Qu98N3G1mPYBvARPMbKPUMlHtfYVmAoPMLOSCmc1ZGRw8nPL6GnBrhTymEwOTD7v7iirXW66h5aYDN7r7CY0sl5UFYSIiItKENRLIuPurZjYZuMTMjgbWIw60/QAzG0YMcv4IvENs1chYGRDMAbYEnmhitQ8QB/qeZmaXpjx/CFyTylRnZucBPzGzLsAEd683s4+l+cYCDvw9zTPW3d8ws77AHu5ecWxNmTnAwLKWn6uBx8zsQeDBVL+hQF93f6zKfEVERKTMmhzsewixm2Ym8DgwvsJ8PYCzid019cRBtAe5+7sp/XTgXDNbYGbXVFqZuy8kDujdk3iV1ENpnZfn5jmH2IV1HPCqmb0J3AH8E5idWmEOIG6Xv5nZIuApYEQz6n1XqvMcM6s3syHu/k9gP+CkVM+5wM3EcTsiIiLSQiHL1JtRNGHcMu00kYLIxhTu77pEOqJQKUH3WhIREZHCUiAjIiIihaVARkRERApLnbcFNHHYJGprdeNwERERtciIiIhIYSmQERERkcJSICMiIiKFpUBGRERECkuBjIiIiBSWAhkREREpLAUyIiIiUli611IB6V5La47uiyMi0iHpXksiIiKy9lEgIyIiIoWlQEZEREQKS4GMiIiIFFaHCWTMrM7MDmvD9b1iZkc1kp6Z2c6tvM5lZjaiNfMUERHpzDpMINMRmNnhZjapvcshIiIi1VEgs6oDgHvbuxAiIiJSnY72pxkDzex3wA5AHXCcu/8JwMyOBb4HDACmAj9094dT2rbAlcBwoCvwF+BEd5+S0rsDFwOHASuAK8pXbGbrAnsBJ6w62a4CPgo4cKy7v5ISegIXAgcC6wFPAKPdfUZK7wVcBdQCi4CzcpluCLwGfNbdn81N/yPwiLuf14JtJyIi0ul0tBaZUcBoYH3gEeAWADM7DvghcCiwIXA6cLeZbZGWy4CxwGbAYGAx8ItcvqcC+wGfBYakeQaVrXsk8E93n5ObdhxwMNAPeAGYaGZdU9oVwI7pMQiYD9yXS/8xsCWwFbAN8CVikIW7LwDuAo4prcjMhgI7ATc2uZVEREQE6HiBzDXu/oK7LweuB7Yws/WJwc257v6cu69w998CfwC+DuDuz7v7H9x9qbsvBM4BdjSzD6V8jwAudvdX3P0dYAwx+MlrqFvpstwyPyC2zOxgZl1Snme4+yx3XwKcBHwc+ExKPxQ4093npDL9sCzva4FDUksQwDeBB919Vss2nYiISOfT0bqWZudeL0nPvYitKD8zsytz6d2AVwHM7KPApcQuqV6sDFL6pHz6E7uqAHD3JWY2t/Q+taLsR+wqyssv87aZzUt59QXWJXZxldIXpzxLXV/r5JcHpuUzdvcnzGwWcLCZ3QkcSWwBEhERkSp1tECmkunA2e5+V4X0nxPHnGzj7m+Y2dbAP1h5b4ZZxO4kAFJLTb/c8jsD89z9P2X55pfpSQxgXgXmAUuJAVZpHE5NynNmSn8vLT8lZTGkgXJfS2yJWQwsBx6oUD8RERFpQFECmSuAsWb2H+A5YmvI9sB8d38J6A38B6g3sz7AuWXL3wp838wmEwOeS1j1BlQHAPc0sN6T0zKzgIuILS1PufsKMxsPnGdmLwL1wGXAS8DTKf124Bwz+yfwDh9s7QEYn6afDdyUutRERESkSh1tjEyD3P06YvBxE7AAmAGcCXRPs5wM7AK8BTwO3F+WxYXAQ8Srmaal5afn0itddn09cDexhWVb4Eu5YONk4pVMf035bQLsn0v/XlrXS8TWofuIrS75etUDv05539DkhhAREZFVhCwrH/PauZjZdsB97j6gndY/lngZ9shqlwnjlnXunbYGZWOK0kgpItKphEoJ+tSGHsTWlTZnZh8BjkWDfEVERFqk0wcy7v408HRbr9fMLgeOB251dw3yFRERaYFO37VUROpaWnPUtSQi0iGpa2ltMnHYJGpra9u7GCIiIu2uEFctiYiIiDREgYyIiIgUlgIZERERKSwFMiIiIlJYCmRERESksBTIiIiISGEpkBEREZHC0h/iFVBb/CGe/hhOREQ6kIp/iKcWGRERESksBTIiIiJSWApkREREpLAUyIiIiEhhKZBpR2Z2lJm90t7lEBERKarCXJpiZoOBacAAd3+1lfIcmPJ8zN13L0ubDDzq7ufnpo0Fdnb3PVtj/SIiIrJ6Ol2LjJl1z709BqgHdjOzoe1UJBEREWmhZrfImFkNMBY4EOgLzACOB54BLkzT1wOeAEa7+4y03GQ+2MKRAbu4+xOptWMX4CligAHwf+5+dnr9XHr+d1ruYnc/z8w2Ai4BRgLrAn8Avuvur6d11AE3ArsBnwG+CdxpZl2BUanMRwLHAWPSMlelsuxkZqcCs4CzgNOALma2OJVlG+A94Hpge6AH8Dxwkrv/LVfPA9OyWwLvADe4++kNbNu9gZuAY939/kr7QERERKKWtMjcAOwA7AH0Bg4A5gBXADumxyBgPnBfChiqtSsxMNoUqAVOM7PPpbRt0/Mwd69JQUwA7gUyYOu03kXA7WX5HgucAtQAE9K0WuAjwK3EQOdIM1sHwN1PBB4HzkvrGubuvwQuACanaTXuPpW4Da9O696YGNDdXWr5MbN9gFuIwd9GwFBgUnnFzex44DpgPwUxIiIi1WlWi4yZ9QO+Cmzt7tPS5P+YWRfgCGB/d5+V5j0JeJPYCvLnKlfxsrv/PL1+ysz+DhjwZIX5t0+PPd19aVrvD4D5ZtY/N5bmOnd/Nr1+Jz0fBzzg7q+b2a3ARcTWpDuqLCsAqcVpRum9mZ0BjCa2vrwIfBf4eS44eYvYWlUSzOxiYD/i+JvpzVm/iIhIZ9bcrqXB6fnlsul9id06U0sT3H2xmc0FBlB9IDO77P0SoFcj8w8B1gFeN7P89HeBgUApkKnLJ5rZIOALwJdTWeeb2URiF1mzAhkz6wNcDowANgBWpKS+6XkwcE8jWfQDTgROVBAjIiLSPM0NZOrSc6m1oWQesJQYWEyB/46l6QfMTPMsBj5UWsDMNm3mulc0MG06Mdj5sLs3lF5p2WOJXULXm1kprSfQy8yGufu/K6yvoWkXApsAO7j7bDPrRWx1Kd0Xoo64vSp5nThGZ4KZLXP3WxuZV0RERHKaNUbG3ecCvwauNrPBZhbMbAtgc2A8cJ6ZbWpmPYHLgJeAp0uLA18ys77py/5/m1nWecRAIh8UOPB34Cdp0C8p/69XysTMugFHE7uStgG2S4+hqbzHpVnnAFuULT4HGGhmPXLTegNvAwtS8HZx2TI/A75lZvuYWTcz650b9xMr4f4kcbDyODP7TiPbQERERHJaMth3FDF4eIw4sHYCcZDrycTA4q/EMSObEMfMLE/LXUEMFKak5R9ozkrd/R3gTOAOM6s3s9NTK8wBqR5/M7NFxKueRjSSVS3wYeAKd5+Tf6Qylgb9XgFYWtcLadm7iC1Mc9L0IcDZxJanN4hXLP0JKNUZd3+AeBXWBcQxQ/8G9m6gfs8Qr6w6PV0pJSIiIk0IWZa1dxmkmcK4ZWt8p2VjCvNfiSIisvYLlRI63R/iiYiIyNpDgYyIiIgUlgIZERERKSwNhCigicMmUVtb297FEBERaXdqkREREZHCUiAjIiIihaVARkRERApLgYyIiIgUlgIZERERKSwFMiIiIlJYCmRERESksHSvpQJqzXst6Z5KIiJSALrXkoiIiKx9FMiIiIhIYSmQERERkcJSICMiIiKFpZGeq8HM+gMzgSHuXreaeQ0GpgED3P3V1S+diIjI2k8tMiIiIlJYCmRERESksNS11AxmtjFwLfB54HXgklzazcAydz8mN60OOMPdf5He7wJcCGwFLACuBi53d/2Zj4iISAsokGme24C3gIHAesCvq13QzIYDvwUOA+4HtgQmAfOA8a1eUhERkU5AgUyVzGwzYHdgC3dfCCw0s3OAh6vM4tvAXe4+Ib1/ycyuAo5AgYyIiEiLKJCpXv/0PD03bVozlh8C7G5mB+amdSFe9SQiIiItoECmerPS8yBgSno9JJe+GNio9MbMugH9cunTgRvd/YQ1WUgREZHORIFMldz9VTObDFxiZkcTx8icmZ8lpQ0BXgPOBbrn0q8GHjOzB4EHgQwYCvR198faoAoiIiJrHV1+3TyHAOsQu4MeZ9WxLbcBE4FniC02M1jZioO7/xPYDzgJmA3MBW4G+rZBuUVERNZKIct05W/RhHHLWm2nZWPUKCciIh1eqJSgFhkREREpLAUyIiIiUlgKZERERKSwNECigCYOm0RtbW17F0NERKTdqUVGRERECkuBjIiIiBSWAhkREREpLAUyIiIiUlgKZERERKSwFMiIiIhIYSmQERERkcLSvZYKqLXutaT7LImISEHoXksiIiKy9lEgIyIiIoWlQEZEREQKS4GMiIiIFJYCGRERESmsDn/ZipllwC7u/kQ7l2MwMA14G8hfNbSfu09ezbxHAI+6e4ffHyIiIh3JWvHFaWbd3f39NlrdMHd/tY3WJSIiIo3oMIGMmY0GTgb6AG8BtwD7puSHzWwFcKe7H2NmdcCNwG7AZ4BvmtlE4ELgQGA94AlgtLvPSPlPBv4GDAZGAnOBU9x9QkoPwI+A7wA90/q3AR5397FVlP8g4KyUfx0w1t3vaSrdzDYFJgFdzWxxmv0Ed7+lmu0mIiLSmXWIMTJmNhS4iNhN0wsYDkx0923TLCPdvcbdj8ktdixwClADTACuAHZMj0HAfOA+M+uaW+ZI4HJgfeAq4BYz65nSDge+B9QCHwFmA7tWWf6dgNuAU4GNgNOAO8xsh6bS3f01YB9geapjjYIYERGR6nSUFpllxH/tG25m0929HvhLE8tc5+7PApjZUuAIYH93n5WmnQS8SWyx+XNa5pfu/mRKv5YY1GwJPJeWvyaX56XACQ2s94U0bgdgqrt/Cjga+I27T0rTHzCze4BRwFNVpIuIiEgLdIgWGXefChxKbGV5zcyeMLORTSxWl3vdF1gXmJrLczGx+2hAbr7ZufQl6WWv9LwZMD2XngEzG1jvcHffID0+laYNyK87mZJbd1PpIiIi0gIdIpABcPe73X0v4hiZXwETUrdPpfsKrci9ngcsBYaUJphZDdCPhoORhswidkmVlg9UH2jMzK872Ty37qbSVyAiIiLN1iG6lsxsGPGL/o/AO8BCYgCzAphD7P6pePm1u68ws/HAeWb2IlAPXAa8BDxdZTFuBS42s98ALwKjgU2rXPZm4HdmdivwKHEw8YHAiCrT5xAH+w5x92lVrlNERKTT6ygtMj2As4ldP/XEIOIgd38XOB0418wWmNk1jeRxMuDAX4EZwCbEMTPLqyzDeOBnxCuIXgf6E8fpLG1qQXf/E3Eg8ThgAXAJcJi7/6XK9JeBq4GnzazezA6vsswiIiKdWsiySj03nZuZdSEGRD9w99vbuzx5YdyyVtlp2ZgO0SAnIiLSlFApQd9kOWb2NeKl3F2I/ynzIWILjYiIiHRAHaVrqaP4LrFbaTawO/BFd1/QvkUSERGRStQik+PuO7d3GURERKR6CmQKaOKwSdTW1rZ3MURERNqdupZERESksBTIiIiISGEpkBEREZHCUiAjIiIihaVARkRERApLgYyIiIgUlgIZERERKSwFMiIiIlJYumlkAa3OTSN1o0gRESmgijeNVIuMiIiIFJYCGRERESksBTIiIiJSWApkREREpLA08rMBZpYBu7j7E7lpk4GdgPeB5cBU4Hx3/027FFJERETUIpNnZt2bmOU8d68BNgLuAH5pZkPXfMlERESkIR3y8mszGw2cDPQB3gJuAa4FpgFHAT8EBgGPAYem96OAFcRg42cpn/7A9cD2QA/geeAkd/9bSh8L7Ao8AxyenjcFtgHeSfnd6e7HpBaZR939/LRsDbAIOBioBy4AhgLLgN8Bo919bpr368DZQH/gbWCSux9lZgE4Hzga6AW8AVzm7j9tbPvo8msREelkinP5dWrhuAjYz917AcOBiblZDgJ2BgYCg4GngCnEAORo4MdmNjDN2wW4mhj0bEwMVO4ua3nZFZgNDAAOcvdt0/SR7l7j7sc0UMYewAnEbqbngKXAiUBf4BOpLD9J8/YEbgVOSPXZHLghZbUXcCSwQ0rbAXiyGZtLRESkU+uIP8+XESOv4WY23d3rgb+Y2eCUfp67vwlgZvcD+7r7dSltkpktAD4JzHD3GcCMUsZmdgYwGtgSeDFNnuHul6XX7zVRttPNbEya7xVi4PNKel0yx8wuAW7MTXsf+JiZ/T2V/fHc+tZNdZ3n7q8DrzdRBhEREUk6XCDj7lPN7FDg28D1ZvY8cC7wcppldm72t8vel6b1AjCzPsDlwAhgA2JXEcSWk5K6ZhTvf0tdS3lmtj2xa2lboCcxEKtJ9XnbzL4InAL8r5lNJXYf3e7uk83sNOAM4Fdm9mfgdHf3ZpRJRESk0+pwXUsA7n63u+9FHCPzK2ACMUBorguBTYhdN72J3Uewal/big8sBc0dg3InsdtqaFrPN/KJ7j7Z3fcn1ud84Bdm9tGUdq2770zs+noOuLuZ6xYREem0OlyLjJkNA4YAfyQOuF1IDCwaCjia0pvYQrMgDc69uMrl5hC7n55oasbcehYCi9L4nFNLCWb2EeKYnkfdfaGZ1aek5Wb2aWAd4K/EcTaLiF1rIiIiUoWO2CLTg3iFz2zi1UCjiQN8321BXmcD/YhXAz0P/In4HzBNOR0418wWmNk1Vcx/HHAMMRC5G7grl9aFODC4zswWAT8DjnT3OmIX2JXA/FTGkcDXq1ifiIiI0EEvv5bG6fJrERHpZIpz+bWIiIhItRTIiIiISGEpkBEREZHCUiAjIiIihaWRnwU0cdgkamtr27sYIiIi7U4tMiIiIlJYCmRERESksBTIiIiISGEpkBEREZHCUiAjIiIihaVARkRERApLgYyIiIgUlm4aWUDV3jRSN4gUEZG1hG4aKSIiImsfBTIiIiJSWApkREREpLAUyIiIiEhhKZARERGRwlIgIyIiIoWlQGY1mVn3IuQpIiKyNuq0fzRiZjXAWOBAoC8wAzgeeAa4ME1fD3gCGO3uM9Jyk4G/A4OB3YELzOxjQFfgXeArwBLgXHe/Jre+XVK+WwELgKuBy909M7MRwKPA0cA5qTy91lTdRURE1haduUXmBmAHYA+gN3AAMAe4AtgxPQYB84H7zKxrbtlRwJXA+ukZ4GDgPuDDwHeBq8xsEICZDQd+C1xKDFL2BU4EDs/l2RXYB/gk8JHWraqIiMjaqVO2yJhZP+CrwNbuPi1N/o+ZdQGOAPZ391lp3pOAN4HPAH9O8/7a3X+fXr9tZgC/d/eJadrdZlYPbAdMB74N3OXuE1L6S2Z2VVrX+FzRTnX3ha1cXRERkbVWpwxkiN1CAC+XTe8LrAtMLU1w98VmNhcYwMpApq6BPGeXvV/Cyu6hIcDuZnZgLr0LMDP3fkXZexEREWlCZw1k6tLzlsCLuenzgKXEwGMK/HcsTT8+GHQ0x3TgRnc/oZF5MnfXja9ERESaoVMGMu4+18x+DVxtZkcRA42PpuTxwHlm9iJQD1wGvAQ8vRqrvBp4zMweBB4EMmAo0NfdH1uNfEVERDq1zjzYdxTx6qPHgEXABGBj4GTAgb8Sr2TahDhmZnlLV+Tu/wT2A04idkHNBW4mdmWJiIhIC4UsU29G0YRxy6raadmYTtngJiIia59QKaEzt8iIiIhIwSmQERERkcJSICMiIiKFpUEUBTRx2CRqa2vbuxgiIiLtTi0yIiIiUlgKZERERKSwFMiIiIhIYSmQERERkcJSICMiIiKFpUBGRERECkuBjIiIiBSW7rVUQLrXkoiIdDK615KIiIisfRTIiIiISGEpkBEREZHCUiAjIiIihdVhR4Oa2WBgGjDA3V9t5+J8gJn9FPgGsC6wubvPbeciiYiIdEw9p/AAABsGSURBVDodNpBpDWY2GXjU3c/PTRsLnAG8myYtAiYCJ7n7O1Xm+1lgFDDY3ee1ZplFRESkep21a2myu9e4ew1gwE7Amc1YfnNgdkuDGDMLZrZWB5EiIiJtoUN8mZrZaOBkoA/wFnALcG1K3s3MfgQMAP4MHOnus9NyGwFXAHsRrzF/CDjZ3d80s6uAXYCdzOxUYJa7Dytft7vPMrOHgK1z5ekG/AA4CugHvACMdve/mdkPgHOBHma2GHja3Xc3s0HAlcDngHeA3wA/KrXymFkGnAQcDgxP9Xo+5XUQsD7wNHCiu7+yWhtURESkk2j3FhkzGwpcBOzn7r2IX/ITc7N8DdgV2Az4EPGLv+Q2YENgK+DjxEDoVgB3PxF4HDgvtb58IIhJ6x8E7AM8kZt8LvAlYG9gI+BG4CEz29DdLwG+BUxN+e6eAp8HgDnAIGBHYkAzrmx130z1qQGeBa4HPpbm3xh4CrjfzLo3sdlERESEDhDIAMuIrSnDzazG3evd/S+59HPcfb67vwXcTuwKwsw2Bb4AnOLuC9x9AXAK8EUz26SJdX7ezOrN7C2gjjhO5saUbwC+C3zf3ae6+3J3vwGYDexbIb/PAFumsixx91nEcTijUn4l49x9irsvB3oRBwt/x91fd/f3gHOATYAdmtpoIiIi0gG6ltx9qpkdCnwbuD7X3fJymmV2bvYlxAAAYlcTxCubSqbk0vLLlXvM3fcEMLNewPnAk2a2DdCb2GJyX+oOKukO9K+Q3wBgrrsvKSvLukBfoHRFU10ufUh6ft7M8nl1z9VNREREGtHugQyAu98N3G1mPYjdNhOA7ZtYbGZ6HgyUxpRsXpa2oop1LzKz64DRxHEyzxADpj3d/a9VVmEm0M/Merr727myvAvMz82XL8/09LylrnwSERFpmXbvWjKzYWa2t5n1BN4HFgIZTQQh7v4a8DBwmZltYGYbApcBk0qDgYljVrZoYv09iWNXlgBT3D0DfgKMM7Mt0zw1ZvaF1J3VkKeJwdRlZtYzzXcecJO7N1iP9L8ztwNXm9lmaT0bmNmXzaymsTKLiIhI1O6BDNADOJvYFVRPbBk5iJX/89KYw4jjW15Kj3rgiFz6FYCl8TAv5KaPMLPF6aqjWcCngC+6e31KP5vYKjQhjaP5D7GlqMHt5e7LgP2IXU8ziIHNU8CYJsp/LPBvYLKZLQL+AXyFGMiJiIhIE0KW6TuzaMK4ZVXttGxMh+g5FBERWV2hUkJHaJERERERaREFMiIiIlJYCmRERESksDSIooAmDptEbW1texdDRESk3alFRkRERApLgYyIiIgUlgIZERERKSwFMiIiIlJYCmRERESksBTIiIiISGEpkBEREZHC0r2WCqixey3p/koiIrIW0r2WREREZO2jQEZEREQKS4GMiIiIFJYCGRERESksBTKJmY0ws2XtXQ4RERGpni5xqZKZ3QwcCizNTf6Tu49shbwnA4+6+/mrm5eIiEhnokCmeW5x92PauxAiIiISdfhAxswOAi5w92Hp/XnAGcBH3X2qme0APAxsBHwMuAzYHngbuA04y93fT8veBOwJbADMBM5399srrNeAe4Bz3f26Jsq4EXAFsBfxWveHgJPd/c2m0s3sKmAXYCczOxWYVaqriIiINK4IY2R+D2xhZgPT+z2BV9Jz6f1k4MPAY8DdwKbATsTA4Ue5vJ4AtiMGMucCN5vZVuUrNLP9gfuBY5sKYpLbgA2BrYCPA32AW6tJd/cTgceB89y9RkGMiIhI9Tp8i4y7LzCzZ4A9zezXwHBgNLAvcC0xkLkbOAJ4zt2vSYvOMrMLgYuJQQvufkMu6zvNbAwwAnixNNHMRgNjgL3d/e9lxTnczA7OvT+OGBx9ARjq7gtSHqcAL5nZJsQWmIrp7j675VtHRESkc+vwgUzyKDFgeQP4M/BbYJyZ1RBbXr4DnAh8zszqc8sFoCuAmXUBxgJfAzYGMuBDQN/c/F2A04GfNxDEANxaPkYmdW0BTMtNnpKeB7Dyb5UrpSuQERERaaEiBTK3AW8Cj7j7XDObBZwEvOHu/zKz6cQrf/atkMc3gGOAkcCL7r7CzJxV79+wAtgVeNTMlrr7BVWUbWZ6Hkzs8gLYPJcWmkgvrVdERESaqQhjZCB23/QGDgceSdN+B3yfGOQAjCeO0R1lZuuaWRcz29zM9k7pvYFlwDygi5mNArYtX5G7/5s4+PabqWuqUe7+GnGw8WVmtoGZbUgccDzJ3Wc3lZ6ymQNs0awtIiIiIsUIZNx9KTGYeRd4Pk1+lBicPJrmmQPsBhwA1AELiFcdlVo/bgGeIraKzCIOvH28wvrqiMHM/mZ2tZlVvOtmchiwCHgpPeqJY3aqTb+CGITVm9kLTaxLREREkpBlWXuXQZopjFtWcadlY4rSWygiIlK1ig0KhWiREREREWmIAhkREREpLAUyIiIiUlgaUFFAE4dNora2tr2LISIi0u7UIiMiIiKFpUBGRERECkuBjIiIiBSWAhkREREpLAUyIiIiUlgKZERERKSwFMiIiIhIYSmQERERkcJSICMiIiKFpUBGRERECkuBjIiIiBSWAhkREREpLAUyIiIiUlgKZERERKSwFMiIiIhIYSmQERERkcJSICMiIiKFFbIsa+8ySDOts846/3zvvffebe9yrEndunXrs2zZsvntXY41rTPUszPUETpHPTtDHaFz1LOAdZyfZdneDSV0a+uSyOr7xCc+8a67W3uXY00yM1/b6wido56doY7QOerZGeoInaOea1Md1bUkIiIihaVARkRERApLgUwxXdveBWgDnaGO0Dnq2RnqCJ2jnp2hjtA56rnW1FGDfUVERKSw1CIjIiIihaWrljooMxsK3AJsBLwBHOHu/ymbpytwJbA3kAEXufv1bV3W1VFlPUcCFwCfAH7q7mPavKCroco6ngl8HViWHqe5+0NtXdaWqrKORwMnAyuArsB17n5lW5d1dVRTz9y8w4BngauLdMxWuS/HAt8BXkuTnnT3E9qynKur2n1pZl8FzgQC8XN2T3d/vS3L2lJV7svxwDa5SdsAB7j7xDYr6GpSi0zH9XPgZ+4+FPgZcE0D8xwKbAFsCewEjDWzwW1WwtZRTT2nAscCl7ZlwVpRNXV8Gvi0u28LjAJ+aWbrtWEZV1c1dfwNsK27bwd8FvgfM9umgfk6smrqWfqRcQ1wbxuWrbVUVUdgvLtvlx6FCmKSJutpZgaMBfZy962BnYGFbVnI1dRkHd39iNJ+BI4EFgCF+REFCmQ6JDPrB3wKuCNNugP4lJn1LZv1a8RftSvcfR7xQ/MrbVfS1VNtPd39FXd/lthSUSjNqOND7v52evs88dffRm1W0NXQjDq+5e6lQXk9ge7EX7iF0IzzEuBU4H7g5TYqXqtoZh0Lqxn1PBkY5+5zANx9obsX4s9IW7gvvwnc5u5L13T5WpMCmY5pADDL3ZcDpOfX0vS8gcD03PsZDczTkVVbzyJrSR2PAKa4+6ttUL7WUHUdzWx/M3uBeNxe6u7/aNOSrp6q6plamb4AXNHmJVx9zTlev25mz5vZw2a2U1sWshVUW8+tgM3N7I9m9oyZnWFmoY3L2lLN+uwxsx7AIcCNbVbCVqJARqQDMbPPA+cB32jvsqwJ7j7R3YcDQ4HD0ziStYaZdQeuA75V+gJZS/0cGOLu2xC7fCeYWSFaEJupG3HMyF7A54F9gMPbtURrzgHADHf/e3sXpLkUyHRMM4HNUj97qb990zQ9bwYwKPd+YAPzdGTV1rPIqq5j+lX7C+JAu3+3aSlXT7P3o7vPII4L2q9NStg6qqnnJsBHgd+aWR1wEnCsmRXlPzuq2pfuPsfd30+vH0npW7dxWVdHtcfsdODX7r7U3RcBE4DPtGlJW6655+UoCtgaAwpkOiR3nwv8nZW/yr8BPJvGweTdRfyQ7JL6PQ8gDqgshGbUs7CqraOZfRr4JXCwuz/TtqVcPc2o48dyr/sAuwGF6Vqqpp7uPsPd+7j7YHcfDPyYOI7tuDYvcAs0Y19ulnu9HTAYKEzw3YzPntuBkWYWUmvbHsBzbVfSlmvO56uZ9Qd2Ida3cHT5dcf1LeAWMzuLOIr8CAAz+y1wlrs7cCuwA1C6nO5cd5/aHoVdDU3W08x2Bu4EegPBzL4OfLNAlydXsy+vBtYDrokXSgBweIHGkFRTx+PTpfTvEwczX+XuD7dXgVuomnoWXTV1vMDMtgeWA+8Rj9U57VXgFqqmnncCBrxI/NuAh4Ab2qe4LVLt8XokcJ+7v9k+xVw9+mdfERERKSx1LYmIiEhhKZARERGRwlIgIyIiIoWlQEZEREQKS4GMiIiIFJYCGWkTIYQvhBAez70fEUKoa8citZkQws0hhFa7K3kIYXAIIcu97xtCmB5C6FPFst8KIdzaWmUpghDCLiGE+vYuR2cUQjisOed5a58r0rg1dW60YL9fHEI4r6XrUyAja1wIIRDvO3N2E/N9O4TwzxDCWyGEBSEEDyF8LZdeF0I4rIHlPjA9RC+nvGrK0kaEELIQwuL0eC2EcFMI4cOrV9P2kWXZPOIfWTW1fT8EnEu8m2+nkWXZ41mWbdDe5agkhDA2hPBoe5ejM1hT2zqEMDmEcEZr57umlZ8b7XgsXgScEELYrMk5G6BARtrCSKAH8IdKM4QQvkH8Iv4msD7xr7RPJv6JU0vsBmxO/BOrhu5btDzLsposy2qAnYGdiP/CWlQ3AkeHEHo3Ms9hwD+yLJvSRmVaRQihawhBnzkisoosyxYAk4DjW7K8PlTWMql14owQwh9Sa8M/QgjbhBC+EUJ4JYSwMIRwfQihW26ZgSGEX4cQZqfHtSGEXrn0C0IIU1N+U0IIJ+XSBqfWjcNDCC+GEBaFEB4OIWySK9YBwKNZ4/+++Fngj1mWPZVF76RfCy3959fjgQeJ/37c6MmRZdlU4H7gk+VpIYRuaZt8qWz6LSGEG9PrPUIIT6VWpHkhhDtDCP0qrS9tr51z70eEEJaVrfO01KJUH0J4MoSwfRN1+A8wH9izkdkOAB4pK8v3Qggvpf02I4RwYQiha0obF0K4p2z+3dK8H0rvtw4hPBRCmJ9bvntKKx0b3wwhvAi8DfQLIXw9hPBcai2bHUK4ppRfWm7jEMJ96Vh9OS2fhRAG5+Y5NrXeLQwhPBtCGFmp0g1s35tDCLeGEG5M23dWOj+2CyH8NdXvDyGETXPL1IUQzgohPJHOAw8hfDqX3ugxEELonvbpv1P+U0IIB4XY4ngaMCKsbCHcvEI9Pp/WsTDts+NzaSNCCMtCCF9LeS8MIfwqfx43kF9LPiu2CSH8PtVzalq+ay79M2nbLA4hPEH8MZFfZ890XE0LIbwZQngwhLBFpTI2UOaNQgjj03EzJ8Tz8MO59FVaZ3PHYP9K2zqEcFSq7w9TvnNDCJc1cBz3z+V7VAjhlfT6KuLf+5+Z8mzwVg0htnb8LsRulHkhhDdCCKeEEAalbboohPC3EMLHc8us1rmSO9avyx3rHzhu0utGt09ZXVbpAmyl/f4I8TOq+bIs02MtegB1xFsWfBzoTrwJ4RTgWuBDxBtLzgUOSfOvC7xC7HJYD9gQ+C1wYy7Pw4gtJAHYHXgH+EJKGwxkxECgD/E2Ak8C1+WWfwoYXVbOEUBd7v1XgHeB84n3M9mgQt0Oa2o60BdYChwIbJfKt33Zupfl3m9BvE/MjRW26SXAvbn3NcBiYJf0fmfg08RbfmwM/BG4Izf/zcD1ufcZsHMj5bkgbbPNga7EVqr5wIb5bd5AOe8Dzm/k2Hgd2L9s2kHAkLRvP5nmOT6lbUX8+/m+uflvAW5Ir/sBbxADxR7AZoADZ5UdG79L26VHqs8+wHDiD6ktiH//fmFuHb8j3jOsd1rH5JTP4JR+HPGY3Tbl8cW0P7aoUO/y7Xsz8RjeNy3/rbT8RKA/0BP4PXBt2TH2GrB9qsepwDygd5XHwMWpntukbd0f2CaljSUG+o2d10NSmY9O69gReBP4Sq6OGfHv82uAjxA/B05vxc+K9dPxcSawTlpuKvD9XPobadv0SNtjDque57cTPys+kuY5B3gJ6N7QudJAmR8kHucbpscDwAONfBYMTtulf6VtDRxFvG3Gz4ifgR8FXgZ+1FAeuWVeyb2fDJzRxD4cm9ZzDCvPg+XAo2X74OHcMqt7rtxMPG72T3kcmMowqMK5UWn7vFI27b/7qTX2e5pne2ILeo/GtmOD27a5C+jRsR/pRP5+7v0X04Gd/zL6FXBFen0wMKUsj+2JgUDXCuv4NXBJel06yT+dSz8BeDb3/mXgqLI8RuQP9DRtP+Bu4oflcmJX1NZldVsC1Jc9VrDqh9cPiB/ApQ/HZ4BrytadpWUXANOAn9NA8JTm/zjxC71fej8KeLmRfbAfMDf3/r8nfXpfMZAhfsktAnYty/MfpTpSOZC5Dbi6kXK9B4xo4vgZB/wq9/4p4OT0uhfxC/9z6f0Y4Pdlyx9E+tDLHRu7NrHOE4Gn0+v+aZnNc+l7sOqH8z+BI8ryuI8KXyQ0HMjkv/x6pvy/kpv2HVY9huuA83LvA/Hu84c0dQykeRcD+1aYdyxNBzKnAU+WTbsQeKjsmM6f55cC9zSSZx3N+6w4hHjn5JBLPx74d3p9aNom+fT/JZ3nxB86GTAwl94FWEg6H2gkkCH+mMqALXPThqVpm+Tq1JJAZinQMzftGNI5Xp5HbpmWBDIvlE2b28A+WNCK58rN5I71NG0e8KUK50al7dNYILPa+z1N2zLN16+x7djQQzeNXDvNzr1+mzgeZF7ZtFKT8xBgYPjgyPWM+MtyVghhNHAs8cQJxF8t5XdJza9zSS5/iMFCY2M34gqz7H5i1E4I4WPEGyneH0IYkqUjndha8Iv8ciE3Oj6EEFJZf5Fl2ftp8g3ARSGE/8mybHGatjyrcgBolmX/CiE8Q2yZupz4q/im3Dq3J7aibEv8UgzEX8Ut0Scte1/IXZlE/LXWv+FF/qs3MSir5AP7IcSxSacQW3+6EX8t/SU3y03EL/UrgK8Cs7IsezKlDQE+V3bsBOKvzby6snXuBZwFfIz4y74r8QMdYqsOxA/Gkull+Q0BfhZCuDI3rRvwKtX77/GaZdnb8bD5wHlT3i1Tl1smCyHMIO2TJo6BvsQWjpebUb5yA4itH3lTgHyXZ/l5Xn4eNqQ5nxUDiF9O+eNySpoOcVtML0vPH49D0vPzaXuXdM/l0ZjSPPk8p+TSZtNyc7Msezv3vo6mz7eWKC/j2zRy3LXCudLQOqs5LpqjtfZ7b1b+wGwWjZGR6cRfHhuUPdbNsmxWCOFzxGbx44E+6cv/PuIHdbWeJXZTVC3LspeIX56DiE3I1dqD2AQ7KvWhzyE2Y9YQf1G21E3AUalfd0dgfC7tTmKrz9Asy3rT8ODivCXEL7aSTXOv56f0Pcv2x4eyLLuoiXy3Jm7rSlbZDyGEAcSm7POJv2jXJzav5/ftncCWIYRPEX+Z3ZRLm0789ZYv5/pZHECdtyK3zh7AvSnfgWl7/TC3zlnpeWBu+fzr0npHla23JsuybzdS99YwuPQiBcwDWRk8NXYMzCPu0y0r5LuiwvS8maz8QijZPE1vKzOBQWHVb6N8GWY1kJ4vc+lLdsuyfdczy7I7qlw/5PYDK8dilNIWU/ncgsrbul8IoWfu/WBW7tvSj5+W5NtirXSuNFdD9SjfprBq/Vtrv29NbLF6r7mFViAj9wOlgYi9QrRZCOHLKb03sZtnHpCFEPYl9ts2x73EAKOiEMKoEMJXQvovlDSw7lvAi1mWNefW8scRxyd8jDg+ZjviCXITLRwRn9xJDJCuBB7JsmxWLq03sZl0UQhhILGvuDEOHBlC6JEG5Z1SSki/an4CjAshbAkQQqgJ8X94yj88/ysFWH2J/e2V3Muqg4FriJ8B84D3Qwg7AofnF8iyrB64hxjslAdw4wFL+27dEEKXNDhw70bK0IM4LmtBlmXvhBC2IjaXl9b3KrGZ/qJ0PPYDyi9rvQIYG+Lg3BBCWC+EsHNqxVuTRoUQPhXiINDvE1teHkhpFY+BtE//D7gkxMHRpXPsE2mWOcRW0R6NrPsOYPsQwhEhDgb/DPF4vqFVa9i4B4j77rR07A4jfrGWynA/8Zj6foiDmz9F7IYFIMuyucSW3KtDusw2hLBBCOHLoewvEhqSZdlrwMPAZWm5DYHLgElZlpVaHRz4Rjpn+hLH8+RV2tZdiMfceiEOth5DHA9GlmXzScFziFfefYLY6lueb9WDlqvUGudKczW0fZ4lBnr7pXP8y8CuufTW2u97ET+jmk2BTCeXmlP3IP5Sf4n4Yfw7YgAA8BDxyp+nia0FBxO/2JrjIWBZCGFEI/MsIHZh/CuEsIQ4NqOeONagKulEPgAYl2XZnPyD2Kr0yRCCNbPsAGRZtpBY732IlzrnHUfsU19EHONzVxPZnUj80HuTOAbh5rL0s4EJwIQQwlvEAZnfovHzdRRwcypnJbcC26YParIs+1duXfXEL9+GfhnfRKz3Q+nLhLT8HOJl7gcQm+IXELdRg1fdpGUWA98mfqkvJrYAlXdTHkIMEl4FnmDl9lya8riOOAD7prTOGcQvrO6N1L01XEsMZBcAXyOOeSlt76aOgdOJ+/reNM9jrGyhuYvYojAnxCtLylteyLJsGnH8xInEgZW3EgdV/6rVateEVNeRxGD4deJ5PZ7Y3VoKevclbpsFxG31f2XZHEscWD85hLCIOPbrK8QuhWocRtx+L6VHPXBELv0M4g+v2cQv+TvLlq+0racTWxamET97HiQeYyVHEj+LFqb6lgeQVxCD+voQwgtV1qVRrXGutMAHtk8W/67he8Tj/01g7/9v545RGoiCMI5/X2knwV5vYJM62HkKIWWuoFgGcoVcITfIJWz0AHoEi/STYp5Fgohskt1M+P+6B7vw4O3bnTczrLLB+GeeB6+77Wvl873sMmnvlrWA02in9JeImLTxg/LDezfkvCpqWZzPiHAb30h6kzTe62/47d6Zsln36a/rzontR2WwdRUDvbCcfViv+/1ZqM/2VLm2x86o9O4c9koXthfK/qxOGSWafdGLiFgrTzk4spb6vv3ntUt1PPX0xfa98qT2oay1zyWtKr2YgT5cyl6JiOdD7qe0hKF8qfafdIf0rWxgvlQjZXlmo0yXvytT2wB2sVdEaQkAABRGRgYAAJRFIAMAAMoikAEAAGURyAAAgLIIZAAAQFkEMgAAoKwtbts5/kmcW/8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "shap.summary_plot(shap_values[1], df_train[features], plot_type=\"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interpretation is now as easy as it could be. The longer the bar the more important the feature. And now, guess what the 4 most important features were in the logistic regression: Correct, *distToGoalLine*, *distToCenter*, *head/body* and *counterAttack* :-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While lightGBM is a black box model, it is still rather simple to identify which features are the most important ones and whether they affect the scoring probability positively or negatively. As said before, in my opinion a lack of feature interpretation should not be a reason not to use lightGBM. \n",
    "\n",
    "A whole different story, however, is the next challenge!\n",
    "\n",
    "<a id=\"monotonicity\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge 3: Missing monotonicity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You remember that for the logistic regression all features are montone in the target, given that all other features stay constant? Well, this is not the case for lightGBM. While the non-monotonicity is often useful as lightGBM can easily learn more complex representations, it is sometimes not what we would expect nor what we want to algorithm to do. \n",
    "\n",
    "Let's look at an example to make things a little bit clearer. More precisely, let's look at the scoring probabilities when shooting with the strong foot from a distance of 23m - 30m."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_sample_shots(dist_goal_line, dist_center, duel, corner, smart_pass, shot_before, counter_attack, body_part):\n",
    "    \"\"\"\n",
    "    Helper function to create sample shots for different distance to the goal line and all other variables being fixed.\n",
    "    \"\"\"\n",
    "    \n",
    "    # build a data frame with all the shots\n",
    "    df_shots = pd.DataFrame()\n",
    "    df_shots[\"distToGoalLine\"] = dist_goal_line \n",
    "    df_shots[\"distToCenter\"] = dist_center\n",
    "    df_shots[\"duel\"] = duel\n",
    "    df_shots[\"corner\"] = corner\n",
    "    df_shots[\"smartPass\"] = smart_pass\n",
    "    df_shots[\"shotBefore\"] = shot_before\n",
    "    df_shots[\"counterAttack\"] = counter_attack\n",
    "\n",
    "    # set up the body part\n",
    "    for i, col in enumerate([\"head/body\", \"weakFoot\", \"strongFoot\"]):\n",
    "        df_shots[col] = 1*(col == body_part)\n",
    "    \n",
    "    return df_shots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "LightGBM",
         "type": "scatter",
         "x": [
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          2.9724252787151175,
          2.9724252787151175,
          2.551621201921541,
          2.638757501093832,
          2.638757501093832,
          2.6597235600622215,
          2.3974456176053907,
          2.3365478057238214
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Scoring probability"
        },
        "xaxis": {
         "title": {
          "text": "Distance to goal line (in m)"
         }
        },
        "yaxis": {
         "title": {
          "text": "Scoring probability (in %)"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"1747d860-3a36-4012-89fd-c1d2a5e5eec5\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"1747d860-3a36-4012-89fd-c1d2a5e5eec5\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '1747d860-3a36-4012-89fd-c1d2a5e5eec5',\n",
       "                        [{\"mode\": \"lines\", \"name\": \"LightGBM\", \"type\": \"scatter\", \"x\": [23, 24, 25, 26, 27, 28, 29, 30], \"y\": [2.9724252787151175, 2.9724252787151175, 2.551621201921541, 2.638757501093832, 2.638757501093832, 2.6597235600622215, 2.3974456176053907, 2.3365478057238214]}],\n",
       "                        {\"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Scoring probability\"}, \"xaxis\": {\"title\": {\"text\": \"Distance to goal line (in m)\"}}, \"yaxis\": {\"title\": {\"text\": \"Scoring probability (in %)\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('1747d860-3a36-4012-89fd-c1d2a5e5eec5');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Set assumptions about the shot\n",
    "dist_goal_line = np.arange(23,31)\n",
    "dist_center = 0\n",
    "counter_attack = 0\n",
    "duel = 0\n",
    "corner = 0\n",
    "smart_pass = 0\n",
    "shot_before = 0\n",
    "body_part = \"strongFoot\"\n",
    "\n",
    "# create a sample data frame with the shots\n",
    "df_sample = create_sample_shots(dist_goal_line, dist_center, counter_attack, duel, corner, smart_pass, \n",
    "                                shot_before, body_part)\n",
    "\n",
    "# get the prediction from the model\n",
    "df_sample[\"predLGBM\"] = model.predict_proba(df_sample[features])[:,1] * 100\n",
    "\n",
    "# plot the scoring probability\n",
    "fig = go.Figure()\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLGBM\"],\n",
    "                         mode='lines',\n",
    "                         name='LightGBM'))\n",
    "\n",
    "fig.update_layout(title=\"Scoring probability\",\n",
    "                  xaxis_title=\"Distance to goal line (in m)\",\n",
    "                  yaxis_title=\"Scoring probability (in %)\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm, that looks very weird... While the scoring probability from 24 meters is ~2.97%, it is only ~2.55% from 25 meters and then goes up to ~2.64% for 26 meters. That doesn't make any sense, does it? I mean it is exactly the same shot and the model tells us that if you go back and take the shot from 26m instead of 25m, the scoring probability raises... So, can we fix this problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution attempt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of you might know that you can add montone constraints to both lightGBM and XGBoost (unfortunately, this does not exist for a random forest). So this one is easy, isn't it? We can just add a constraint telling the model to be monotone in *distToGoalLine* and everything is good! So let's do exactly this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='gbdt', class_weight=None,\n",
       "               colsample_bytree=0.9053563761947043, importance_type='split',\n",
       "               is_unbalance=False, learning_rate=0.03811321514778682,\n",
       "               max_depth=-1, min_child_samples=395, min_child_weight=0.001,\n",
       "               min_split_gain=0.0,\n",
       "               monotone_constraints=[-1, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "               n_estimators=118, n_jobs=-1, num_leaves=20, objective='binary',\n",
       "               random_state=None, reg_alpha=0.5617832258471822,\n",
       "               reg_lambda=0.34491347534610917, silent=True,\n",
       "               subsample=0.5869782389570299, subsample_for_bin=140000,\n",
       "               subsample_freq=0)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# notice that -1 means monotone decreasing\n",
    "monotone_features = [0 if feat != \"distToGoalLine\" else -1 for feat in features]\n",
    "\n",
    "model_mon = lgb.LGBMClassifier(**hyp, monotone_constraints=monotone_features)\n",
    "model_mon.fit(df_train[features], df_train[\"goal\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's now check how our predictions look like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "LightGBM",
         "type": "scatter",
         "x": [
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          3.662276900977642,
          3.6397310924984416,
          2.8209635529739674,
          2.7600328434376444,
          2.7600328434376444,
          2.7600328434376444,
          2.338432558983722,
          2.1749856022307243
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Scoring probability"
        },
        "xaxis": {
         "title": {
          "text": "Distance to goal line (in m)"
         }
        },
        "yaxis": {
         "title": {
          "text": "Scoring probability (in %)"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"099978a5-e613-440d-aba1-819dc73d478a\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"099978a5-e613-440d-aba1-819dc73d478a\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '099978a5-e613-440d-aba1-819dc73d478a',\n",
       "                        [{\"mode\": \"lines\", \"name\": \"LightGBM\", \"type\": \"scatter\", \"x\": [23, 24, 25, 26, 27, 28, 29, 30], \"y\": [3.662276900977642, 3.6397310924984416, 2.8209635529739674, 2.7600328434376444, 2.7600328434376444, 2.7600328434376444, 2.338432558983722, 2.1749856022307243]}],\n",
       "                        {\"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Scoring probability\"}, \"xaxis\": {\"title\": {\"text\": \"Distance to goal line (in m)\"}}, \"yaxis\": {\"title\": {\"text\": \"Scoring probability (in %)\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('099978a5-e613-440d-aba1-819dc73d478a');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get the prediction from the model\n",
    "df_sample[\"predLGBM\"] = model_mon.predict_proba(df_sample[features])[:,1] * 100\n",
    "\n",
    "# plot the scoring probability\n",
    "fig = go.Figure()\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLGBM\"],\n",
    "                         mode='lines',\n",
    "                         name='LightGBM'))\n",
    "\n",
    "fig.update_layout(title=\"Scoring probability\",\n",
    "                  xaxis_title=\"Distance to goal line (in m)\",\n",
    "                  yaxis_title=\"Scoring probability (in %)\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, this looks way better! So, we solved the problem, didn't we? But wait, what about shots that are close to the goal line but not in front of the goal? They should get a relatively small scoring probability as the angle it pretty small. Does our model capture this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Non-monotone model",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14
         ],
         "y": [
          4.404248422876747,
          4.404248422876747,
          4.404248422876747,
          4.328819958263193,
          4.504967133476284,
          4.3018665473477,
          5.038359446271304,
          5.963342980959372,
          6.076044391570604,
          5.78956693897007,
          5.78956693897007,
          4.859277262684083,
          4.156495888424782,
          3.9501275444065556
         ]
        },
        {
         "mode": "lines",
         "name": "Monotone model",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14
         ],
         "y": [
          6.8752090896143825,
          6.8752090896143825,
          6.8752090896143825,
          6.6852981496851624,
          6.205767197061064,
          5.956817350147063,
          5.940187818750239,
          5.8439594261228605,
          5.80118739969208,
          5.428884757432341,
          5.428884757432341,
          5.180181460771498,
          4.882156226376888,
          4.43660822669196
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Scoring probability"
        },
        "xaxis": {
         "title": {
          "text": "Distance to goal line (in m)"
         }
        },
        "yaxis": {
         "title": {
          "text": "Scoring probability (in %)"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"3e7648da-90fb-4eb1-bda5-a705b15ea44e\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"3e7648da-90fb-4eb1-bda5-a705b15ea44e\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '3e7648da-90fb-4eb1-bda5-a705b15ea44e',\n",
       "                        [{\"mode\": \"lines\", \"name\": \"Non-monotone model\", \"type\": \"scatter\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], \"y\": [4.404248422876747, 4.404248422876747, 4.404248422876747, 4.328819958263193, 4.504967133476284, 4.3018665473477, 5.038359446271304, 5.963342980959372, 6.076044391570604, 5.78956693897007, 5.78956693897007, 4.859277262684083, 4.156495888424782, 3.9501275444065556]}, {\"mode\": \"lines\", \"name\": \"Monotone model\", \"type\": \"scatter\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], \"y\": [6.8752090896143825, 6.8752090896143825, 6.8752090896143825, 6.6852981496851624, 6.205767197061064, 5.956817350147063, 5.940187818750239, 5.8439594261228605, 5.80118739969208, 5.428884757432341, 5.428884757432341, 5.180181460771498, 4.882156226376888, 4.43660822669196]}],\n",
       "                        {\"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Scoring probability\"}, \"xaxis\": {\"title\": {\"text\": \"Distance to goal line (in m)\"}}, \"yaxis\": {\"title\": {\"text\": \"Scoring probability (in %)\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('3e7648da-90fb-4eb1-bda5-a705b15ea44e');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Set assumptions about the shot\n",
    "dist_goal_line = np.arange(1, 15)\n",
    "dist_center = 15\n",
    "counter_attack = 0\n",
    "duel = 0\n",
    "corner = 0\n",
    "smart_pass = 0\n",
    "shot_before = 0\n",
    "body_part = \"strongFoot\"\n",
    "\n",
    "# create a sample data frame with the shots\n",
    "df_sample = create_sample_shots(dist_goal_line, dist_center, counter_attack, duel, corner, smart_pass, \n",
    "                                shot_before, body_part)\n",
    "\n",
    "# get the prediction from the model\n",
    "df_sample[\"predLGBM\"] = model.predict_proba(df_sample[features])[:,1] * 100\n",
    "df_sample[\"predLGBMMonotone\"] = model_mon.predict_proba(df_sample[features])[:,1] * 100\n",
    "\n",
    "# plot the scoring probability\n",
    "fig = go.Figure()\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLGBM\"],\n",
    "                         mode='lines',\n",
    "                         name='Non-monotone model'))\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLGBMMonotone\"],\n",
    "                         mode='lines',\n",
    "                         name='Monotone model'))\n",
    "\n",
    "fig.update_layout(title=\"Scoring probability\",\n",
    "                  xaxis_title=\"Distance to goal line (in m)\",\n",
    "                  yaxis_title=\"Scoring probability (in %)\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no, while the non-monotone model was pretty good in capturing the fact that even though the distance gets bigger the scoring probability gets bigger as the angle get wider, we destroyed this by implementing the monotone constraint. :-( \n",
    "\n",
    "Hm, let's think about whether there is any way to fix this... So what we said was, that given all other features stay the same, the monotone constraint makes sure that the further from the goal line, the less likely it is to score. So, what happens if we add the feature *angleClip* (to remind you: this measures the angle to the post, but was clipped to only keep the small angles)? This way we can make sure that when the angle stays the same, i.e. we are in front of the goal or the angle is very wide, the monotonicity constraint is active. For small angles, however, we do have the freedom to be non-monotone as this is what we want. And that way we can even say that we also want to be monotone decreasing in the distance to the center. So, let's try this out..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='gbdt', class_weight=None,\n",
       "               colsample_bytree=0.9053563761947043, importance_type='split',\n",
       "               is_unbalance=False, learning_rate=0.03811321514778682,\n",
       "               max_depth=-1, min_child_samples=395, min_child_weight=0.001,\n",
       "               min_split_gain=0.0,\n",
       "               monotone_constraints=[-1, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "               n_estimators=118, n_jobs=-1, num_leaves=20, objective='binary',\n",
       "               random_state=None, reg_alpha=0.5617832258471822,\n",
       "               reg_lambda=0.34491347534610917, silent=True,\n",
       "               subsample=0.5869782389570299, subsample_for_bin=140000,\n",
       "               subsample_freq=0)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# notice that -1 means monotone decreasing\n",
    "features_mon = features + [\"angleClip\"]\n",
    "monotone_features = [0 if feat not in [\"distToGoalLine\", \"distToCenter\"] else -1 for feat in features_mon]\n",
    "\n",
    "model_mon2 = lgb.LGBMClassifier(**hyp, monotone_constraints=monotone_features)\n",
    "model_mon2.fit(df_train[features_mon], df_train[\"goal\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Non-monotone model",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14
         ],
         "y": [
          4.404248422876747,
          4.404248422876747,
          4.404248422876747,
          4.328819958263193,
          4.504967133476284,
          4.3018665473477,
          5.038359446271304,
          5.963342980959372,
          6.076044391570604,
          5.78956693897007,
          5.78956693897007,
          4.859277262684083,
          4.156495888424782,
          3.9501275444065556
         ]
        },
        {
         "mode": "lines",
         "name": "Monotone model",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14
         ],
         "y": [
          4.69500497736004,
          4.69500497736004,
          4.69500497736004,
          4.69500497736004,
          4.69500497736004,
          4.69500497736004,
          5.172718783047914,
          6.403235841991249,
          6.461769607579729,
          6.00455980759151,
          5.351760941619733,
          5.87949389887135,
          5.079580820847889,
          3.9519115253823345
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Scoring probability"
        },
        "xaxis": {
         "title": {
          "text": "Distance to goal line (in m)"
         }
        },
        "yaxis": {
         "title": {
          "text": "Scoring probability (in %)"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"7073b510-bcb6-4579-a8a2-b3001caa3a6b\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"7073b510-bcb6-4579-a8a2-b3001caa3a6b\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '7073b510-bcb6-4579-a8a2-b3001caa3a6b',\n",
       "                        [{\"mode\": \"lines\", \"name\": \"Non-monotone model\", \"type\": \"scatter\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], \"y\": [4.404248422876747, 4.404248422876747, 4.404248422876747, 4.328819958263193, 4.504967133476284, 4.3018665473477, 5.038359446271304, 5.963342980959372, 6.076044391570604, 5.78956693897007, 5.78956693897007, 4.859277262684083, 4.156495888424782, 3.9501275444065556]}, {\"mode\": \"lines\", \"name\": \"Monotone model\", \"type\": \"scatter\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], \"y\": [4.69500497736004, 4.69500497736004, 4.69500497736004, 4.69500497736004, 4.69500497736004, 4.69500497736004, 5.172718783047914, 6.403235841991249, 6.461769607579729, 6.00455980759151, 5.351760941619733, 5.87949389887135, 5.079580820847889, 3.9519115253823345]}],\n",
       "                        {\"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Scoring probability\"}, \"xaxis\": {\"title\": {\"text\": \"Distance to goal line (in m)\"}}, \"yaxis\": {\"title\": {\"text\": \"Scoring probability (in %)\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('7073b510-bcb6-4579-a8a2-b3001caa3a6b');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# add the angle to the sample data\n",
    "df_sample = egm.create_features(df_sample, overwrite=False)\n",
    "\n",
    "df_sample[\"predLGBMMonotone\"] = model_mon2.predict_proba(df_sample[features_mon])[:,1] * 100\n",
    "\n",
    "\n",
    "# plot the scoring probability\n",
    "fig = go.Figure()\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLGBM\"],\n",
    "                         mode='lines',\n",
    "                         name='Non-monotone model'))\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLGBMMonotone\"],\n",
    "                         mode='lines',\n",
    "                         name='Monotone model'))\n",
    "\n",
    "fig.update_layout(title=\"Scoring probability\",\n",
    "                  xaxis_title=\"Distance to goal line (in m)\",\n",
    "                  yaxis_title=\"Scoring probability (in %)\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, nice, we are now capable of capturing the non-monotonicity by adding the *angle* feature. However, to be completely honest with you, the bump at 11 meters compared to 10m and 12m does still look a little bit weird, doesn't it? I would bet that in reality this curve is smoother and the bump doesn't happen..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While lightGBM comes with the possibility to enforce monotonicity in features, it can in reality be rather difficult to do so in a meaningful way. Having said this, enforcing \"smoothness\" for non-monotone features is often almost impossible. So:\n",
    "\n",
    "<b>If it is highly relevant for your model to have monotone and \"smooth\" predictions and you would be willing to potentially trade this for some model accuracy, there are often better alternatives out there than lightGBM.</b>\n",
    "\n",
    "<a id=\"extrapolation\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge 4: No extrapolation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this chapter we want to look into another potential drawback of lightGBM, namely the non-ability for extrapolation. \n",
    "\n",
    "But first, what is extrapolation? Extrapolation is the estimation of the value of a new data point which is outside the range of previously observed values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Toy example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let me try to explain this to you with an toy *regression* example. I know that we have not yet looked into regression problems but I believe that the problems becomes way clearer with a regression rather than a classification example. And trust me, the underlying problem is the same as we will see below. \n",
    "\n",
    "Ok, so let's build our toy example in which the otucome raises monotone in a feature X. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "x = np.arange(1,98)\n",
    "vals = x / 2 + np.random.normal(0,1,len(x))\n",
    "df = pd.DataFrame({\"x\": np.arange(1,98), \n",
    "                   \"value\": vals})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check how the data points look like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "x=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "",
         "showlegend": false,
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60,
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97
         ],
         "xaxis": "x",
         "y": [
          0.9967141530112327,
          0.8617356988288154,
          2.1476885381006925,
          3.5230298564080256,
          2.265846625276664,
          2.7658630430508193,
          5.0792128155073915,
          4.7674347291529084,
          4.030525614065048,
          5.542560043585965,
          5.036582307187538,
          5.5342702464297435,
          6.741962271566035,
          5.086719755342202,
          5.775082167486968,
          7.437712470759028,
          7.487168879665576,
          9.314247332595274,
          8.591975924478788,
          8.587696298664708,
          11.965648768921554,
          10.774223699513465,
          11.567528204687925,
          10.575251813786544,
          11.955617275474818,
          13.110922589709865,
          12.349006422577697,
          14.375698018345672,
          13.899361310081195,
          14.708306250206723,
          14.898293387770604,
          17.852278184508936,
          16.486502775262068,
          15.9422890710441,
          18.32254491210319,
          16.779156350028977,
          18.708863595004754,
          17.040329876120225,
          18.17181395110157,
          20.19686123586912,
          21.23846657999541,
          21.17136828118997,
          21.38435171761176,
          21.69889630441071,
          21.021478009632574,
          22.280155791605292,
          23.039361229040214,
          25.057122226218915,
          24.84361828956846,
          23.236959844637266,
          25.824083969394795,
          25.614917719583683,
          25.823077999694043,
          27.61167628884087,
          28.53099952249595,
          28.9312801191162,
          27.66078247677736,
          28.690787624148786,
          29.831263431403563,
          30.97554512712236,
          30.02082576215471,
          30.814341023336183,
          30.393665025993972,
          30.80379337591933,
          33.3125258223942,
          34.356240028570824,
          33.427989878419666,
          35.00353289789202,
          34.86163602504764,
          34.354880245394874,
          35.86139560550841,
          37.53803656646597,
          36.46417396089005,
          38.56464365581401,
          34.880254895910255,
          38.821902504375224,
          38.58704706823817,
          38.70099264953413,
          39.5917607765355,
          38.01243108539911,
          40.280328112162486,
          41.35711257151175,
          42.97789404474152,
          41.48172978172635,
          41.691506397106814,
          42.49824295641546,
          44.41540211770207,
          44.328751109659684,
          43.97023979623296,
          45.513267433113356,
          45.59707754934804,
          46.96864499053289,
          45.79794690612265,
          46.67233785340223,
          47.107891846867844,
          46.536485051867885,
          48.796120277064574
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "x"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "value"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"51fcba0f-8cdb-457b-b0b2-d6c472127beb\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"51fcba0f-8cdb-457b-b0b2-d6c472127beb\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '51fcba0f-8cdb-457b-b0b2-d6c472127beb',\n",
       "                        [{\"hovertemplate\": \"x=%{x}<br>value=%{y}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\", \"symbol\": \"circle\"}, \"mode\": \"markers\", \"name\": \"\", \"showlegend\": false, \"type\": \"scatter\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97], \"xaxis\": \"x\", \"y\": [0.9967141530112327, 0.8617356988288154, 2.1476885381006925, 3.5230298564080256, 2.265846625276664, 2.7658630430508193, 5.0792128155073915, 4.7674347291529084, 4.030525614065048, 5.542560043585965, 5.036582307187538, 5.5342702464297435, 6.741962271566035, 5.086719755342202, 5.775082167486968, 7.437712470759028, 7.487168879665576, 9.314247332595274, 8.591975924478788, 8.587696298664708, 11.965648768921554, 10.774223699513465, 11.567528204687925, 10.575251813786544, 11.955617275474818, 13.110922589709865, 12.349006422577697, 14.375698018345672, 13.899361310081195, 14.708306250206723, 14.898293387770604, 17.852278184508936, 16.486502775262068, 15.9422890710441, 18.32254491210319, 16.779156350028977, 18.708863595004754, 17.040329876120225, 18.17181395110157, 20.19686123586912, 21.23846657999541, 21.17136828118997, 21.38435171761176, 21.69889630441071, 21.021478009632574, 22.280155791605292, 23.039361229040214, 25.057122226218915, 24.84361828956846, 23.236959844637266, 25.824083969394795, 25.614917719583683, 25.823077999694043, 27.61167628884087, 28.53099952249595, 28.9312801191162, 27.66078247677736, 28.690787624148786, 29.831263431403563, 30.97554512712236, 30.02082576215471, 30.814341023336183, 30.393665025993972, 30.80379337591933, 33.3125258223942, 34.356240028570824, 33.427989878419666, 35.00353289789202, 34.86163602504764, 34.354880245394874, 35.86139560550841, 37.53803656646597, 36.46417396089005, 38.56464365581401, 34.880254895910255, 38.821902504375224, 38.58704706823817, 38.70099264953413, 39.5917607765355, 38.01243108539911, 40.280328112162486, 41.35711257151175, 42.97789404474152, 41.48172978172635, 41.691506397106814, 42.49824295641546, 44.41540211770207, 44.328751109659684, 43.97023979623296, 45.513267433113356, 45.59707754934804, 46.96864499053289, 45.79794690612265, 46.67233785340223, 47.107891846867844, 46.536485051867885, 48.796120277064574], \"yaxis\": \"y\"}],\n",
       "                        {\"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"x\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"value\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('51fcba0f-8cdb-457b-b0b2-d6c472127beb');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = px.scatter(df, x=\"x\", y=\"value\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, let's now do the following experiment. We use the data points 1 - 60 for training and the data points > 60 for testing. You see that this way values in our test are *outside* the range of values we are seeing in our training set, i.e. we are clearly in a situation of extrapolation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check how good a linear regression is capable of predicting the values in the test set. And notice, that as mentioned above, as this is a regression rather than a classification problem, we use linear regression instead of logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy_train = df[df[\"x\"] <= 60].copy()\n",
    "dummy_test = df[df[\"x\"] > 60].copy()\n",
    "\n",
    "lr = LinearRegression()\n",
    "lr.fit(np.array(dummy_train[\"x\"]).reshape(-1, 1), dummy_train[\"value\"])\n",
    "\n",
    "pred_vals = lr.predict(np.array(dummy_test[\"x\"]).reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "set=Train<br>x=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "Train",
         "marker": {
          "color": "#636efa",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "Train",
         "showlegend": true,
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60
         ],
         "xaxis": "x",
         "y": [
          0.9967141530112327,
          0.8617356988288154,
          2.1476885381006925,
          3.5230298564080256,
          2.265846625276664,
          2.7658630430508193,
          5.0792128155073915,
          4.7674347291529084,
          4.030525614065048,
          5.542560043585965,
          5.036582307187538,
          5.5342702464297435,
          6.741962271566035,
          5.086719755342202,
          5.775082167486968,
          7.437712470759028,
          7.487168879665576,
          9.314247332595274,
          8.591975924478788,
          8.587696298664708,
          11.965648768921554,
          10.774223699513465,
          11.567528204687925,
          10.575251813786544,
          11.955617275474818,
          13.110922589709865,
          12.349006422577697,
          14.375698018345672,
          13.899361310081195,
          14.708306250206723,
          14.898293387770604,
          17.852278184508936,
          16.486502775262068,
          15.9422890710441,
          18.32254491210319,
          16.779156350028977,
          18.708863595004754,
          17.040329876120225,
          18.17181395110157,
          20.19686123586912,
          21.23846657999541,
          21.17136828118997,
          21.38435171761176,
          21.69889630441071,
          21.021478009632574,
          22.280155791605292,
          23.039361229040214,
          25.057122226218915,
          24.84361828956846,
          23.236959844637266,
          25.824083969394795,
          25.614917719583683,
          25.823077999694043,
          27.61167628884087,
          28.53099952249595,
          28.9312801191162,
          27.66078247677736,
          28.690787624148786,
          29.831263431403563,
          30.97554512712236
         ],
         "yaxis": "y"
        },
        {
         "hovertemplate": "set=Test<br>x=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "Test",
         "marker": {
          "color": "#EF553B",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "Test",
         "showlegend": true,
         "type": "scatter",
         "x": [
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97
         ],
         "xaxis": "x",
         "y": [
          30.02082576215471,
          30.814341023336183,
          30.393665025993972,
          30.80379337591933,
          33.3125258223942,
          34.356240028570824,
          33.427989878419666,
          35.00353289789202,
          34.86163602504764,
          34.354880245394874,
          35.86139560550841,
          37.53803656646597,
          36.46417396089005,
          38.56464365581401,
          34.880254895910255,
          38.821902504375224,
          38.58704706823817,
          38.70099264953413,
          39.5917607765355,
          38.01243108539911,
          40.280328112162486,
          41.35711257151175,
          42.97789404474152,
          41.48172978172635,
          41.691506397106814,
          42.49824295641546,
          44.41540211770207,
          44.328751109659684,
          43.97023979623296,
          45.513267433113356,
          45.59707754934804,
          46.96864499053289,
          45.79794690612265,
          46.67233785340223,
          47.107891846867844,
          46.536485051867885,
          48.796120277064574
         ],
         "yaxis": "y"
        },
        {
         "mode": "lines",
         "name": "prediction",
         "type": "scatter",
         "x": [
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97
         ],
         "y": [
          30.323558822354684,
          30.82284451105715,
          31.322130199759613,
          31.821415888462077,
          32.32070157716454,
          32.819987265867006,
          33.31927295456947,
          33.818558643271935,
          34.3178443319744,
          34.81713002067686,
          35.31641570937933,
          35.81570139808179,
          36.31498708678426,
          36.81427277548672,
          37.313558464189185,
          37.81284415289165,
          38.312129841594114,
          38.81141553029658,
          39.31070121899904,
          39.80998690770151,
          40.30927259640397,
          40.808558285106436,
          41.3078439738089,
          41.807129662511365,
          42.30641535121383,
          42.80570103991629,
          43.30498672861876,
          43.80427241732122,
          44.30355810602369,
          44.80284379472615,
          45.302129483428615,
          45.80141517213108,
          46.300700860833544,
          46.79998654953601,
          47.29927223823847,
          47.79855792694094,
          48.2978436156434
         ]
        }
       ],
       "layout": {
        "legend": {
         "title": {
          "text": "set"
         },
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Linear Regression"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "x"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "value"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"866f7119-01e4-455d-b112-3dc4f5126efc\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"866f7119-01e4-455d-b112-3dc4f5126efc\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '866f7119-01e4-455d-b112-3dc4f5126efc',\n",
       "                        [{\"hovertemplate\": \"set=Train<br>x=%{x}<br>value=%{y}<extra></extra>\", \"legendgroup\": \"Train\", \"marker\": {\"color\": \"#636efa\", \"symbol\": \"circle\"}, \"mode\": \"markers\", \"name\": \"Train\", \"showlegend\": true, \"type\": \"scatter\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60], \"xaxis\": \"x\", \"y\": [0.9967141530112327, 0.8617356988288154, 2.1476885381006925, 3.5230298564080256, 2.265846625276664, 2.7658630430508193, 5.0792128155073915, 4.7674347291529084, 4.030525614065048, 5.542560043585965, 5.036582307187538, 5.5342702464297435, 6.741962271566035, 5.086719755342202, 5.775082167486968, 7.437712470759028, 7.487168879665576, 9.314247332595274, 8.591975924478788, 8.587696298664708, 11.965648768921554, 10.774223699513465, 11.567528204687925, 10.575251813786544, 11.955617275474818, 13.110922589709865, 12.349006422577697, 14.375698018345672, 13.899361310081195, 14.708306250206723, 14.898293387770604, 17.852278184508936, 16.486502775262068, 15.9422890710441, 18.32254491210319, 16.779156350028977, 18.708863595004754, 17.040329876120225, 18.17181395110157, 20.19686123586912, 21.23846657999541, 21.17136828118997, 21.38435171761176, 21.69889630441071, 21.021478009632574, 22.280155791605292, 23.039361229040214, 25.057122226218915, 24.84361828956846, 23.236959844637266, 25.824083969394795, 25.614917719583683, 25.823077999694043, 27.61167628884087, 28.53099952249595, 28.9312801191162, 27.66078247677736, 28.690787624148786, 29.831263431403563, 30.97554512712236], \"yaxis\": \"y\"}, {\"hovertemplate\": \"set=Test<br>x=%{x}<br>value=%{y}<extra></extra>\", \"legendgroup\": \"Test\", \"marker\": {\"color\": \"#EF553B\", \"symbol\": \"circle\"}, \"mode\": \"markers\", \"name\": \"Test\", \"showlegend\": true, \"type\": \"scatter\", \"x\": [61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97], \"xaxis\": \"x\", \"y\": [30.02082576215471, 30.814341023336183, 30.393665025993972, 30.80379337591933, 33.3125258223942, 34.356240028570824, 33.427989878419666, 35.00353289789202, 34.86163602504764, 34.354880245394874, 35.86139560550841, 37.53803656646597, 36.46417396089005, 38.56464365581401, 34.880254895910255, 38.821902504375224, 38.58704706823817, 38.70099264953413, 39.5917607765355, 38.01243108539911, 40.280328112162486, 41.35711257151175, 42.97789404474152, 41.48172978172635, 41.691506397106814, 42.49824295641546, 44.41540211770207, 44.328751109659684, 43.97023979623296, 45.513267433113356, 45.59707754934804, 46.96864499053289, 45.79794690612265, 46.67233785340223, 47.107891846867844, 46.536485051867885, 48.796120277064574], \"yaxis\": \"y\"}, {\"mode\": \"lines\", \"name\": \"prediction\", \"type\": \"scatter\", \"x\": [61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97], \"y\": [30.323558822354684, 30.82284451105715, 31.322130199759613, 31.821415888462077, 32.32070157716454, 32.819987265867006, 33.31927295456947, 33.818558643271935, 34.3178443319744, 34.81713002067686, 35.31641570937933, 35.81570139808179, 36.31498708678426, 36.81427277548672, 37.313558464189185, 37.81284415289165, 38.312129841594114, 38.81141553029658, 39.31070121899904, 39.80998690770151, 40.30927259640397, 40.808558285106436, 41.3078439738089, 41.807129662511365, 42.30641535121383, 42.80570103991629, 43.30498672861876, 43.80427241732122, 44.30355810602369, 44.80284379472615, 45.302129483428615, 45.80141517213108, 46.300700860833544, 46.79998654953601, 47.29927223823847, 47.79855792694094, 48.2978436156434]}],\n",
       "                        {\"legend\": {\"title\": {\"text\": \"set\"}, \"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Linear Regression\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"x\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"value\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('866f7119-01e4-455d-b112-3dc4f5126efc');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df[\"set\"] = np.where(df[\"x\"] <= 60, \"Train\", \"Test\")\n",
    "fig = px.scatter(df, x=\"x\", y=\"value\", color=\"set\")\n",
    "fig.add_trace(go.Scatter(x=dummy_test[\"x\"], y=pred_vals,\n",
    "                         mode='lines',\n",
    "                         name='prediction'))\n",
    "\n",
    "fig.update_layout(title=\"Linear Regression\")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, that looks very well, doesn't it? So, let's do the same thing for lightGBM..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "set=Train<br>x=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "Train",
         "marker": {
          "color": "#636efa",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "Train",
         "showlegend": true,
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60
         ],
         "xaxis": "x",
         "y": [
          0.9967141530112327,
          0.8617356988288154,
          2.1476885381006925,
          3.5230298564080256,
          2.265846625276664,
          2.7658630430508193,
          5.0792128155073915,
          4.7674347291529084,
          4.030525614065048,
          5.542560043585965,
          5.036582307187538,
          5.5342702464297435,
          6.741962271566035,
          5.086719755342202,
          5.775082167486968,
          7.437712470759028,
          7.487168879665576,
          9.314247332595274,
          8.591975924478788,
          8.587696298664708,
          11.965648768921554,
          10.774223699513465,
          11.567528204687925,
          10.575251813786544,
          11.955617275474818,
          13.110922589709865,
          12.349006422577697,
          14.375698018345672,
          13.899361310081195,
          14.708306250206723,
          14.898293387770604,
          17.852278184508936,
          16.486502775262068,
          15.9422890710441,
          18.32254491210319,
          16.779156350028977,
          18.708863595004754,
          17.040329876120225,
          18.17181395110157,
          20.19686123586912,
          21.23846657999541,
          21.17136828118997,
          21.38435171761176,
          21.69889630441071,
          21.021478009632574,
          22.280155791605292,
          23.039361229040214,
          25.057122226218915,
          24.84361828956846,
          23.236959844637266,
          25.824083969394795,
          25.614917719583683,
          25.823077999694043,
          27.61167628884087,
          28.53099952249595,
          28.9312801191162,
          27.66078247677736,
          28.690787624148786,
          29.831263431403563,
          30.97554512712236
         ],
         "yaxis": "y"
        },
        {
         "hovertemplate": "set=Test<br>x=%{x}<br>value=%{y}<extra></extra>",
         "legendgroup": "Test",
         "marker": {
          "color": "#EF553B",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "Test",
         "showlegend": true,
         "type": "scatter",
         "x": [
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97
         ],
         "xaxis": "x",
         "y": [
          30.02082576215471,
          30.814341023336183,
          30.393665025993972,
          30.80379337591933,
          33.3125258223942,
          34.356240028570824,
          33.427989878419666,
          35.00353289789202,
          34.86163602504764,
          34.354880245394874,
          35.86139560550841,
          37.53803656646597,
          36.46417396089005,
          38.56464365581401,
          34.880254895910255,
          38.821902504375224,
          38.58704706823817,
          38.70099264953413,
          39.5917607765355,
          38.01243108539911,
          40.280328112162486,
          41.35711257151175,
          42.97789404474152,
          41.48172978172635,
          41.691506397106814,
          42.49824295641546,
          44.41540211770207,
          44.328751109659684,
          43.97023979623296,
          45.513267433113356,
          45.59707754934804,
          46.96864499053289,
          45.79794690612265,
          46.67233785340223,
          47.107891846867844,
          46.536485051867885,
          48.796120277064574
         ],
         "yaxis": "y"
        },
        {
         "mode": "lines",
         "name": "prediction",
         "type": "scatter",
         "x": [
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97
         ],
         "y": [
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714,
          24.976137129582714
         ]
        }
       ],
       "layout": {
        "legend": {
         "title": {
          "text": "set"
         },
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "LightGBM"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "x"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "value"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"9b0f32e7-1f01-4b13-ad43-7a16248e3bd7\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"9b0f32e7-1f01-4b13-ad43-7a16248e3bd7\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '9b0f32e7-1f01-4b13-ad43-7a16248e3bd7',\n",
       "                        [{\"hovertemplate\": \"set=Train<br>x=%{x}<br>value=%{y}<extra></extra>\", \"legendgroup\": \"Train\", \"marker\": {\"color\": \"#636efa\", \"symbol\": \"circle\"}, \"mode\": \"markers\", \"name\": \"Train\", \"showlegend\": true, \"type\": \"scatter\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60], \"xaxis\": \"x\", \"y\": [0.9967141530112327, 0.8617356988288154, 2.1476885381006925, 3.5230298564080256, 2.265846625276664, 2.7658630430508193, 5.0792128155073915, 4.7674347291529084, 4.030525614065048, 5.542560043585965, 5.036582307187538, 5.5342702464297435, 6.741962271566035, 5.086719755342202, 5.775082167486968, 7.437712470759028, 7.487168879665576, 9.314247332595274, 8.591975924478788, 8.587696298664708, 11.965648768921554, 10.774223699513465, 11.567528204687925, 10.575251813786544, 11.955617275474818, 13.110922589709865, 12.349006422577697, 14.375698018345672, 13.899361310081195, 14.708306250206723, 14.898293387770604, 17.852278184508936, 16.486502775262068, 15.9422890710441, 18.32254491210319, 16.779156350028977, 18.708863595004754, 17.040329876120225, 18.17181395110157, 20.19686123586912, 21.23846657999541, 21.17136828118997, 21.38435171761176, 21.69889630441071, 21.021478009632574, 22.280155791605292, 23.039361229040214, 25.057122226218915, 24.84361828956846, 23.236959844637266, 25.824083969394795, 25.614917719583683, 25.823077999694043, 27.61167628884087, 28.53099952249595, 28.9312801191162, 27.66078247677736, 28.690787624148786, 29.831263431403563, 30.97554512712236], \"yaxis\": \"y\"}, {\"hovertemplate\": \"set=Test<br>x=%{x}<br>value=%{y}<extra></extra>\", \"legendgroup\": \"Test\", \"marker\": {\"color\": \"#EF553B\", \"symbol\": \"circle\"}, \"mode\": \"markers\", \"name\": \"Test\", \"showlegend\": true, \"type\": \"scatter\", \"x\": [61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97], \"xaxis\": \"x\", \"y\": [30.02082576215471, 30.814341023336183, 30.393665025993972, 30.80379337591933, 33.3125258223942, 34.356240028570824, 33.427989878419666, 35.00353289789202, 34.86163602504764, 34.354880245394874, 35.86139560550841, 37.53803656646597, 36.46417396089005, 38.56464365581401, 34.880254895910255, 38.821902504375224, 38.58704706823817, 38.70099264953413, 39.5917607765355, 38.01243108539911, 40.280328112162486, 41.35711257151175, 42.97789404474152, 41.48172978172635, 41.691506397106814, 42.49824295641546, 44.41540211770207, 44.328751109659684, 43.97023979623296, 45.513267433113356, 45.59707754934804, 46.96864499053289, 45.79794690612265, 46.67233785340223, 47.107891846867844, 46.536485051867885, 48.796120277064574], \"yaxis\": \"y\"}, {\"mode\": \"lines\", \"name\": \"prediction\", \"type\": \"scatter\", \"x\": [61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97], \"y\": [24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714, 24.976137129582714]}],\n",
       "                        {\"legend\": {\"title\": {\"text\": \"set\"}, \"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"LightGBM\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"x\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"value\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('9b0f32e7-1f01-4b13-ad43-7a16248e3bd7');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lgbm = lgb.LGBMRegressor()\n",
    "\n",
    "lgbm.fit(np.array(dummy_train[\"x\"]).reshape(-1, 1), dummy_train[\"value\"])\n",
    "\n",
    "pred_vals = lgbm.predict(np.array(dummy_test[\"x\"]).reshape(-1, 1))\n",
    "\n",
    "fig = px.scatter(df, x=\"x\", y=\"value\", color=\"set\")\n",
    "fig.add_trace(go.Scatter(x=dummy_test[\"x\"], y=pred_vals,\n",
    "                         mode='lines',\n",
    "                         name='prediction'))\n",
    "fig.update_layout(title=\"LightGBM\")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whut?! What the hell is this? The prediction is completely off... That is what I meant above, a gradient booster is extremely bad in extrapolation. Or to put in other words:\n",
    "\n",
    "<b> Gradient (tree) boosters will always predict values that are in the same range as observed on the training set. </b>\n",
    "\n",
    "And as in the above example the training set only contains values ~ 0 - 30, it will *always* only predict values between 0 and 30."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Long-range problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, good, so we look at this toy example for which lightGBM did not work. But why should we care in our real-world example? Let's look into the prediction we make when shooting the ball from a long-range. (Btw, notice that above we did not deal with outliers. I actually did that on purpose to show you this example ;-)) \n",
    "\n",
    "Let's start by creating same sample shots with the strong foot, but from long-range, i.e. shots between 30m and 100m from the goal.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set some assumptions about the shot (other than the distance)\n",
    "dist_goal_line = np.arange(28,101)\n",
    "dist_center = 0\n",
    "counter_attack = 0\n",
    "duel = 0\n",
    "corner = 0\n",
    "smart_pass = 0\n",
    "shot_before = 0\n",
    "body_part = \"strongFoot\"\n",
    "\n",
    "df_sample = create_sample_shots(dist_goal_line, dist_center, counter_attack, duel, corner, smart_pass, \n",
    "                                shot_before, body_part)\n",
    "# add the angle to the sample data\n",
    "df_sample = egm.create_features(df_sample, overwrite=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's predict the scoring probability with both the logistic regression and lightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Logistic Regression",
         "type": "scatter",
         "x": [
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60,
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97,
          98,
          99,
          100
         ],
         "y": [
          4.232267559755421,
          3.7317317945428536,
          3.2883600521101943,
          2.896081141090486,
          2.5493648568168594,
          2.2431982511187116,
          1.973056344967013,
          1.734869504800278,
          1.5249891979889152,
          1.3401534184961066,
          1.1774527254726397,
          1.0342975574658628,
          0.9083872640742103,
          0.7976811260388539,
          0.7003715052493849,
          0.6148591701394218,
          0.539730772670643,
          0.4737384048009734,
          0.41578113023789043,
          0.36488836750779324,
          0.3202049898114055,
          0.280978003317481,
          0.2465446665464496,
          0.21632191782628601,
          0.18979698433358952,
          0.16651905411438178,
          0.1460919010838636,
          0.1281673618772324,
          0.11243957224230806,
          0.09863987920800353,
          0.08653235338558064,
          0.07590983337382233,
          0.06659044129786831,
          0.058414514997504906,
          0.05124190829817937,
          0.04494961616520311,
          0.03942968638611423,
          0.03458738378116762,
          0.030339576843526672,
          0.026613320196039902,
          0.023344609357265626,
          0.02047728707116731,
          0.017962082906320138,
          0.015755770003045802,
          0.01382042476976509,
          0.012122777029763101,
          0.010633639620843653,
          0.009327407775066672,
          0.008181619773796962,
          0.007176571402500472,
          0.006294977636086678,
          0.005521675783377896,
          0.0048433650211909185,
          0.004248377865830787,
          0.0037264796725364254,
          0.0032686927304528897,
          0.0028671419398839716,
          0.0025149194268369226,
          0.0022059657733312955,
          0.0019349658260076341,
          0.001697257295000362,
          0.0014887505740272663,
          0.0013058584048883275,
          0.0011454341783098607,
          0.0010047178111747205,
          0.0008812882701601291,
          0.0007730219258717245,
          0.0006780560216580774,
          0.0005947566291210143,
          0.000521690539402834,
          0.0004576006069487411,
          0.0004013841217679208,
          0.0003520738382648944
         ]
        },
        {
         "mode": "lines",
         "name": "LightGBM",
         "type": "scatter",
         "x": [
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60,
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97,
          98,
          99,
          100
         ],
         "y": [
          2.863413367123862,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578,
          2.598407096979578
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Prediction of long-range shots: Logistic regression vs. lightGBM"
        },
        "xaxis": {
         "title": {
          "text": "Distance to goal line (in m)"
         }
        },
        "yaxis": {
         "title": {
          "text": "Scoring probability (in %)"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"776ff020-9b04-434d-9bce-51443df56de1\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"776ff020-9b04-434d-9bce-51443df56de1\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '776ff020-9b04-434d-9bce-51443df56de1',\n",
       "                        [{\"mode\": \"lines\", \"name\": \"Logistic Regression\", \"type\": \"scatter\", \"x\": [28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100], \"y\": [4.232267559755421, 3.7317317945428536, 3.2883600521101943, 2.896081141090486, 2.5493648568168594, 2.2431982511187116, 1.973056344967013, 1.734869504800278, 1.5249891979889152, 1.3401534184961066, 1.1774527254726397, 1.0342975574658628, 0.9083872640742103, 0.7976811260388539, 0.7003715052493849, 0.6148591701394218, 0.539730772670643, 0.4737384048009734, 0.41578113023789043, 0.36488836750779324, 0.3202049898114055, 0.280978003317481, 0.2465446665464496, 0.21632191782628601, 0.18979698433358952, 0.16651905411438178, 0.1460919010838636, 0.1281673618772324, 0.11243957224230806, 0.09863987920800353, 0.08653235338558064, 0.07590983337382233, 0.06659044129786831, 0.058414514997504906, 0.05124190829817937, 0.04494961616520311, 0.03942968638611423, 0.03458738378116762, 0.030339576843526672, 0.026613320196039902, 0.023344609357265626, 0.02047728707116731, 0.017962082906320138, 0.015755770003045802, 0.01382042476976509, 0.012122777029763101, 0.010633639620843653, 0.009327407775066672, 0.008181619773796962, 0.007176571402500472, 0.006294977636086678, 0.005521675783377896, 0.0048433650211909185, 0.004248377865830787, 0.0037264796725364254, 0.0032686927304528897, 0.0028671419398839716, 0.0025149194268369226, 0.0022059657733312955, 0.0019349658260076341, 0.001697257295000362, 0.0014887505740272663, 0.0013058584048883275, 0.0011454341783098607, 0.0010047178111747205, 0.0008812882701601291, 0.0007730219258717245, 0.0006780560216580774, 0.0005947566291210143, 0.000521690539402834, 0.0004576006069487411, 0.0004013841217679208, 0.0003520738382648944]}, {\"mode\": \"lines\", \"name\": \"LightGBM\", \"type\": \"scatter\", \"x\": [28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100], \"y\": [2.863413367123862, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578, 2.598407096979578]}],\n",
       "                        {\"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Prediction of long-range shots: Logistic regression vs. lightGBM\"}, \"xaxis\": {\"title\": {\"text\": \"Distance to goal line (in m)\"}}, \"yaxis\": {\"title\": {\"text\": \"Scoring probability (in %)\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('776ff020-9b04-434d-9bce-51443df56de1');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_sample[\"predLR\"] = model_lr.predict_proba(df_sample[features])[:,1] * 100\n",
    "df_sample[\"predLGBM\"] = model_mon2.predict_proba(df_sample[features_mon])[:,1] * 100\n",
    "\n",
    "fig = go.Figure()\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLR\"],\n",
    "                         mode='lines',\n",
    "                         name='Logistic Regression'))\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predLGBM\"],\n",
    "                         mode='lines',\n",
    "                         name='LightGBM'))\n",
    "\n",
    "fig.update_layout(title=\"Prediction of long-range shots: Logistic regression vs. lightGBM\",\n",
    "                  xaxis_title=\"Distance to goal line (in m)\",\n",
    "                  yaxis_title=\"Scoring probability (in %)\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm, which one looks more plausible? :-) \n",
    "\n",
    "You see, the problem is the following: Almost all the shots we see in the data have a probability of scoring of > 2% as they were taken somehow close to the goal. Now, all of the sudden we want to get values for situations that are pretty unplausible, i.e. outside the range of truly observed values. This, however, is a situation of extrapolation, which lightGBM isn't good at at all. Hence, it is predicting a scoring probability of ~2.3% no matter whether you are 30 meters from the goal line or 100 meters... \n",
    "\n",
    "You might argue now: \"Ok, but as I haven't seen it in reality, it does not realy matter what the algorithm predicts for shots from 100 meters.\" And yes, I agree, from a pure machine-learning, log-loss standpoint it indeed does not really matter. However, as a data scientist, in the end you will always have to sell your algorithm to some non-data scientist who is going to use it and needs to be convinced that the model works (if you like that or not ;-)). And imagine going to the coach and him asking about the scoring proability when shooting from 100 meters. Answering 2.3% is probably not the best selling story, is it? ;-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution attempt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So how do we overcome the problem above in a way that our model predicts something close to 0% when shooting from 100 meters? Unfortunately, as to my knowledge there is neither a very straight nor optimal way to deal with this. However, there are three potential routes I can think of:\n",
    "\n",
    "1. <b>Overwrite the results of the model</b><br> We just take the prediction values and hand-craft some rules to overwrite the prediction manually. One such rule might be that for all shots from > 40 m we just assume that the scoring probability is 0\n",
    "    - Advantages: Simple; predictable outcome\n",
    "    - Disadvantages: Hand-crafting rules is cumbersome; setting probability to 0 might be exagerated; hard to capture all \"outliers\", e.g. what about headers from 25 meters\n",
    "    \n",
    "    \n",
    "2. <b>Add outlier data points to the training set</b><br> As we noticed, the underlying problem is that we do not have a lot of shots with a very low probability. So we just add \"outlier\" shots to the training dataset and give them a probability of 0. This way the model can learn that e.g. shots from a long-range have a probability of 0 going in\n",
    "    - Advantages: Learning is inherit in model, rather than done afterwards; predictable outcome\n",
    "    - Disadvantages: Needs definition for \"outliers\"; might screw up the feature interpretation; setting probability to 0 might be exagerated; might need a lot of additional data points to see significant impact\n",
    "    \n",
    "    \n",
    "3. <b>Build ensemble with logistic regression</b><br> This is basically the same idea as in 1. However, instead of overwriting with a fixed value, one might set up a rule such as \"whenever the predicted scoring probability of a logistic regression model is below 2%, we use this value instead of the lightGBM prediction\"\n",
    "    - Advantages: Nice, smooth curve for outliers; takes the best out of two world\n",
    "    - Disadvantages: Requires training of two models; only makes sense if logistic regression is well-performing; combined model becomes rather complex and hard to interpret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As I have said above, unfortunately none of these solutions is optimal. Let's nevertheless quickly walk through a possible solution following the first approach. However, instead of actually hand-crafting rules I will show you a way to define outliers algorithmically and have the computer do the work for you. :-)\n",
    "\n",
    "You remember how we identified outliers in the previous notebook by checking whether in a certain grid cell less than 1% of the actions were actually shots? While this was a first way to go it has one significant drawback: As we only considered the position on the field to determine outliers, it did e.g. not matter at all whether it was a header or a shot. More precisely, while a shot with the strong foot from 31 meters might have been an outlier, a header from 29 meters was not. That does not sound too plaubsible, does it?\n",
    "\n",
    "So, let's take the body part into account when defining outliers using an outlier detecting algorithm, namely an [isolation forest](https://en.wikipedia.org/wiki/Isolation_forest). Again, I'm not going to go into the details on how this algorithms work but if you are interested in outlier detection algorithms in general, there is a nice blog post [here](https://www.analyticsvidhya.com/blog/2019/02/outlier-detection-python-pyod/) to get started. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the feature upon which outliers should be defined\n",
    "outlier_features = [\"distToGoalLine\", \"distToCenter\", \"head/body\", \"weakFoot\", \"strongFoot\"]\n",
    "\n",
    "clf = IsolationForest(random_state=42, contamination=0.02).fit(df_train[outlier_features])\n",
    "df_train[\"outlier\"] = 1*(clf.predict(df_train[outlier_features]) == -1)\n",
    "df_test[\"outlier\"] = 1*(clf.predict(df_test[outlier_features]) == -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check what the algorithm detected as an outlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>distToGoalLine</th>\n",
       "      <th>distToCenter</th>\n",
       "      <th>head/body</th>\n",
       "      <th>weakFoot</th>\n",
       "      <th>strongFoot</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11.55</td>\n",
       "      <td>17.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.037816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>23.10</td>\n",
       "      <td>30.60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.019681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221</th>\n",
       "      <td>2.10</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.586270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246</th>\n",
       "      <td>22.05</td>\n",
       "      <td>21.76</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.020947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357</th>\n",
       "      <td>30.45</td>\n",
       "      <td>1.36</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.023562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>93.45</td>\n",
       "      <td>10.20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.022568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>475</th>\n",
       "      <td>3.15</td>\n",
       "      <td>10.88</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.129321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>529</th>\n",
       "      <td>48.30</td>\n",
       "      <td>2.72</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.023139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>612</th>\n",
       "      <td>47.25</td>\n",
       "      <td>19.04</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.015089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>648</th>\n",
       "      <td>12.60</td>\n",
       "      <td>14.28</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.046865</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     distToGoalLine  distToCenter  head/body  weakFoot  strongFoot  prediction\n",
       "10            11.55         17.00          1         0           0    0.037816\n",
       "143           23.10         30.60          0         0           1    0.019681\n",
       "221            2.10          0.00          1         0           0    0.586270\n",
       "246           22.05         21.76          0         1           0    0.020947\n",
       "357           30.45          1.36          0         1           0    0.023562\n",
       "381           93.45         10.20          0         0           1    0.022568\n",
       "475            3.15         10.88          0         1           0    0.129321\n",
       "529           48.30          2.72          0         0           1    0.023139\n",
       "612           47.25         19.04          0         0           1    0.015089\n",
       "648           12.60         14.28          1         0           0    0.046865"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train[df_train[\"outlier\"] == 1][outlier_features + [\"prediction\"]].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, the first one was a header from 11.55 meters but very far out. I can see that this is an outlier. Even more so with the second example, a shot from 23 meters that was right next to the out line (remember that the out line is at 34 meters as the width of the field is 68 meters). But what about the third one? A header from 2.10 meters right in front of the goal?! \n",
    "\n",
    "Unfortunately, the algorithm is right in the sense that this is indeed a very unusual situation and can therefore seen as an outlier. We, however, do obviously not want to manually override the scoring probability to 0 :-) \n",
    "\n",
    "So let's define our outlier in a slightly more complex way: A data point which is detected by the isolation forest as outlier and for which the predicted shooting probability is low, e.g. < 3%. This way we can ensure that we indeed only overwrite outliers that would have had a low scoring probability anyway. \n",
    "\n",
    "The complete model would therefore look the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_proba_final(df):\n",
    "    X = df[features_mon].copy()\n",
    "    X[\"prediction\"] = model_mon2.predict_proba(X)[:,1]\n",
    "    X[\"outlier\"] = 1*(clf.predict(X[outlier_features]) == -1)\n",
    "    return np.where((X[\"outlier\"] == 1) & (X[\"prediction\"] < 0.03), 0, X[\"prediction\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's the how the prediction look like for different distances when shooting with the strong foot. We compare the results of our final model where we deal with both monotonicity and extrapolation to the original model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Original model",
         "type": "scatter",
         "x": [
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60,
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97,
          98,
          99,
          100
         ],
         "y": [
          0.2050547324279167,
          0.2050547324279167,
          0.20024761453610718,
          0.1709703746427905,
          0.15527225947064247,
          0.13172554172610365,
          0.10216933485548974,
          0.09532184349300894,
          0.0971481449220999,
          0.08979001605504533,
          0.05254330787119877,
          0.05076121927186307,
          0.029724252787151174,
          0.029724252787151174,
          0.029724252787151174,
          0.02551621201921541,
          0.02638757501093832,
          0.02638757501093832,
          0.026597235600622215,
          0.023974456176053906,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213,
          0.023365478057238213
         ]
        },
        {
         "mode": "lines",
         "name": "Monotone model considering outliers",
         "type": "scatter",
         "x": [
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60,
          61,
          62,
          63,
          64,
          65,
          66,
          67,
          68,
          69,
          70,
          71,
          72,
          73,
          74,
          75,
          76,
          77,
          78,
          79,
          80,
          81,
          82,
          83,
          84,
          85,
          86,
          87,
          88,
          89,
          90,
          91,
          92,
          93,
          94,
          95,
          96,
          97,
          98,
          99,
          100
         ],
         "y": [
          0.20877914483012533,
          0.20877914483012533,
          0.19870816506908717,
          0.1647830872890409,
          0.1435313044653683,
          0.12396551086015305,
          0.09458976718444206,
          0.08647707287383315,
          0.08647707287383315,
          0.08084778702684105,
          0.057474482176602656,
          0.05518980446162107,
          0.03904576772702051,
          0.03904576772702051,
          0.03783839556085331,
          0.029125322466419697,
          0.02868149435640384,
          0.02863413367123862,
          0.02863413367123862,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0.02598407096979578,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Scoring probability for original model and updated, monotone model"
        },
        "xaxis": {
         "title": {
          "text": "Distance to goal line (in m)"
         }
        },
        "yaxis": {
         "title": {
          "text": "Scoring probability (in %)"
         }
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"d209cde1-296d-4a0d-a9bf-32c3158507b5\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"d209cde1-296d-4a0d-a9bf-32c3158507b5\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        'd209cde1-296d-4a0d-a9bf-32c3158507b5',\n",
       "                        [{\"mode\": \"lines\", \"name\": \"Original model\", \"type\": \"scatter\", \"x\": [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100], \"y\": [0.2050547324279167, 0.2050547324279167, 0.20024761453610718, 0.1709703746427905, 0.15527225947064247, 0.13172554172610365, 0.10216933485548974, 0.09532184349300894, 0.0971481449220999, 0.08979001605504533, 0.05254330787119877, 0.05076121927186307, 0.029724252787151174, 0.029724252787151174, 0.029724252787151174, 0.02551621201921541, 0.02638757501093832, 0.02638757501093832, 0.026597235600622215, 0.023974456176053906, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213, 0.023365478057238213]}, {\"mode\": \"lines\", \"name\": \"Monotone model considering outliers\", \"type\": \"scatter\", \"x\": [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100], \"y\": [0.20877914483012533, 0.20877914483012533, 0.19870816506908717, 0.1647830872890409, 0.1435313044653683, 0.12396551086015305, 0.09458976718444206, 0.08647707287383315, 0.08647707287383315, 0.08084778702684105, 0.057474482176602656, 0.05518980446162107, 0.03904576772702051, 0.03904576772702051, 0.03783839556085331, 0.029125322466419697, 0.02868149435640384, 0.02863413367123862, 0.02863413367123862, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.02598407096979578, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}],\n",
       "                        {\"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Scoring probability for original model and updated, monotone model\"}, \"xaxis\": {\"title\": {\"text\": \"Distance to goal line (in m)\"}}, \"yaxis\": {\"title\": {\"text\": \"Scoring probability (in %)\"}}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('d209cde1-296d-4a0d-a9bf-32c3158507b5');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create a number of different shots with varying distance\n",
    "dist_goal_line = np.arange(10,101)\n",
    "dist_center = 0\n",
    "counter_attack = 0\n",
    "duel = 0\n",
    "corner = 0\n",
    "smart_pass = 0\n",
    "shot_before = 0\n",
    "body_part = \"strongFoot\"\n",
    "\n",
    "df_sample = create_sample_shots(dist_goal_line, dist_center, counter_attack, duel, corner, smart_pass, \n",
    "                                shot_before, body_part)\n",
    "# add the angle to the sample data\n",
    "df_sample = egm.create_features(df_sample, overwrite=False)\n",
    "\n",
    "# predict the scoring probability with both our final model and the original model\n",
    "df_sample[\"predFinal\"] = predict_proba_final(df_sample)\n",
    "df_sample[\"predOriginal\"] = model.predict_proba(df_sample[features])[:,1]\n",
    "\n",
    "# create a plotly graph\n",
    "fig = go.Figure()\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predOriginal\"],\n",
    "                         mode='lines',\n",
    "                         name='Original model'))\n",
    "\n",
    "fig.add_trace(go.Scatter(x=df_sample[\"distToGoalLine\"], \n",
    "                         y=df_sample[\"predFinal\"],\n",
    "                         mode='lines',\n",
    "                         name='Monotone model considering outliers'))\n",
    "\n",
    "fig.update_layout(title=\"Scoring probability for original model and updated, monotone model\",\n",
    "                  xaxis_title=\"Distance to goal line (in m)\",\n",
    "                  yaxis_title=\"Scoring probability (in %)\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this might still not be completely convincing, it does look way better now than it did before, doesn't it? :-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we learned above, lightGBM is not capable of extrapolation. There are some strategies out there how to deal with this issue, which unfortunately so, come with some cost such as increased complexity, reduced interpretability or additional work. To summarize the above, I would recommend you the following:\n",
    "\n",
    "1. If you decide to use lightGBM or XGBoost, you should actively think about how to deal with extrapolation. These algorithms unfortunately will not do it for you ;-) \n",
    "\n",
    "2. However, in case you need an algorithm that extrapolates well to new data, lightGBM and XGBoost might not be the best choice for you"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}