fastprop_benchmark/air_pollution_prop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "hide_input"
    ]
   },
   "source": [
    "# Propositionalization: Predicting air pollution in Beijing\n",
    "\n",
    "In this notebook we will compare getML to featuretools and tsfresh, both of which open-source libraries for feature engineering. We find that advanced algorithms featured in getML yield significantly better predictions on this dataset. We then discuss why that is.\n",
    "\n",
    "Summary:\n",
    "\n",
    "- Prediction type: __Regression model__\n",
    "- Domain: __Air pollution__\n",
    "- Prediction target: __pm 2.5 concentration__ \n",
    "- Source data: __Multivariate time series__\n",
    "- Population size: __41757__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Background\n",
    "\n",
    "A common approach to feature engineering is to generate attribute-value representations from relational data by applying a fixed set of aggregations to columns of interest and perform a feature selection on the (possibly large) set of generated features afterwards. In academia, this approach is called _propositionalization._\n",
    "\n",
    "getML's [FastProp](https://docs.getml.com/latest/user_guide/feature_engineering/feature_engineering.html#fastprop) is an implementation of this propositionalization approach that has been optimized for speed and memory efficiency. In this notebook, we want to demonstrate how – well – fast FastProp is. To this end, we will benchmark FastProp against the popular feature engineering libraries [featuretools](https://www.featuretools.com/) and [tsfresh](https://tsfresh.readthedocs.io/en/latest/). Both of these libraries use propositionalization approaches for feature engineering.\n",
    "\n",
    "As our example dataset, we use a publicly available dataset on air pollution in Beijing, China (https://archive.ics.uci.edu/dataset/381/beijing+pm2+5+data). For further details about the data set refer to [the full notebook](../air_pollution.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "\n",
    "1. [Loading data](#1.-Loading-data)\n",
    "2. [Predictive modeling](#2.-Predictive-modeling)\n",
    "3. [Comparison](#3.-Comparison)\n",
    "4. [Conclusion](#4.-Conclusion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import os\n",
    "os.environ[\"PYARROW_IGNORE_TIMEZONE\"] = \"1\"\n",
    "from pathlib import Path\n",
    "\n",
    "import sys\n",
    "import time\n",
    "from urllib import request\n",
    "\n",
    "import getml\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.stats import pearsonr\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "parent = Path(os.getcwd()).parent.as_posix()\n",
    "\n",
    "if parent not in sys.path:\n",
    "   sys.path.append(parent) \n",
    "\n",
    "from utils import Benchmark, FTTimeSeriesBuilder, TSFreshBuilder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Download from source\n",
    "\n",
    "We begin by downloading the data from the UCI Machine Learning repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "FEATURETOOLS_FILES = [\"featuretools_training.csv\", \"featuretools_test.csv\"]\n",
    "\n",
    "for fname in FEATURETOOLS_FILES:\n",
    "    if not os.path.exists(fname):\n",
    "        fname, res = request.urlretrieve(\n",
    "            \"https://static.getml.com/datasets/air_pollution/featuretools/\" + fname,\n",
    "            fname,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "TSFRESH_FILES = [\"tsfresh_training.csv\", \"tsfresh_test.csv\"]\n",
    "\n",
    "for fname in TSFRESH_FILES:\n",
    "    if not os.path.exists(fname):\n",
    "        fname, res = request.urlretrieve(\n",
    "            \"https://static.getml.com/datasets/air_pollution/tsfresh/\" + fname, fname\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = \"PRSA_data_2010.1.1-2014.12.31.csv\"\n",
    "\n",
    "if not os.path.exists(fname):\n",
    "    fname, res = request.urlretrieve(\n",
    "        \"https://archive.ics.uci.edu/ml/machine-learning-databases/00381/\" + fname,\n",
    "        fname,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Prepare data for tsfresh and getML\n",
    "\n",
    "Our our goal is to predict the pm2.5 concentration from factors such as weather or time of day. However, there are some **missing entries** for pm2.5, so we get rid of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_full_pandas = pd.read_csv(fname)\n",
    "\n",
    "data_full_pandas = data_full_pandas[\n",
    "    data_full_pandas[\"pm2.5\"] == data_full_pandas[\"pm2.5\"]\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tsfresh requires a date column, so we build one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_leading_zero(val):\n",
    "    if len(str(val)) == 1:\n",
    "        return \"0\" + str(val)\n",
    "    return str(val)\n",
    "\n",
    "\n",
    "data_full_pandas[\"month\"] = [add_leading_zero(val) for val in data_full_pandas[\"month\"]]\n",
    "\n",
    "data_full_pandas[\"day\"] = [add_leading_zero(val) for val in data_full_pandas[\"day\"]]\n",
    "\n",
    "data_full_pandas[\"hour\"] = [add_leading_zero(val) for val in data_full_pandas[\"hour\"]]\n",
    "\n",
    "\n",
    "def make_date(year, month, day, hour):\n",
    "    return year + \"-\" + month + \"-\" + day + \" \" + hour + \":00:00\"\n",
    "\n",
    "\n",
    "data_full_pandas[\"date\"] = [\n",
    "    make_date(str(year), month, day, hour)\n",
    "    for year, month, day, hour in zip(\n",
    "        data_full_pandas[\"year\"],\n",
    "        data_full_pandas[\"month\"],\n",
    "        data_full_pandas[\"day\"],\n",
    "        data_full_pandas[\"hour\"],\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tsfresh also requires the time series to have ids. Since there is only a single time series, that series has the same id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_full_pandas[\"id\"] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset now contains many columns that we do not need or that tsfresh cannot process. For instance, *cbwd* might actually contain useful information, but it is a categorical variable, which is difficult to handle for tsfresh, so we remove it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want to split our data into a training and testing set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train_pandas = data_full_pandas[data_full_pandas[\"year\"] < 2014]\n",
    "data_test_pandas = data_full_pandas[data_full_pandas[\"year\"] == 2014]\n",
    "data_full_pandas = data_full_pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_unwanted_columns(df):\n",
    "    del df[\"cbwd\"]\n",
    "    del df[\"year\"]\n",
    "    del df[\"month\"]\n",
    "    del df[\"day\"]\n",
    "    del df[\"hour\"]\n",
    "    del df[\"No\"]\n",
    "    return df\n",
    "\n",
    "\n",
    "data_full_pandas = remove_unwanted_columns(data_full_pandas)\n",
    "data_train_pandas = remove_unwanted_columns(data_train_pandas)\n",
    "data_test_pandas = remove_unwanted_columns(data_test_pandas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pm2.5</th>\n",
       "      <th>DEWP</th>\n",
       "      <th>TEMP</th>\n",
       "      <th>PRES</th>\n",
       "      <th>Iws</th>\n",
       "      <th>Is</th>\n",
       "      <th>Ir</th>\n",
       "      <th>date</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>129.0</td>\n",
       "      <td>-16</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>1020.0</td>\n",
       "      <td>1.79</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 00:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>148.0</td>\n",
       "      <td>-15</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>1020.0</td>\n",
       "      <td>2.68</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 01:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>159.0</td>\n",
       "      <td>-11</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>1021.0</td>\n",
       "      <td>3.57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 02:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>181.0</td>\n",
       "      <td>-7</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>1022.0</td>\n",
       "      <td>5.36</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 03:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>138.0</td>\n",
       "      <td>-7</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>1022.0</td>\n",
       "      <td>6.25</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 04:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43819</th>\n",
       "      <td>8.0</td>\n",
       "      <td>-23</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>1034.0</td>\n",
       "      <td>231.97</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-12-31 19:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43820</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-22</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>1034.0</td>\n",
       "      <td>237.78</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-12-31 20:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43821</th>\n",
       "      <td>10.0</td>\n",
       "      <td>-22</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>1034.0</td>\n",
       "      <td>242.70</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-12-31 21:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43822</th>\n",
       "      <td>8.0</td>\n",
       "      <td>-22</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>1034.0</td>\n",
       "      <td>246.72</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-12-31 22:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43823</th>\n",
       "      <td>12.0</td>\n",
       "      <td>-21</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>1034.0</td>\n",
       "      <td>249.85</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-12-31 23:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>41757 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       pm2.5  DEWP  TEMP    PRES     Iws  Is  Ir                 date  id\n",
       "24     129.0   -16  -4.0  1020.0    1.79   0   0  2010-01-02 00:00:00   1\n",
       "25     148.0   -15  -4.0  1020.0    2.68   0   0  2010-01-02 01:00:00   1\n",
       "26     159.0   -11  -5.0  1021.0    3.57   0   0  2010-01-02 02:00:00   1\n",
       "27     181.0    -7  -5.0  1022.0    5.36   1   0  2010-01-02 03:00:00   1\n",
       "28     138.0    -7  -5.0  1022.0    6.25   2   0  2010-01-02 04:00:00   1\n",
       "...      ...   ...   ...     ...     ...  ..  ..                  ...  ..\n",
       "43819    8.0   -23  -2.0  1034.0  231.97   0   0  2014-12-31 19:00:00   1\n",
       "43820   10.0   -22  -3.0  1034.0  237.78   0   0  2014-12-31 20:00:00   1\n",
       "43821   10.0   -22  -3.0  1034.0  242.70   0   0  2014-12-31 21:00:00   1\n",
       "43822    8.0   -22  -4.0  1034.0  246.72   0   0  2014-12-31 22:00:00   1\n",
       "43823   12.0   -21  -3.0  1034.0  249.85   0   0  2014-12-31 23:00:00   1\n",
       "\n",
       "[41757 rows x 9 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_full_pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then **load the data into the getML engine**. We begin by setting a project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getML engine is already running.\n",
      "Loading pipelines... 100% |██████████| [elapsed: 00:01, remaining: 00:00]          \n",
      "\n",
      "Connected to project 'air_pollution'\n"
     ]
    }
   ],
   "source": [
    "getml.engine.launch(home_directory=Path.home(), allow_remote_ips=True, token='token')\n",
    "getml.engine.set_project(\"air_pollution\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_full = getml.data.DataFrame.from_pandas(data_full_pandas, name=\"full\")\n",
    "df_full[\"date\"] = df_full[\"date\"].as_ts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to **assign roles** to the columns, such as defining the target column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "  table {\n",
       "    font-family: Helvetica, sans-serif;\n",
       "  }\n",
       "\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right !important;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.sub-header {\n",
       "    font-weight: normal;\n",
       "    font-style: italic;\n",
       "  }\n",
       "  .join_key,\n",
       "  .numerical,\n",
       "  .target,\n",
       "  .unused_float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "\n",
       "  .char-align {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  span.left {\n",
       "    text-align: right;\n",
       "    width: 3em;\n",
       "  }\n",
       "  span.right {\n",
       "    float: right;\n",
       "    text-align: left;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th> name</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"time_stamp\">                       date</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"target\"> pm2.5</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"numerical\">     DEWP</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"numerical\">     TEMP</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"numerical\">     PRES</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"numerical\">      Iws</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"numerical\">       Is</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"numerical\">       Ir</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"unused_float\">          id</th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "      <tr>\n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header\"> role</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header time_stamp\">                 time_stamp</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header target\">target</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">numerical</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">numerical</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">numerical</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">numerical</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">numerical</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">numerical</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header unused_float\">unused_float</th>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header\"> unit</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header time_stamp\">time stamp, comparison only</th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header target\">      </th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">         </th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">         </th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">         </th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">         </th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">         </th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header numerical\">         </th>\n",
       "          \n",
       "        \n",
       "          \n",
       "            <th class=\"sub-header unused_float\">            </th>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2010-01-02</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">129</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-16</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-4</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1020</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.79</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>1</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2010-01-02 01:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">148</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-15</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-4</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1020</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">2</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.68</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>2</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2010-01-02 02:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">159</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-11</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-5</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1021</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">3</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.57</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>3</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2010-01-02 03:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">181</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-7</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-5</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1022</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">5</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.36</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>4</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2010-01-02 04:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">138</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-7</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-5</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1022</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">6</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.25</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">2</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th></th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">...</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">...</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>41752</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2014-12-31 19:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">8</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-23</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-2</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1034</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">231</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.97</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>41753</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2014-12-31 20:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">10</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-22</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-3</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1034</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">237</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.78</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>41754</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2014-12-31 21:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">10</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-22</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-3</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1034</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">242</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.7</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>41755</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2014-12-31 22:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">8</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-22</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-4</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1034</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">246</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.72</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>41756</th>\n",
       "        \n",
       "          \n",
       "            <td class=\"time_stamp\">2014-12-31 23:00:00</td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align target\">\n",
       "              <span class=\"left\">12</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-21</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">-3</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">1034</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "            \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">249</span\n",
       "              ><span class=\"right\" style=\"width: 2ch\"\n",
       "                >.85</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align numerical\">\n",
       "              <span class=\"left\">0</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "          \n",
       "            \n",
       "              \n",
       "              \n",
       "          \n",
       "            <td class=\"char-align unused_float\">\n",
       "              <span class=\"left\">1</span\n",
       "              ><span class=\"right\" style=\"width: 0ch\"\n",
       "                >&nbsp;</span\n",
       "              >\n",
       "            </td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>\n",
       "\n",
       "  <p>\n",
       "    41757 rows x 9 columns<br />\n",
       "    memory usage: 3.01 MB<br />\n",
       "    name: full<br />\n",
       "    type: getml.DataFrame<br />\n",
       "    \n",
       "  </p>\n"
      ],
      "text/plain": [
       " name                          date    pm2.5        DEWP        TEMP        PRES         Iws          Is          Ir\n",
       " role                    time_stamp   target   numerical   numerical   numerical   numerical   numerical   numerical\n",
       " unit   time stamp, comparison only                                                                                 \n",
       "    0                    2010-01-02      129        -16          -4         1020        1.79           0           0\n",
       "    1           2010-01-02 01:00:00      148        -15          -4         1020        2.68           0           0\n",
       "    2           2010-01-02 02:00:00      159        -11          -5         1021        3.57           0           0\n",
       "    3           2010-01-02 03:00:00      181         -7          -5         1022        5.36           1           0\n",
       "    4           2010-01-02 04:00:00      138         -7          -5         1022        6.25           2           0\n",
       "                                ...      ...         ...         ...         ...       ...           ...         ...\n",
       "41752           2014-12-31 19:00:00        8        -23          -2         1034      231.97           0           0\n",
       "41753           2014-12-31 20:00:00       10        -22          -3         1034      237.78           0           0\n",
       "41754           2014-12-31 21:00:00       10        -22          -3         1034      242.7            0           0\n",
       "41755           2014-12-31 22:00:00        8        -22          -4         1034      246.72           0           0\n",
       "41756           2014-12-31 23:00:00       12        -21          -3         1034      249.85           0           0\n",
       "\n",
       " name             id\n",
       " role   unused_float\n",
       " unit               \n",
       "    0              1\n",
       "    1              1\n",
       "    2              1\n",
       "    3              1\n",
       "    4              1\n",
       "                 ...\n",
       "41752              1\n",
       "41753              1\n",
       "41754              1\n",
       "41755              1\n",
       "41756              1\n",
       "\n",
       "\n",
       "41757 rows x 9 columns\n",
       "memory usage: 3.01 MB\n",
       "type: getml.DataFrame"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_full.set_role([\"date\"], getml.data.roles.time_stamp)\n",
    "df_full.set_role([\"pm2.5\"], getml.data.roles.target)\n",
    "df_full.set_role(\n",
    "    [\"DEWP\", \"TEMP\", \"PRES\", \"Iws\", \"Is\", \"Ir\"], getml.data.roles.numerical\n",
    ")\n",
    "df_full"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th.sub-header {\n",
       "    font-weight: normal;\n",
       "    font-style: italic;\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right !important;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.numerical {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.numerical {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "\n",
       "  td.char-align {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  span.left {\n",
       "    text-align: right;\n",
       "    width: 3em;\n",
       "  }\n",
       "  span.right {\n",
       "    float: right;\n",
       "    text-align: left;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th>     </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th>     </th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "        \n",
       "          \n",
       "            <td>train</td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>1</th>\n",
       "        \n",
       "          \n",
       "            <td>train</td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>2</th>\n",
       "        \n",
       "          \n",
       "            <td>train</td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>3</th>\n",
       "        \n",
       "          \n",
       "            <td>train</td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>4</th>\n",
       "        \n",
       "          \n",
       "            <td>train</td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th></th>\n",
       "        \n",
       "          \n",
       "            <td>...</td>\n",
       "          \n",
       "        \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>\n",
       "\n",
       "  <p>\n",
       "    41757 rows<br />\n",
       "    \n",
       "    type: StringColumnView<br />\n",
       "    \n",
       "  </p>\n"
      ],
      "text/plain": [
       "             \n",
       "    0   train\n",
       "    1   train\n",
       "    2   train\n",
       "    3   train\n",
       "    4   train\n",
       "        ...  \n",
       "\n",
       "\n",
       "41757 rows\n",
       "type: StringColumnView"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "split = getml.data.split.time(\n",
    "    population=df_full, time_stamp=\"date\", test=getml.data.time.datetime(2014, 1, 1)\n",
    ")\n",
    "split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Predictive modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Propositionalization with getML's FastProp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style='font-size: 1.2rem; font-weight: 500;'>data model</span>\n",
       "            <div style='margin-top: 15px; margin-bottom: 5px;'>\n",
       "            <div style='margin-bottom: 10px; font-size: 1rem;'>diagram</div>\n",
       "            <div style=\"height:100px;width:660px;position:relative;\"><svg height=\"90\" width=\"650\"><rect y=\"0\" x=\"0\" rx=\"10\" ry=\"10\" width=\"150\" height=\"90\" style=\"fill:#6829c2;stroke-width:0;\" /><text y=\"73.8\" x=\"75.0\" dominant-baseline=\"middle\" text-anchor=\"middle\" fill=\"white\">full</text><rect x=\"51\" y=\"10\" rx=\"4\" ry=\"4\" width=\"48\" height=\"48\" style=\" fill:#6829c2;stroke:#ffffff;stroke-width:3;\" /><line x1=\"67.0\" y1=\"10\" x2=\"67.0\" y2=\"58\" style=\"stroke:white;stroke-width:3\" /><line x1=\"83.0\" y1=\"10\" x2=\"83.0\" y2=\"58\" style=\"stroke:white;stroke-width:3\" /><line x1=\"51\" y1=\"26.0\" x2=\"99\" y2=\"26.0\" style=\"stroke:white;stroke-width:3\" /><line x1=\"51\" y1=\"42.0\" x2=\"99\" y2=\"42.0\" style=\"stroke:white;stroke-width:3\" /><rect y=\"0\" x=\"500\" rx=\"10\" ry=\"10\" width=\"150\" height=\"90\" style=\"fill:#6829c2;stroke-width:0;\" /><text y=\"73.8\" x=\"575.0\" dominant-baseline=\"middle\" text-anchor=\"middle\" fill=\"white\">population</text><rect x=\"551\" y=\"10\" rx=\"4\" ry=\"4\" width=\"48\" height=\"48\" style=\" fill:#6829c2;stroke:#ffffff;stroke-width:3;\" /><line x1=\"567.0\" y1=\"10\" x2=\"567.0\" y2=\"58\" style=\"stroke:white;stroke-width:3\" /><line x1=\"583.0\" y1=\"10\" x2=\"583.0\" y2=\"58\" style=\"stroke:white;stroke-width:3\" /><line x1=\"551\" y1=\"26.0\" x2=\"599\" y2=\"26.0\" style=\"stroke:white;stroke-width:3\" /><line x1=\"551\" y1=\"42.0\" x2=\"599\" y2=\"42.0\" style=\"stroke:white;stroke-width:3\" /><line x1=\"150\" y1=\"43.0\" x2=\"490\" y2=\"43.0\" style=\"stroke:#808080;;stroke-width:4\" /><polygon points=\"500, 43.0 490, 37.0 490, 49.0 \" style=\"fill:#808080;;stroke-width:0;\" /><rect y=\"10.0\" x=\"249.0\" rx=\"10\" ry=\"10\" width=\"150\" height=\"70\" style=\"fill:#6829c2;stroke-width:0;\" /><text dominant-baseline=\"middle\" text-anchor=\"middle\" fill=\"white\"><tspan y=\"40.0\" x=\"324.0\" font-size=\"7pt\" >date &lt;= date</tspan><tspan y=\"50.0\" x=\"324.0\" font-size=\"7pt\" >Memory: 1.0 days</tspan></text></svg></div>\n",
       "            </div>\n",
       "\n",
       "            <div style='margin-top: 15px;'>\n",
       "            <div style='margin-bottom: 10px; font-size: 1rem;'>staging</div>\n",
       "            <style>\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  th.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">data frames</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">staging table              </th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"str\">population</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">POPULATION__STAGING_TABLE_1</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>1</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"str\">full</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">FULL__STAGING_TABLE_2</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>\n",
       "            </div>\n",
       "            \n",
       "<span style='font-size: 1.2rem; font-weight: 500;'>container</span>\n",
       "<div style='margin-top: 15px;'>\n",
       "<div style='float: left; margin-right: 50px;'>\n",
       "<div style='margin-bottom: 10px; font-size: 1rem;'>population</div>\n",
       "    <style>\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  th.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">subset</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">name</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> rows</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">type</th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"str\">test</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">full</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"int\">8661</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">View</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>1</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"str\">train</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">full</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"int\">33096</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">View</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "<div style='float: left;'>\n",
       "<div style='margin-bottom: 10px; font-size: 1rem;'>peripheral</div>\n",
       "    <style>\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  th.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">name</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> rows</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">type     </th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"str\">full</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"int\">41757</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">DataFrame</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "</div>"
      ],
      "text/plain": [
       "data model\n",
       "\n",
       "  population:\n",
       "    columns:\n",
       "    - DEWP: numerical\n",
       "    - TEMP: numerical\n",
       "    - PRES: numerical\n",
       "    - Iws: numerical\n",
       "    - Is: numerical\n",
       "    - ...\n",
       "\n",
       "    joins:\n",
       "    - right: 'full'\n",
       "      time_stamps: (population.date, full.date)\n",
       "      relationship: 'many-to-many'\n",
       "      memory: 86400.0\n",
       "      lagged_targets: False\n",
       "\n",
       "  full:\n",
       "    columns:\n",
       "    - DEWP: numerical\n",
       "    - TEMP: numerical\n",
       "    - PRES: numerical\n",
       "    - Iws: numerical\n",
       "    - Is: numerical\n",
       "    - ...\n",
       "\n",
       "\n",
       "container\n",
       "\n",
       "  population\n",
       "      subset   name    rows   type\n",
       "  0   test     full    8661   View\n",
       "  1   train    full   33096   View\n",
       "\n",
       "  peripheral\n",
       "      name    rows   type     \n",
       "  0   full   41757   DataFrame"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_series = getml.data.TimeSeries(\n",
    "    population=df_full,\n",
    "    alias=\"population\",\n",
    "    split=split,\n",
    "    time_stamps=\"date\",\n",
    "    memory=getml.data.time.days(1),\n",
    ")\n",
    "\n",
    "time_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Pipeline(data_model='population',\n",
       "         feature_learners=['FastProp'],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=['full'],\n",
       "         predictors=[],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['memory: 1d', 'simple features'])</pre>"
      ],
      "text/plain": [
       "Pipeline(data_model='population',\n",
       "         feature_learners=['FastProp'],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=['full'],\n",
       "         predictors=[],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['memory: 1d', 'simple features'])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fast_prop = getml.feature_learning.FastProp(\n",
    "    loss_function=getml.feature_learning.loss_functions.SquareLoss,\n",
    "    num_threads=1,\n",
    "    aggregation=getml.feature_learning.FastProp.agg_sets.All,\n",
    ")\n",
    "\n",
    "pipe_fp_fl = getml.pipeline.Pipeline(\n",
    "    tags=[\"memory: 1d\", \"simple features\"],\n",
    "    data_model=time_series.data_model,\n",
    "    feature_learners=[fast_prop],\n",
    ")\n",
    "\n",
    "pipe_fp_fl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking data model...\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Checking... 100% |██████████| [elapsed: 00:02, remaining: 00:00]          \n",
      "\n",
      "OK.\n"
     ]
    }
   ],
   "source": [
    "pipe_fp_fl.check(time_series.train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "benchmark = Benchmark()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking data model...\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n",
      "OK.\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "FastProp: Trying 331 features... 100% |██████████| [elapsed: 00:05, remaining: 00:00]          \n",
      "\n",
      "Trained pipeline.\n",
      "Time taken: 0h:0m:5.370066\n",
      "\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "FastProp: Building features... 100% |██████████| [elapsed: 00:03, remaining: 00:00]          \n",
      "\n"
     ]
    }
   ],
   "source": [
    "with benchmark(\"fastprop\"):\n",
    "    pipe_fp_fl.fit(time_series.train)\n",
    "    fastprop_train = pipe_fp_fl.transform(time_series.train, df_name=\"fastprop_train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "FastProp: Building features... 100% |██████████| [elapsed: 00:01, remaining: 00:00]          \n",
      "\n"
     ]
    }
   ],
   "source": [
    "fastprop_test = pipe_fp_fl.transform(time_series.test, df_name=\"fastprop_test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = getml.predictors.XGBoostRegressor()\n",
    "\n",
    "pipe_fp_pr = getml.pipeline.Pipeline(\n",
    "    tags=[\"prediction\", \"fastprop\"], predictors=[predictor]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking data model...\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Checking... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n",
      "OK.\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "XGBoost: Training as predictor... 100% |██████████| [elapsed: 00:16, remaining: 00:00]          \n",
      "\n",
      "Trained pipeline.\n",
      "Time taken: 0h:0m:16.267519\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre>Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['prediction', 'fastprop'])</pre>"
      ],
      "text/plain": [
       "Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['prediction', 'fastprop'])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_fp_pr.fit(fastprop_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  th.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"datetime\">date time          </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">set used      </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">target</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">     mae</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">    rmse</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">rsquared</th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"datetime\">2024-02-21 16:13:14</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">fastprop_train</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">pm2.5</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">38.3028</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">55.2472</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">0.6438</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>1</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"datetime\">2024-02-21 16:13:14</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">fastprop_test</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">pm2.5</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">44.2526</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">63.4191</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">0.5462</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "    date time             set used         target        mae       rmse   rsquared\n",
       "0   2024-02-21 16:13:14   fastprop_train   pm2.5     38.3028    55.2472     0.6438\n",
       "1   2024-02-21 16:13:14   fastprop_test    pm2.5     44.2526    63.4191     0.5462"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_fp_pr.score(fastprop_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Using featuretools\n",
    "\n",
    "To make things a bit easier, we have written a high-level wrapper around featuretools which we placed in a separate module (`utils`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft_builder = FTTimeSeriesBuilder(\n",
    "    num_features=200,\n",
    "    horizon=pd.Timedelta(days=0),\n",
    "    memory=pd.Timedelta(days=1),\n",
    "    column_id=\"id\",\n",
    "    time_stamp=\"date\",\n",
    "    target=\"pm2.5\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "featuretools: Trying features...\n",
      "Selecting the best out of 298 features...\n",
      "Time taken: 0h:34m:1.838204\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with benchmark(\"featuretools\"):\n",
    "    featuretools_training = ft_builder.fit(data_train_pandas)\n",
    "\n",
    "featuretools_test = ft_builder.transform(data_test_pandas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_featuretools_training = getml.data.DataFrame.from_pandas(\n",
    "    featuretools_training, name=\"featuretools_training\"\n",
    ")\n",
    "df_featuretools_test = getml.data.DataFrame.from_pandas(\n",
    "    featuretools_test, name=\"featuretools_test\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_roles_featuretools(df):\n",
    "    df[\"date\"] = df[\"date\"].as_ts()\n",
    "    df.set_role([\"pm2.5\"], getml.data.roles.target)\n",
    "    df.set_role([\"date\"], getml.data.roles.time_stamp)\n",
    "    df.set_role(df.roles.unused, getml.data.roles.numerical)\n",
    "    df.set_role([\"id\"], getml.data.roles.unused_float)\n",
    "    return df\n",
    "\n",
    "\n",
    "df_featuretools_training = set_roles_featuretools(df_featuretools_training)\n",
    "df_featuretools_test = set_roles_featuretools(df_featuretools_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['featuretools', 'memory: 1d'])</pre>"
      ],
      "text/plain": [
       "Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['featuretools', 'memory: 1d'])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor = getml.predictors.XGBoostRegressor()\n",
    "\n",
    "pipe_ft_pr = getml.pipeline.Pipeline(\n",
    "    tags=[\"featuretools\", \"memory: 1d\"], predictors=[predictor]\n",
    ")\n",
    "\n",
    "pipe_ft_pr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking data model...\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Checking... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n",
      "OK.\n"
     ]
    }
   ],
   "source": [
    "pipe_ft_pr.check(df_featuretools_training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking data model...\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n",
      "OK.\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "XGBoost: Training as predictor... 100% |██████████| [elapsed: 00:10, remaining: 00:00]          \n",
      "\n",
      "Trained pipeline.\n",
      "Time taken: 0h:0m:10.460135\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre>Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['featuretools', 'memory: 1d'])</pre>"
      ],
      "text/plain": [
       "Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['featuretools', 'memory: 1d'])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_ft_pr.fit(df_featuretools_training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  th.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"datetime\">date time          </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">set used             </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">target</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">     mae</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">    rmse</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">rsquared</th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"datetime\">2024-02-21 16:56:07</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">featuretools_training</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">pm2.5</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">38.277</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">54.8781</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">0.6506</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>1</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"datetime\">2024-02-21 16:56:07</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">featuretools_test</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">pm2.5</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">43.9151</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">62.5672</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">0.5594</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "    date time             set used                target        mae       rmse   rsquared\n",
       "0   2024-02-21 16:56:07   featuretools_training   pm2.5     38.277     54.8781     0.6506\n",
       "1   2024-02-21 16:56:07   featuretools_test       pm2.5     43.9151    62.5672     0.5594"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_ft_pr.score(df_featuretools_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Using tsfresh\n",
    "\n",
    "tsfresh is a rather low-level library. To make things a bit easier, we have written a high-level wrapper which we placed in a separate module (`utils`).\n",
    "\n",
    "To limit the memory consumption, we undertake the following steps:\n",
    "\n",
    "- We limit ourselves to a memory of 1 day from any point in time. This is necessary, because tsfresh duplicates records for every time stamp. That means that looking back 7 days instead of one day, the memory consumption would be  seven times as high.\n",
    "- We extract only tsfresh's **MinimalFCParameters** and **IndexBasedFCParameters** (the latter is a superset of **TimeBasedFCParameters**).\n",
    "\n",
    "In order to make sure that tsfresh's features can be compared to getML's features, we also do the following:\n",
    "\n",
    "- We apply tsfresh's built-in feature selection algorithm.\n",
    "- Of the remaining features, we only keep the 40 features most correlated with the target (in terms of the absolute value of the correlation).\n",
    "- We add the original columns as additional features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pm2.5</th>\n",
       "      <th>DEWP</th>\n",
       "      <th>TEMP</th>\n",
       "      <th>PRES</th>\n",
       "      <th>Iws</th>\n",
       "      <th>Is</th>\n",
       "      <th>Ir</th>\n",
       "      <th>date</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>129.0</td>\n",
       "      <td>-16</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>1020.0</td>\n",
       "      <td>1.79</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 00:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>148.0</td>\n",
       "      <td>-15</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>1020.0</td>\n",
       "      <td>2.68</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 01:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>159.0</td>\n",
       "      <td>-11</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>1021.0</td>\n",
       "      <td>3.57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 02:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>181.0</td>\n",
       "      <td>-7</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>1022.0</td>\n",
       "      <td>5.36</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 03:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>138.0</td>\n",
       "      <td>-7</td>\n",
       "      <td>-5.0</td>\n",
       "      <td>1022.0</td>\n",
       "      <td>6.25</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-01-02 04:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35059</th>\n",
       "      <td>22.0</td>\n",
       "      <td>-19</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1013.0</td>\n",
       "      <td>114.87</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2013-12-31 19:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35060</th>\n",
       "      <td>18.0</td>\n",
       "      <td>-21</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1014.0</td>\n",
       "      <td>119.79</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2013-12-31 20:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35061</th>\n",
       "      <td>23.0</td>\n",
       "      <td>-21</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1014.0</td>\n",
       "      <td>125.60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2013-12-31 21:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35062</th>\n",
       "      <td>20.0</td>\n",
       "      <td>-21</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1014.0</td>\n",
       "      <td>130.52</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2013-12-31 22:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35063</th>\n",
       "      <td>23.0</td>\n",
       "      <td>-20</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1014.0</td>\n",
       "      <td>137.67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2013-12-31 23:00:00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33096 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       pm2.5  DEWP  TEMP    PRES     Iws  Is  Ir                 date  id\n",
       "24     129.0   -16  -4.0  1020.0    1.79   0   0  2010-01-02 00:00:00   1\n",
       "25     148.0   -15  -4.0  1020.0    2.68   0   0  2010-01-02 01:00:00   1\n",
       "26     159.0   -11  -5.0  1021.0    3.57   0   0  2010-01-02 02:00:00   1\n",
       "27     181.0    -7  -5.0  1022.0    5.36   1   0  2010-01-02 03:00:00   1\n",
       "28     138.0    -7  -5.0  1022.0    6.25   2   0  2010-01-02 04:00:00   1\n",
       "...      ...   ...   ...     ...     ...  ..  ..                  ...  ..\n",
       "35059   22.0   -19   7.0  1013.0  114.87   0   0  2013-12-31 19:00:00   1\n",
       "35060   18.0   -21   7.0  1014.0  119.79   0   0  2013-12-31 20:00:00   1\n",
       "35061   23.0   -21   7.0  1014.0  125.60   0   0  2013-12-31 21:00:00   1\n",
       "35062   20.0   -21   6.0  1014.0  130.52   0   0  2013-12-31 22:00:00   1\n",
       "35063   23.0   -20   7.0  1014.0  137.67   0   0  2013-12-31 23:00:00   1\n",
       "\n",
       "[33096 rows x 9 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train_pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Rolling: 100%|██████████| 40/40 [00:22<00:00,  1.82it/s]\n",
      "Feature Extraction: 100%|██████████| 40/40 [00:26<00:00,  1.48it/s]\n",
      "Feature Extraction: 100%|██████████| 40/40 [00:29<00:00,  1.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selecting the best out of 78 features...\n",
      "Time taken: 0h:1m:30.572509\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Rolling: 100%|██████████| 40/40 [00:03<00:00, 10.93it/s]\n",
      "Feature Extraction: 100%|██████████| 40/40 [00:06<00:00,  5.84it/s]\n",
      "Feature Extraction: 100%|██████████| 40/40 [00:07<00:00,  5.61it/s]\n"
     ]
    }
   ],
   "source": [
    "tsfresh_builder = TSFreshBuilder(\n",
    "    num_features=200, memory=24, column_id=\"id\", time_stamp=\"date\", target=\"pm2.5\"\n",
    ")\n",
    "\n",
    "with benchmark(\"tsfresh\"):\n",
    "    tsfresh_training = tsfresh_builder.fit(data_train_pandas)\n",
    "\n",
    "tsfresh_test = tsfresh_builder.transform(data_test_pandas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tsfresh does not contain built-in machine learning algorithms. In order to ensure a fair comparison, we use the exact same machine learning algorithm we have also used for getML: An XGBoost regressor with all hyperparameters set to their default value.\n",
    "\n",
    "In order to do so, we load the tsfresh features into the getML engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tsfresh_training = getml.data.DataFrame.from_pandas(\n",
    "    tsfresh_training, name=\"tsfresh_training\"\n",
    ")\n",
    "df_tsfresh_test = getml.data.DataFrame.from_pandas(tsfresh_test, name=\"tsfresh_test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, we need to set roles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_roles_tsfresh(df):\n",
    "    df[\"date\"] = df[\"date\"].as_ts()\n",
    "    df.set_role([\"pm2.5\"], getml.data.roles.target)\n",
    "    df.set_role([\"date\"], getml.data.roles.time_stamp)\n",
    "    df.set_role(df.roles.unused, getml.data.roles.numerical)\n",
    "    df.set_role([\"id\"], getml.data.roles.unused_float)\n",
    "    return df\n",
    "\n",
    "\n",
    "df_tsfresh_training = set_roles_tsfresh(df_tsfresh_training)\n",
    "df_tsfresh_test = set_roles_tsfresh(df_tsfresh_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, our pipeline is very simple. It only consists of a single XGBoostRegressor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['tsfresh', 'memory: 1d'])</pre>"
      ],
      "text/plain": [
       "Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['tsfresh', 'memory: 1d'])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor = getml.predictors.XGBoostRegressor()\n",
    "\n",
    "pipe_tsf_pr = getml.pipeline.Pipeline(\n",
    "    tags=[\"tsfresh\", \"memory: 1d\"], predictors=[predictor]\n",
    ")\n",
    "\n",
    "pipe_tsf_pr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking data model...\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Checking... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n",
      "OK.\n"
     ]
    }
   ],
   "source": [
    "pipe_tsf_pr.check(df_tsfresh_training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking data model...\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n",
      "OK.\n",
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "XGBoost: Training as predictor... 100% |██████████| [elapsed: 00:06, remaining: 00:00]          \n",
      "\n",
      "Trained pipeline.\n",
      "Time taken: 0h:0m:5.783919\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre>Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['tsfresh', 'memory: 1d'])</pre>"
      ],
      "text/plain": [
       "Pipeline(data_model='population',\n",
       "         feature_learners=[],\n",
       "         feature_selectors=[],\n",
       "         include_categorical=False,\n",
       "         loss_function='SquareLoss',\n",
       "         peripheral=[],\n",
       "         predictors=['XGBoostRegressor'],\n",
       "         preprocessors=[],\n",
       "         share_selected_features=0.5,\n",
       "         tags=['tsfresh', 'memory: 1d'])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_tsf_pr.fit(df_tsfresh_training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Staging... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "Preprocessing... 100% |██████████| [elapsed: 00:00, remaining: 00:00]          \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "  th {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  td {\n",
       "    text-align: left !important;\n",
       "  }\n",
       "  th:nth-child(1) {\n",
       "    text-align: right;\n",
       "    border-right: 1px solid LightGray;\n",
       "  }\n",
       "  th.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.float {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  th.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "  td.int {\n",
       "    text-align: right !important;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "<table class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      \n",
       "        \n",
       "          <th class=\"int\"> </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"datetime\">date time          </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">set used        </th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"str\">target</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">     mae</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">    rmse</th>\n",
       "        \n",
       "      \n",
       "        \n",
       "          <th class=\"float\">rsquared</th>\n",
       "        \n",
       "      \n",
       "    </tr>\n",
       "    \n",
       "  </thead>\n",
       "  <tbody>\n",
       "    \n",
       "      <tr>\n",
       "        <th>0</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"datetime\">2024-02-21 16:58:06</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">tsfresh_training</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">pm2.5</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">40.8917</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">57.9517</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">0.6099</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "      <tr>\n",
       "        <th>1</th>\n",
       "          \n",
       "            \n",
       "              <td class=\"datetime\">2024-02-21 16:58:06</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">tsfresh_test</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"str\">pm2.5</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">47.1106</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">66.6</td>\n",
       "            \n",
       "          \n",
       "            \n",
       "              <td class=\"float\">0.5015</td>\n",
       "            \n",
       "          \n",
       "      </tr>\n",
       "    \n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "    date time             set used           target        mae       rmse   rsquared\n",
       "0   2024-02-21 16:58:06   tsfresh_training   pm2.5     40.8917    57.9517     0.6099\n",
       "1   2024-02-21 16:58:06   tsfresh_test       pm2.5     47.1106    66.6        0.5015"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_tsf_pr.score(df_tsfresh_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>runtime</th>\n",
       "      <th>num_features</th>\n",
       "      <th>features_per_second</th>\n",
       "      <th>normalized_runtime</th>\n",
       "      <th>normalized_runtime_per_feature</th>\n",
       "      <th>mae</th>\n",
       "      <th>rmse</th>\n",
       "      <th>rsquared</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>getML: FastProp</th>\n",
       "      <td>0 days 00:00:08.637513</td>\n",
       "      <td>289</td>\n",
       "      <td>33.458244</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>44.252635</td>\n",
       "      <td>63.419113</td>\n",
       "      <td>0.546164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>featuretools</th>\n",
       "      <td>0 days 00:34:01.839960</td>\n",
       "      <td>114</td>\n",
       "      <td>0.055832</td>\n",
       "      <td>236.392114</td>\n",
       "      <td>599.266495</td>\n",
       "      <td>43.915071</td>\n",
       "      <td>62.567175</td>\n",
       "      <td>0.559369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tsfresh</th>\n",
       "      <td>0 days 00:01:30.572946</td>\n",
       "      <td>72</td>\n",
       "      <td>0.794939</td>\n",
       "      <td>10.485998</td>\n",
       "      <td>42.089066</td>\n",
       "      <td>47.110594</td>\n",
       "      <td>66.599982</td>\n",
       "      <td>0.501524</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               runtime  num_features  features_per_second  \\\n",
       "getML: FastProp 0 days 00:00:08.637513           289            33.458244   \n",
       "featuretools    0 days 00:34:01.839960           114             0.055832   \n",
       "tsfresh         0 days 00:01:30.572946            72             0.794939   \n",
       "\n",
       "                 normalized_runtime  normalized_runtime_per_feature  \\\n",
       "getML: FastProp            1.000000                        1.000000   \n",
       "featuretools             236.392114                      599.266495   \n",
       "tsfresh                   10.485998                       42.089066   \n",
       "\n",
       "                       mae       rmse  rsquared  \n",
       "getML: FastProp  44.252635  63.419113  0.546164  \n",
       "featuretools     43.915071  62.567175  0.559369  \n",
       "tsfresh          47.110594  66.599982  0.501524  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_features = dict(\n",
    "    fastprop=289,\n",
    "    featuretools=114,\n",
    "    tsfresh=72,\n",
    ")\n",
    "\n",
    "runtime_per_feature = [\n",
    "    benchmark.runtimes[\"fastprop\"] / num_features[\"fastprop\"],\n",
    "    benchmark.runtimes[\"featuretools\"] / num_features[\"featuretools\"],\n",
    "    benchmark.runtimes[\"tsfresh\"] / num_features[\"tsfresh\"],\n",
    "]\n",
    "\n",
    "features_per_second = [1.0 / r.total_seconds() for r in runtime_per_feature]\n",
    "\n",
    "normalized_runtime_per_feature = [\n",
    "    r / runtime_per_feature[0] for r in runtime_per_feature\n",
    "]\n",
    "\n",
    "comparison = pd.DataFrame(\n",
    "    dict(\n",
    "        runtime=[\n",
    "            benchmark.runtimes[\"fastprop\"],\n",
    "            benchmark.runtimes[\"featuretools\"],\n",
    "            benchmark.runtimes[\"tsfresh\"],\n",
    "        ],\n",
    "        num_features=num_features.values(),\n",
    "        features_per_second=features_per_second,\n",
    "        normalized_runtime=[\n",
    "            1,\n",
    "            benchmark.runtimes[\"featuretools\"] / benchmark.runtimes[\"fastprop\"],\n",
    "            benchmark.runtimes[\"tsfresh\"] / benchmark.runtimes[\"fastprop\"],\n",
    "        ],\n",
    "        normalized_runtime_per_feature=normalized_runtime_per_feature,\n",
    "        mae=[pipe_fp_pr.mae, pipe_ft_pr.mae, pipe_tsf_pr.mae],\n",
    "        rmse=[pipe_fp_pr.rmse, pipe_ft_pr.rmse, pipe_tsf_pr.rmse],\n",
    "        rsquared=[pipe_fp_pr.rsquared, pipe_ft_pr.rsquared, pipe_tsf_pr.rsquared],\n",
    "    )\n",
    ")\n",
    "\n",
    "comparison.index = [\"getML: FastProp\", \"featuretools\", \"tsfresh\"]\n",
    "\n",
    "comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# export for further use\n",
    "comparison.to_csv(\"comparisons/air_pollution.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Conclusion\n",
    "\n",
    "We have compared getML's feature learning algorithms to tsfresh's brute-force feature engineering approaches on a data set related to air pollution in China. We found that getML significantly outperforms featuretools and tsfresh. These results are consistent with the view that feature learning can yield significant improvements over simple propositionalization approaches.\n",
    "\n",
    "However, there are other datasets on which simple propositionalization performs well. Our suggestion is therefore to think of algorithms like `FastProp` and `RelMT` as tools in a toolbox. If a simple tool like `FastProp` gets the job done, then use that. But when you need more advanced approaches, like `RelMT`, you should have them at your disposal as well.\n",
    "\n",
    "You are encouraged to reproduce these results."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}