DeltaPy_Tabular_Augmentation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DeltaPy - Tabular Augmentation.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "pO7lYh-FoCtB",
        "q4hWH6jHye9z",
        "znQuAi0F3evQ",
        "aTSDCYyL0xjn",
        "QzHgiwSuQb10",
        "hKnipH7D-pxr",
        "dNzmV4panJVY",
        "22XpS0DOG0VU",
        "Jt3R7PQYHYB2",
        "LDqzoHLFIMW7",
        "jC_r_sZTgKxM",
        "vaRlVD27WsZE",
        "Ql_2y4TQkp-3",
        "gMiwZhKPliap",
        "iL7iDIYmn-NB",
        "UpFnjnDC-JGd",
        "0cpNd_I8K2_H",
        "9YQsVZl2-5ZV",
        "bCVACKWz91WS",
        "htKpCbhrC8g6",
        "QEzHO1H8Gr_O",
        "hETa-DCpHLpF",
        "IrqjEJlSHzzJ",
        "zrgqcQ_HfPvK",
        "dCdVu99rhFZb",
        "zd_PvZBPXwx0",
        "6Na8Y6ySbpC4",
        "TJI_GMWN6Dip",
        "OsCtrfqtHLeo",
        "dFlObKCMUuP6",
        "kTqf6PZLdyiv",
        "MxSVUu-skUIT",
        "gS0E7xQeqP9L",
        "hp_Pzpa_rEKg",
        "ZUkYNHlFre23",
        "2U6pdg6qr8Uw",
        "Sz0GMiTfsfgx",
        "N_9Ufe3hs1ep",
        "C8Iifk7Ct3rC",
        "IDvTaGA8ufg4",
        "tOMi6j_svQNk",
        "b0gZYnfbwQZT",
        "4p9OXv2CwmBo",
        "1aeY3c-I9mMj",
        "aqs9XcJFxGd1",
        "6MtonX1ryger",
        "L5PNG551zktJ",
        "spXFlQUm0ipH",
        "eZ1apEEC1cde",
        "DxehUR8F2POM",
        "GeO2VGEv3mZk",
        "Aset52xnA5KG",
        "EVLZqSY54MCz",
        "sox3PU6u5D63",
        "rrkyEFjm9oXe",
        "2cMJD8dLLd7m",
        "aJSJ5wSOIe2q",
        "L9F_PUJsSnS7",
        "xrbnRy4V2Gfs",
        "6G3VSHx_2N23",
        "wWKUuaz8EfXv",
        "70BMBD2kzZSS",
        "TKDXk2__SeJA",
        "J2y1Hyjv7HPt",
        "eyZwuKrxKkCu",
        "-67Id-_hw1z0",
        "Gf6z7xIvScmO",
        "uq6543YfXq3a"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jonbaer/googlecolab/blob/master/DeltaPy_Tabular_Augmentation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3uO3lHARqrFW",
        "colab_type": "text"
      },
      "source": [
        "## DeltaPy⁠⁠ — Tabular Data Augmentation\n",
        "\n",
        "### Introduction\n",
        "\n",
        "[Back to GitHub](https://github.com/firmai/deltapy/)\n",
        "\n",
        "Tabular augmentation is a new experimental space that makes use of novel and traditional data generation and synthesisation techniques to improve model prediction success. It is in essence a process of modular feature engineering and observation engineering while emphasising the order of augmentation to achieve the best predicted outcome from a given information set. \n",
        "\n",
        "![](https://github.com/firmai/deltapy/raw/master/assets/Tabularz.png)\n",
        "\n",
        "Data augmentation can be defined as any method that could increase the size or improve the quality of a dataset by generating new features or instances without the collection of additional data-points. Data augmentation is of particular importance in image classification tasks where additional data can be created by cropping, padding, or flipping existing images.\n",
        "\n",
        "Tabular cross-sectional and time-series prediction tasks can also benefit from augmentation. Here we divide tabular augmentation into columnular and row-wise methods. Row-wise methods are further divided into extraction and data synthesisation techniques, whereas columnular methods are divided into transformation, interaction, and mapping methods.  \n",
        "\n",
        "To take full advantage of tabular augmentation for time-series you would perform the techniques in the following order: (1) transforming, (2) interacting, (3) mapping, (4) extracting, and (5) synthesising (forthcoming). What follows is a practical example of how the above methodology can be used. The purpose here is to establish a framework for table augmentation and to point and guide the user to existing packages.\n",
        "\n",
        "See the [Skeleton Example](#example), for a combination of multiple methods that lead to a halfing of the mean squared error.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pO7lYh-FoCtB",
        "colab_type": "text"
      },
      "source": [
        "### Function Glossary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FkGXlRR7nySO",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "**Transformation**\n",
        "```python\n",
        "df_out = transform.robust_scaler(df.copy(), drop=[\"Close_1\"]); df_out.head()\n",
        "df_out = transform.standard_scaler(df.copy(), drop=[\"Close\"]); df_out.head()           \n",
        "df_out = transform.fast_fracdiff(df.copy(), [\"Close\",\"Open\"],0.5); df_out.head()\n",
        "df_out = transform.windsorization(df.copy(),\"Close\",para,strategy='both'); df_out.head()\n",
        "df_out = transform.operations(df.copy(),[\"Close\"]); df_out.head()\n",
        "df_out= transform.triple_exponential_smoothing(df.copy(),[\"Close\"], 12, .2,.2,.2,0); \n",
        "df_out = transform.naive_dec(df.copy(), [\"Close\",\"Open\"]); df_out.head()\n",
        "df_out = transform.bkb(df.copy(), [\"Close\"]); df_out.head()\n",
        "df_out = transform.butter_lowpass_filter(df.copy(),[\"Close\"],4); df_out.head()\n",
        "df_out = transform.instantaneous_phases(df.copy(), [\"Close\"]); df_out.head()\n",
        "df_out = transform.kalman_feat(df.copy(), [\"Close\"]); df_out.head()\n",
        "df_out = transform.perd_feat(df.copy(),[\"Close\"]); df_out.head()\n",
        "df_out = transform.fft_feat(df.copy(), [\"Close\"]); df_out.head()\n",
        "df_out = transform.harmonicradar_cw(df.copy(), [\"Close\"],0.3,0.2); df_out.head()\n",
        "df_out = transform.saw(df.copy(),[\"Close\",\"Open\"]); df_out.head()\n",
        "df_out = transform.modify(df.copy(),[\"Close\"]); df_out.head()\n",
        "df_out = transform.multiple_rolling(df, columns=[\"Close\"]); df_out.head()\n",
        "df_out = transform.multiple_lags(df, start=1, end=3, columns=[\"Close\"]); df_out.head()\n",
        "df_out  = transform.prophet_feat(df.copy().reset_index(),[\"Close\",\"Open\"],\"Date\", \"D\"); df_out.head()\n",
        "```\n",
        "**Interaction**\n",
        "```python\n",
        "df_out = interact.lowess(df.copy(), [\"Open\",\"Volume\"], df[\"Close\"], f=0.25, iter=3); df_out.head()\n",
        "df_out = interact.autoregression(df.copy()); df_out.head()\n",
        "df_out = interact.muldiv(df.copy(), [\"Close\",\"Open\"]); df_out.head()\n",
        "df_out = interact.decision_tree_disc(df.copy(), [\"Close\"]); df_out.head()\n",
        "df_out = interact.quantile_normalize(df.copy(), drop=[\"Close\"]); df_out.head()\n",
        "df_out = interact.tech(df.copy()); df_out.head()\n",
        "df_out = interact.genetic_feat(df.copy()); df_out.head()\n",
        "```\n",
        "**Mapping**\n",
        "```python\n",
        "df_out = mapper.pca_feature(df.copy(),variance_or_components=0.80,drop_cols=[\"Close_1\"]); df_out.head()\n",
        "df_out = mapper.cross_lag(df.copy()); df_out.head()\n",
        "df_out = mapper.a_chi(df.copy()); df_out.head()\n",
        "df_out = mapper.encoder_dataset(df.copy(), [\"Close_1\"], 15); df_out.head()\n",
        "df_out = mapper.lle_feat(df.copy(),[\"Close_1\"],4); df_out.head()\n",
        "df_out = mapper.feature_agg(df.copy(),[\"Close_1\"],4 ); df_out.head()\n",
        "df_out = mapper.neigh_feat(df.copy(),[\"Close_1\"],4 ); df_out.head()\n",
        "```\n",
        "\n",
        "**Extraction**\n",
        "```python\n",
        "extract.abs_energy(df[\"Close\"])\n",
        "extract.cid_ce(df[\"Close\"], True)\n",
        "extract.mean_abs_change(df[\"Close\"])\n",
        "extract.mean_second_derivative_central(df[\"Close\"])\n",
        "extract.variance_larger_than_standard_deviation(df[\"Close\"])\n",
        "extract.var_index(df[\"Close\"].values,var_index_param)\n",
        "extract.symmetry_looking(df[\"Close\"])\n",
        "extract.has_duplicate_max(df[\"Close\"])\n",
        "extract.partial_autocorrelation(df[\"Close\"])\n",
        "extract.augmented_dickey_fuller(df[\"Close\"])\n",
        "extract.gskew(df[\"Close\"])\n",
        "extract.stetson_mean(df[\"Close\"])\n",
        "extract.length(df[\"Close\"])\n",
        "extract.count_above_mean(df[\"Close\"])\n",
        "extract.longest_strike_below_mean(df[\"Close\"])\n",
        "extract.wozniak(df[\"Close\"])\n",
        "extract.last_location_of_maximum(df[\"Close\"])\n",
        "extract.fft_coefficient(df[\"Close\"])\n",
        "extract.ar_coefficient(df[\"Close\"])\n",
        "extract.index_mass_quantile(df[\"Close\"])\n",
        "extract.number_cwt_peaks(df[\"Close\"])\n",
        "extract.spkt_welch_density(df[\"Close\"])\n",
        "extract.linear_trend_timewise(df[\"Close\"])\n",
        "extract.c3(df[\"Close\"])\n",
        "extract.binned_entropy(df[\"Close\"])\n",
        "extract.svd_entropy(df[\"Close\"].values)\n",
        "extract.hjorth_complexity(df[\"Close\"])\n",
        "extract.max_langevin_fixed_point(df[\"Close\"])\n",
        "extract.percent_amplitude(df[\"Close\"])\n",
        "extract.cad_prob(df[\"Close\"])\n",
        "extract.zero_crossing_derivative(df[\"Close\"])\n",
        "extract.detrended_fluctuation_analysis(df[\"Close\"])\n",
        "extract.fisher_information(df[\"Close\"])\n",
        "extract.higuchi_fractal_dimension(df[\"Close\"])\n",
        "extract.petrosian_fractal_dimension(df[\"Close\"])\n",
        "extract.hurst_exponent(df[\"Close\"])\n",
        "extract.largest_lyauponov_exponent(df[\"Close\"])\n",
        "extract.whelch_method(df[\"Close\"])\n",
        "extract.find_freq(df[\"Close\"])\n",
        "extract.flux_perc(df[\"Close\"])\n",
        "extract.range_cum_s(df[\"Close\"])\n",
        "extract.structure_func(df[\"Close\"])\n",
        "extract.kurtosis(df[\"Close\"])\n",
        "extract.stetson_k(df[\"Close\"])\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wKeG6BQb1abl",
        "colab_type": "text"
      },
      "source": [
        "Test sets should ideally not be preprocessed with the training data, as in such a way one could be peaking ahead in the training data. The preprocessing parameters should be identified on the test set and then applied on the test set, i.e., the test set should not have an impact on the transformation applied. As an example, you would learn the parameters of PCA decomposition on the training set and then apply the parameters to both the train and the test set. \n",
        "\n",
        "The benefit of pipelines become clear when one wants to apply multiple augmentation methods. It makes it easy to learn the parameters and then apply them widely. For the most part, this notebook does not concern itself with 'peaking ahead' or pipelines, for some functions, one might have to restructure to code and make use of open source pacakages to create your preferred solution."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bd2uTnK52y5A",
        "colab_type": "text"
      },
      "source": [
        "**Notebook Dependencies**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BVxbIpy4ZZGf",
        "colab_type": "code",
        "outputId": "7451a938-b7c1-4857-f082-8b3c2acfc1c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "!pip install deltapy"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: deltapy in /usr/local/lib/python3.6/dist-packages (0.1.0)\n",
            "Requirement already satisfied: tensorflow in /usr/local/lib/python3.6/dist-packages (from deltapy) (2.2.0rc2)\n",
            "Requirement already satisfied: statsmodels in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.10.2)\n",
            "Requirement already satisfied: seasonal in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.3.1)\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.25.3)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.22.2.post1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from deltapy) (1.18.2)\n",
            "Requirement already satisfied: pykalman in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.9.5)\n",
            "Requirement already satisfied: tsaug in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.1.1)\n",
            "Requirement already satisfied: fbprophet in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.6)\n",
            "Requirement already satisfied: ta in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.5.17)\n",
            "Requirement already satisfied: gplearn in /usr/local/lib/python3.6/dist-packages (from deltapy) (0.4.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from deltapy) (1.4.1)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (1.12.1)\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (3.10.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (1.1.0)\n",
            "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (2.10.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (1.27.2)\n",
            "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (1.6.3)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (1.1.0)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (0.3.3)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (0.9.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (3.2.0)\n",
            "Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (2.2.0)\n",
            "Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0rc0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (2.2.0rc0)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (0.34.2)\n",
            "Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (0.2.0)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->deltapy) (1.12.0)\n",
            "Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from statsmodels->deltapy) (0.5.1)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->deltapy) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->deltapy) (2018.9)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->deltapy) (0.14.1)\n",
            "Requirement already satisfied: Cython>=0.22 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (0.29.16)\n",
            "Requirement already satisfied: cmdstanpy==0.4 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (0.4.0)\n",
            "Requirement already satisfied: pystan>=2.14 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (2.19.1.1)\n",
            "Requirement already satisfied: matplotlib>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (3.2.1)\n",
            "Requirement already satisfied: LunarCalendar>=0.0.9 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (0.0.9)\n",
            "Requirement already satisfied: convertdate>=2.1.2 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (2.2.0)\n",
            "Requirement already satisfied: holidays>=0.9.5 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (0.9.12)\n",
            "Requirement already satisfied: setuptools-git>=1.2 in /usr/local/lib/python3.6/dist-packages (from fbprophet->deltapy) (1.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.8.0->tensorflow->deltapy) (46.1.3)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (1.0.1)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (2.21.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (1.6.0.post2)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (3.2.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (0.4.1)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (1.7.2)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.0.0->fbprophet->deltapy) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.0.0->fbprophet->deltapy) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.0.0->fbprophet->deltapy) (2.4.6)\n",
            "Requirement already satisfied: ephem>=3.7.5.3 in /usr/local/lib/python3.6/dist-packages (from LunarCalendar>=0.0.9->fbprophet->deltapy) (3.7.7.1)\n",
            "Requirement already satisfied: pymeeus<=1,>=0.3.6 in /usr/local/lib/python3.6/dist-packages (from convertdate>=2.1.2->fbprophet->deltapy) (0.3.7)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (2019.11.28)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (2.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (1.3.0)\n",
            "Requirement already satisfied: cachetools<3.2,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (3.1.1)\n",
            "Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (4.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (0.2.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (3.1.0)\n",
            "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow->deltapy) (0.4.8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N4nmmPq3NIhw",
        "colab_type": "code",
        "outputId": "921603df-2504-4eb9-be19-d76209331204",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 863
        }
      },
      "source": [
        "!pip install pykalman\n",
        "!pip install tsaug\n",
        "!pip install ta\n",
        "!pip install tsaug\n",
        "!pip install pandasvault\n",
        "!pip install gplearn\n",
        "!pip install ta\n",
        "!pip install seasonal\n",
        "!pip install pandasvault"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: pykalman in /usr/local/lib/python3.6/dist-packages (0.9.5)\n",
            "Requirement already satisfied: tsaug in /usr/local/lib/python3.6/dist-packages (0.1.1)\n",
            "Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.6/dist-packages (from tsaug) (1.18.2)\n",
            "Requirement already satisfied: scipy>=1.3 in /usr/local/lib/python3.6/dist-packages (from tsaug) (1.4.1)\n",
            "Requirement already satisfied: ta in /usr/local/lib/python3.6/dist-packages (0.5.17)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from ta) (1.18.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from ta) (0.25.3)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->ta) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->ta) (2.8.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.6.1->pandas->ta) (1.12.0)\n",
            "Requirement already satisfied: tsaug in /usr/local/lib/python3.6/dist-packages (0.1.1)\n",
            "Requirement already satisfied: scipy>=1.3 in /usr/local/lib/python3.6/dist-packages (from tsaug) (1.4.1)\n",
            "Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.6/dist-packages (from tsaug) (1.18.2)\n",
            "Requirement already satisfied: pandasvault in /usr/local/lib/python3.6/dist-packages (0.0.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pandasvault) (1.18.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from pandasvault) (0.25.3)\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (from pandasvault) (0.0)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->pandasvault) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->pandasvault) (2.8.1)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn->pandasvault) (0.22.2.post1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.6.1->pandas->pandasvault) (1.12.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn->pandasvault) (0.14.1)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn->pandasvault) (1.4.1)\n",
            "Requirement already satisfied: gplearn in /usr/local/lib/python3.6/dist-packages (0.4.1)\n",
            "Requirement already satisfied: joblib>=0.13.0 in /usr/local/lib/python3.6/dist-packages (from gplearn) (0.14.1)\n",
            "Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.6/dist-packages (from gplearn) (0.22.2.post1)\n",
            "Requirement already satisfied: numpy>=1.11.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20.0->gplearn) (1.18.2)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20.0->gplearn) (1.4.1)\n",
            "Requirement already satisfied: ta in /usr/local/lib/python3.6/dist-packages (0.5.17)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from ta) (1.18.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from ta) (0.25.3)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->ta) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->ta) (2018.9)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.6.1->pandas->ta) (1.12.0)\n",
            "Requirement already satisfied: seasonal in /usr/local/lib/python3.6/dist-packages (0.3.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from seasonal) (1.4.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from seasonal) (1.18.2)\n",
            "Requirement already satisfied: pandasvault in /usr/local/lib/python3.6/dist-packages (0.0.3)\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (from pandasvault) (0.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pandasvault) (1.18.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from pandasvault) (0.25.3)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn->pandasvault) (0.22.2.post1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->pandasvault) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->pandasvault) (2.8.1)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn->pandasvault) (1.4.1)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sklearn->pandasvault) (0.14.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.6.1->pandas->pandasvault) (1.12.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cc0Sv-nhzFKR",
        "colab_type": "text"
      },
      "source": [
        "### Data and Package Load"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W82BXI_GzJ7_",
        "colab_type": "code",
        "outputId": "71114fb4-f107-4c56-8219-de2b3d296977",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from deltapy import transform, interact, mapper, extract \n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "def data_copy():\n",
        "  df = pd.read_csv(\"https://github.com/firmai/random-assets-two/raw/master/numpy/tsla.csv\")\n",
        "  df[\"Close_1\"] = df[\"Close\"].shift(-1)\n",
        "  df = df.dropna()\n",
        "  df[\"Date\"] = pd.to_datetime(df[\"Date\"])\n",
        "  df = df.set_index(\"Date\")\n",
        "  return df\n",
        "df = data_copy(); df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...   Adj Close     Close_1\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...  310.119995  300.359985\n",
              "2019-01-03  309.399994  297.380005  ...  300.359985  317.690002\n",
              "2019-01-04  318.000000  302.730011  ...  317.690002  334.959991\n",
              "2019-01-07  336.739990  317.750000  ...  334.959991  335.350006\n",
              "2019-01-08  344.010010  327.019989  ...  335.350006  338.529999\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DyaWG2av00IZ",
        "colab_type": "text"
      },
      "source": [
        "Some of these categories are fluid and some techniques could fit into multiple buckets. This is an attempt to find an exhaustive number of techniques, but not an exhuastive list of implementations of the techniques. For example, there are thousands of ways to smooth a time-series, but we have only includes 1-2 techniques of interest under each category."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bw7dYcKp4-H9",
        "colab_type": "text"
      },
      "source": [
        "### **(1) [<font color=\"black\">Transformation:</font>](#transformation)**\n",
        "-----------------\n",
        "1. Scaling/Normalisation\n",
        "2. Standardisation\n",
        "10. Differencing\n",
        "3. Capping\n",
        "13. Operations\n",
        "4. Smoothing\n",
        "5. Decomposing\n",
        "6. Filtering\n",
        "7. Spectral Analysis\n",
        "8. Waveforms\n",
        "9. Modifications\n",
        "11. Rolling\n",
        "12. Lagging\n",
        "14. Forecast Model\n",
        "\n",
        "### **(2) [<font color=\"black\">Interaction:</font>](#interaction)**\n",
        "-----------------\n",
        "1. Regressions\n",
        "2. Operators\n",
        "3. Discretising\n",
        "4. Normalising\n",
        "5. Distance\n",
        "6. Speciality\n",
        "7. Genetic\n",
        "\n",
        "### **(3) [<font color=\"black\">Mapping:</font>](#mapping)**\n",
        "-----------------\n",
        "1. Eigen Decomposition\n",
        "2. Cross Decomposition\n",
        "3. Kernel Approximation\n",
        "4. Autoencoder\n",
        "5. Manifold Learning\n",
        "6. Clustering\n",
        "7. Neighbouring\n",
        "\n",
        "### **(4) [<font color=\"black\">Extraction:</font>](#extraction)**\n",
        "-----------------\n",
        "1. Energy\n",
        "2. Distance\n",
        "3. Differencing\n",
        "4. Derivative\n",
        "5. Volatility\n",
        "6. Shape\n",
        "7. Occurence\n",
        "8. Autocorrelation\n",
        "9. Stochasticity\n",
        "10. Averages\n",
        "11. Size\n",
        "13. Count\n",
        "14. Streaks\n",
        "14. Location\n",
        "15. Model Coefficients\n",
        "16. Quantile\n",
        "17. Peaks\n",
        "18. Density\n",
        "20. Linearity\n",
        "20. Non-linearity\n",
        "21. Entropy\n",
        "22. Fixed Points\n",
        "23. Amplitude\n",
        "23. Probability\n",
        "24. Crossings\n",
        "25. Fluctuation\n",
        "26. Information\n",
        "27. Fractals\n",
        "29. Exponent\n",
        "30. Spectral Analysis\n",
        "31. Percentile\n",
        "32. Range\n",
        "33. Structural\n",
        "12. Distribution\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4hWH6jHye9z",
        "colab_type": "text"
      },
      "source": [
        "<a name=\"transformation\"></a>\n",
        "\n",
        "##**(1) Transformation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tTeu0cVey03H",
        "colab_type": "text"
      },
      "source": [
        "Here transformation is any method that includes only one feature as an input to produce a new feature/s. Transformations can be applied to cross-section and time-series data. Some transformations are exclusive to time-series data (smoothing, filtering), but a handful of functions apply to both. \n",
        "\n",
        "Where the time series methods has a centred mean, or are forward-looking, there is a need to recalculate the outputed time series on a running basis to ensure that information of the future does not leak into the model. The last value of this recalculated series or an extracted feature from this series can then be used as a running value that is only backward looking, satisfying the no 'peaking' ahead rule. \n",
        "\n",
        "There are some packaged in Python that dynamically create time series and extracts their features, but none that incoropates the dynamic creation of a time series in combination with a wide application of prespecified list of extractions. Because this technique is expensive, we have a preference for models that only take historical data into account. \n",
        "\n",
        "In this section we will include a list of all types of transformations, those that only use present information (operations), those that incorporate all values (interpolation methods), those that only include past values (smoothing functions), and those that incorporate a subset window of lagging and leading values (select filters). Only those that use historical values or are turned into prediction methods can be used out of the box. The entire time series can be used in the model development process for historical value methods, and only the forecasted values can be used for prediction models. \n",
        "\n",
        "Curve fitting can involve either interpolation, where an exact fit to the data is required, or smoothing, in which a \"smooth\" function is constructed that approximately fits the data. When using an interpolation method, you are taking future information into account e.g, cubic spline. You can use interpolation methods to forecast into the future (extrapolation), and then use those forecasts in a training set. Or you could recalulate the interpolation for each time step and then extract features out of that series (extraction method). Interpolation and other forward-looking methods can be used if they are turned into prediction problems, then the forecasted values can be trained and tested on, and the fitted data can be diregarded. In the list presented below the first five methods can be used for cross-section and time series data, after that the time-series only methods follow."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znQuAi0F3evQ",
        "colab_type": "text"
      },
      "source": [
        "#### **(1) Scaling/Normalisation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwXnRAex-jtB",
        "colab_type": "text"
      },
      "source": [
        "There are a multitude of scaling methods available. Scaling generally gets applied to the entire dataset and is especially necessary for certain algorithms. K-means make use of euclidean distance hence the need for scaling. For PCA because we are trying to identify the feature with maximus variance we also need scaling. Similarly, we need scaled features for gradient descent. Any algorithm that is not based on a distance measure is not affected by feature scaling. Some of the methods include range scalers like minimum-maximum scaler, maximum absolute scaler or even standardisation methods like the standard scaler can be used for scaling. The example used here is robust scaler. Normalisation is a good technique when you don't know the distribution of the data. Scaling looks into the future, so parameters have to be training on a tranining set and applied to a test set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "plYLz7Wg8oZt",
        "colab_type": "text"
      },
      "source": [
        "(i) Robust Scaler\n",
        "\n",
        "Scaling according to the interquartile range, making it robust to outliers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ixAUJ1Xs3eOa",
        "colab_type": "code",
        "outputId": "14ea4974-2582-46b6-c4ad-262c32b4af40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        }
      },
      "source": [
        "def robust_scaler(df, drop=None,quantile_range=(25, 75) ):\n",
        "    if drop:\n",
        "      keep = df[drop]\n",
        "      df = df.drop(drop, axis=1)\n",
        "    center = np.median(df, axis=0)\n",
        "    quantiles = np.percentile(df, quantile_range, axis=0)\n",
        "    scale = quantiles[1] - quantiles[0]\n",
        "    df = (df - center) / scale\n",
        "    if drop:\n",
        "      df = pd.concat((keep,df),axis=1)\n",
        "    return df\n",
        "\n",
        "df_out = transform.robust_scaler(df.copy(), drop=[\"Close_1\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close_1</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.641834</td>\n",
              "      <td>0.571030</td>\n",
              "      <td>0.612450</td>\n",
              "      <td>0.633602</td>\n",
              "      <td>0.852829</td>\n",
              "      <td>0.633602</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.571040</td>\n",
              "      <td>0.552195</td>\n",
              "      <td>0.624332</td>\n",
              "      <td>0.509799</td>\n",
              "      <td>-0.164678</td>\n",
              "      <td>0.509799</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>0.677292</td>\n",
              "      <td>0.623160</td>\n",
              "      <td>0.611130</td>\n",
              "      <td>0.729625</td>\n",
              "      <td>-0.071694</td>\n",
              "      <td>0.729625</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>0.908821</td>\n",
              "      <td>0.822390</td>\n",
              "      <td>0.818668</td>\n",
              "      <td>0.948690</td>\n",
              "      <td>-0.037636</td>\n",
              "      <td>0.948690</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>338.529999</td>\n",
              "      <td>0.998641</td>\n",
              "      <td>0.945351</td>\n",
              "      <td>1.085880</td>\n",
              "      <td>0.953637</td>\n",
              "      <td>-0.155290</td>\n",
              "      <td>0.953637</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Close_1      High       Low  ...     Close    Volume  Adj Close\n",
              "Date                                        ...                               \n",
              "2019-01-02  300.359985  0.641834  0.571030  ...  0.633602  0.852829   0.633602\n",
              "2019-01-03  317.690002  0.571040  0.552195  ...  0.509799 -0.164678   0.509799\n",
              "2019-01-04  334.959991  0.677292  0.623160  ...  0.729625 -0.071694   0.729625\n",
              "2019-01-07  335.350006  0.908821  0.822390  ...  0.948690 -0.037636   0.948690\n",
              "2019-01-08  338.529999  0.998641  0.945351  ...  0.953637 -0.155290   0.953637\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aTSDCYyL0xjn",
        "colab_type": "text"
      },
      "source": [
        "#### **(2) Standardisation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y2DzLlDY05d0",
        "colab_type": "text"
      },
      "source": [
        "When using a standardisation method, it is often more effective when the attribute itself if Gaussian. It is also useful to apply the technique when the model you want to use makes assumptions of Gaussian distributions like linear regression, logistic regression, and linear discriminant analysis. For most applications, standardisation is recommended."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5qQ9rB8s1IJh",
        "colab_type": "text"
      },
      "source": [
        "(i) Standard Scaler\n",
        "\n",
        "Standardize features by removing the mean and scaling to unit variance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z-GwcoHa9Z_Q",
        "colab_type": "code",
        "outputId": "0eec5357-a85c-410a-8810-dbfdc314ec5f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        }
      },
      "source": [
        "def standard_scaler(df,drop ):\n",
        "    if drop:\n",
        "      keep = df[drop]\n",
        "      df = df.drop(drop, axis=1)\n",
        "    mean = np.mean(df, axis=0)\n",
        "    scale = np.std(df, axis=0)\n",
        "    df = (df - mean) / scale  \n",
        "    if drop:\n",
        "      df = pd.concat((keep,df),axis=1)\n",
        "    return df\n",
        "\n",
        "\n",
        "df_out = transform.standard_scaler(df.copy(), drop=[\"Close\"]); df_out.head()           "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>310.119995</td>\n",
              "      <td>0.725937</td>\n",
              "      <td>0.597953</td>\n",
              "      <td>0.650163</td>\n",
              "      <td>0.525624</td>\n",
              "      <td>0.711548</td>\n",
              "      <td>0.509167</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.616624</td>\n",
              "      <td>0.570313</td>\n",
              "      <td>0.667579</td>\n",
              "      <td>-0.459544</td>\n",
              "      <td>0.524695</td>\n",
              "      <td>0.836274</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.780689</td>\n",
              "      <td>0.674452</td>\n",
              "      <td>0.648228</td>\n",
              "      <td>-0.369516</td>\n",
              "      <td>0.856475</td>\n",
              "      <td>1.162247</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>1.138197</td>\n",
              "      <td>0.966818</td>\n",
              "      <td>0.952432</td>\n",
              "      <td>-0.336540</td>\n",
              "      <td>1.187105</td>\n",
              "      <td>1.169609</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>1.276889</td>\n",
              "      <td>1.147259</td>\n",
              "      <td>1.344105</td>\n",
              "      <td>-0.450455</td>\n",
              "      <td>1.194572</td>\n",
              "      <td>1.229632</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                 Close      High       Low  ...    Volume  Adj Close   Close_1\n",
              "Date                                        ...                               \n",
              "2019-01-02  310.119995  0.725937  0.597953  ...  0.525624   0.711548  0.509167\n",
              "2019-01-03  300.359985  0.616624  0.570313  ... -0.459544   0.524695  0.836274\n",
              "2019-01-04  317.690002  0.780689  0.674452  ... -0.369516   0.856475  1.162247\n",
              "2019-01-07  334.959991  1.138197  0.966818  ... -0.336540   1.187105  1.169609\n",
              "2019-01-08  335.350006  1.276889  1.147259  ... -0.450455   1.194572  1.229632\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QzHgiwSuQb10",
        "colab_type": "text"
      },
      "source": [
        "#### **(3) Differencing**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RgNRV32tLVBc",
        "colab_type": "text"
      },
      "source": [
        "Computing the differences between consecutive observation, normally used to obtain a stationary time series."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QqJtJD3AQfYC",
        "colab_type": "text"
      },
      "source": [
        "(i) Fractional Differencing\n",
        "\n",
        "Fractional differencing, allows us to achieve stationarity while maintaining the maximum amount of memory compared to integer differencing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mC14vQVhLdMU",
        "colab_type": "code",
        "outputId": "3aab7ba9-cc8d-41a2-c28e-e97a94d2dbc0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "import pylab as pl\n",
        "\n",
        "def fast_fracdiff(x, cols, d):\n",
        "    for col in cols:\n",
        "      T = len(x[col])\n",
        "      np2 = int(2 ** np.ceil(np.log2(2 * T - 1)))\n",
        "      k = np.arange(1, T)\n",
        "      b = (1,) + tuple(np.cumprod((k - d - 1) / k))\n",
        "      z = (0,) * (np2 - T)\n",
        "      z1 = b + z\n",
        "      z2 = tuple(x[col]) + z\n",
        "      dx = pl.ifft(pl.fft(z1) * pl.fft(z2))\n",
        "      x[col+\"_frac\"] = np.real(dx[0:T])\n",
        "    return x \n",
        "  \n",
        "df_out = transform.fast_fracdiff(df.copy(), [\"Close\",\"Open\"],0.5); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_frac</th>\n",
              "      <th>Open_frac</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>306.100006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>145.299988</td>\n",
              "      <td>153.949997</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>128.745010</td>\n",
              "      <td>114.237499</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>119.187492</td>\n",
              "      <td>111.213751</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>97.272199</td>\n",
              "      <td>111.705459</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Close_frac   Open_frac\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...  310.119995  306.100006\n",
              "2019-01-03  309.399994  297.380005  ...  145.299988  153.949997\n",
              "2019-01-04  318.000000  302.730011  ...  128.745010  114.237499\n",
              "2019-01-07  336.739990  317.750000  ...  119.187492  111.213751\n",
              "2019-01-08  344.010010  327.019989  ...   97.272199  111.705459\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hKnipH7D-pxr",
        "colab_type": "text"
      },
      "source": [
        "#### **(4) Capping**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XD6XT59u-ug2",
        "colab_type": "text"
      },
      "source": [
        "Any method that provides sets a floor and a cap to a feature's value. Capping can affect the distribution of data, so it should not be exagerated. One can cap values by using the average, by using the max and min values, or by an arbitary extreme value.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qrV-9dCSEhbx",
        "colab_type": "text"
      },
      "source": [
        "(i) Winzorisation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SEUnDHhDEGUA",
        "colab_type": "text"
      },
      "source": [
        "The transformation of features by limiting extreme values in the statistical data to reduce the effect of possibly spurious outliers by replacing it with a certain percentile value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6XJmBI4F-SPK",
        "colab_type": "code",
        "outputId": "125a1fb6-c967-49c5-bfb6-32b26f85cd88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 310
        }
      },
      "source": [
        "def outlier_detect(data,col,threshold=1,method=\"IQR\"):\n",
        "  \n",
        "    if method == \"IQR\":\n",
        "      IQR = data[col].quantile(0.75) - data[col].quantile(0.25)\n",
        "      Lower_fence = data[col].quantile(0.25) - (IQR * threshold)\n",
        "      Upper_fence = data[col].quantile(0.75) + (IQR * threshold)\n",
        "    if method == \"STD\":\n",
        "      Upper_fence = data[col].mean() + threshold * data[col].std()\n",
        "      Lower_fence = data[col].mean() - threshold * data[col].std()   \n",
        "    if method == \"OWN\":\n",
        "      Upper_fence = data[col].mean() + threshold * data[col].std()\n",
        "      Lower_fence = data[col].mean() - threshold * data[col].std() \n",
        "    if method ==\"MAD\":\n",
        "      median = data[col].median()\n",
        "      median_absolute_deviation = np.median([np.abs(y - median) for y in data[col]])\n",
        "      modified_z_scores = pd.Series([0.6745 * (y - median) / median_absolute_deviation for y in data[col]])\n",
        "      outlier_index = np.abs(modified_z_scores) > threshold\n",
        "      print('Num of outlier detected:',outlier_index.value_counts()[1])\n",
        "      print('Proportion of outlier detected',outlier_index.value_counts()[1]/len(outlier_index))\n",
        "      return outlier_index, (median_absolute_deviation, median_absolute_deviation)\n",
        "\n",
        "    para = (Upper_fence, Lower_fence)\n",
        "    tmp = pd.concat([data[col]>Upper_fence,data[col]<Lower_fence],axis=1)\n",
        "    outlier_index = tmp.any(axis=1)\n",
        "    print('Num of outlier detected:',outlier_index.value_counts()[1])\n",
        "    print('Proportion of outlier detected',outlier_index.value_counts()[1]/len(outlier_index))\n",
        "    \n",
        "    return outlier_index, para\n",
        "\n",
        "def windsorization(data,col,para,strategy='both'):\n",
        "    \"\"\"\n",
        "    top-coding & bottom coding (capping the maximum of a distribution at an arbitrarily set value,vice versa)\n",
        "    \"\"\"\n",
        "\n",
        "    data_copy = data.copy(deep=True)  \n",
        "    if strategy == 'both':\n",
        "        data_copy.loc[data_copy[col]>para[0],col] = para[0]\n",
        "        data_copy.loc[data_copy[col]<para[1],col] = para[1]\n",
        "    elif strategy == 'top':\n",
        "        data_copy.loc[data_copy[col]>para[0],col] = para[0]\n",
        "    elif strategy == 'bottom':\n",
        "        data_copy.loc[data_copy[col]<para[1],col] = para[1]  \n",
        "    return data_copy\n",
        "\n",
        "_, para = transform.outlier_detect(df, \"Close\")\n",
        "df_out = transform.windsorization(df.copy(),\"Close\",para,strategy='both'); df_out.head()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Num of outlier detected: 8\n",
            "Proportion of outlier detected 0.03187250996015936\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...   Adj Close     Close_1\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...  310.119995  300.359985\n",
              "2019-01-03  309.399994  297.380005  ...  300.359985  317.690002\n",
              "2019-01-04  318.000000  302.730011  ...  317.690002  334.959991\n",
              "2019-01-07  336.739990  317.750000  ...  334.959991  335.350006\n",
              "2019-01-08  344.010010  327.019989  ...  335.350006  338.529999\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dNzmV4panJVY",
        "colab_type": "text"
      },
      "source": [
        "#### **(5) Operations**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TFZXbzUWMNBR",
        "colab_type": "text"
      },
      "source": [
        "Operations here are treated like traditional transformations. It is the replacement of a variable by a function of that variable. In a stronger sense, a transformation is a replacement that changes the shape of a distribution or relationship."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0Tsw70bnWrX",
        "colab_type": "text"
      },
      "source": [
        "(i) Power, Log, Recipricol, Square Root"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JBobjFQTnJC4",
        "colab_type": "code",
        "outputId": "c1b411ba-b468-4217-e47f-bcb25adbad89",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "def operations(df,features):\n",
        "  df_new = df[features]\n",
        "  df_new = df_new - df_new.min()\n",
        "\n",
        "  sqr_name = [str(fa)+\"_POWER_2\" for fa in df_new.columns]\n",
        "  log_p_name = [str(fa)+\"_LOG_p_one_abs\" for fa in df_new.columns]\n",
        "  rec_p_name = [str(fa)+\"_RECIP_p_one\" for fa in df_new.columns]\n",
        "  sqrt_name = [str(fa)+\"_SQRT_p_one\" for fa in df_new.columns]\n",
        "\n",
        "  df_sqr = pd.DataFrame(np.power(df_new.values, 2),columns=sqr_name, index=df.index)\n",
        "  df_log = pd.DataFrame(np.log(df_new.add(1).abs().values),columns=log_p_name, index=df.index)\n",
        "  df_rec = pd.DataFrame(np.reciprocal(df_new.add(1).values),columns=rec_p_name, index=df.index)\n",
        "  df_sqrt = pd.DataFrame(np.sqrt(df_new.abs().add(1).values),columns=sqrt_name, index=df.index)\n",
        "\n",
        "  dfs = [df, df_sqr, df_log, df_rec, df_sqrt]\n",
        "\n",
        "  df=  pd.concat(dfs, axis=1)\n",
        "\n",
        "  return df\n",
        "\n",
        "df_out = transform.operations(df.copy(),[\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_POWER_2</th>\n",
              "      <th>Close_LOG_p_one_abs</th>\n",
              "      <th>Close_RECIP_p_one</th>\n",
              "      <th>Close_SQRT_p_one</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>17200.320899</td>\n",
              "      <td>4.883938</td>\n",
              "      <td>0.007567</td>\n",
              "      <td>11.495651</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>14735.528247</td>\n",
              "      <td>4.807213</td>\n",
              "      <td>0.008171</td>\n",
              "      <td>11.063001</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>19243.238739</td>\n",
              "      <td>4.939640</td>\n",
              "      <td>0.007157</td>\n",
              "      <td>11.820322</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>24332.877053</td>\n",
              "      <td>5.056182</td>\n",
              "      <td>0.006370</td>\n",
              "      <td>12.529565</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>24454.705927</td>\n",
              "      <td>5.058663</td>\n",
              "      <td>0.006354</td>\n",
              "      <td>12.545119</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Close_RECIP_p_one  Close_SQRT_p_one\n",
              "Date                                ...                                     \n",
              "2019-01-02  315.130005  298.799988  ...           0.007567         11.495651\n",
              "2019-01-03  309.399994  297.380005  ...           0.008171         11.063001\n",
              "2019-01-04  318.000000  302.730011  ...           0.007157         11.820322\n",
              "2019-01-07  336.739990  317.750000  ...           0.006370         12.529565\n",
              "2019-01-08  344.010010  327.019989  ...           0.006354         12.545119\n",
              "\n",
              "[5 rows x 11 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "22XpS0DOG0VU",
        "colab_type": "text"
      },
      "source": [
        "#### **(6) Smoothing**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WRi9cup4FGAc",
        "colab_type": "text"
      },
      "source": [
        "Here we maintain that any method that has a component of historical averaging is a smoothing method such as a simple moving average and single, double and tripple exponential smoothing methods. These forms of non-causal filters are also popular in signal processing and are called filters, where exponential smoothing is called an IIR filter and a moving average a FIR filter with equal weighting factors."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NP80iTFiG68a",
        "colab_type": "text"
      },
      "source": [
        "(i) Tripple Exponential Smoothing (Holt-Winters Exponential Smoothing)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j1AMt9wjGqL9",
        "colab_type": "text"
      },
      "source": [
        "The Holt-Winters seasonal method comprises the forecast equation and three smoothing equations — one for the level $ℓt$, one for the trend &bt&, and one for the seasonal component $st$. This particular version is performed by looking at the last 12 periods. For that reason, the first 12 records should be disregarded because they can't make use of the required window size for a fair calculation. The calculation is such that values are still provided for those periods based on whatever data might be available. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uc3S5mWYKeIQ",
        "colab_type": "code",
        "outputId": "38aa8d45-3738-453a-f887-de2b0f3cf311",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "def initial_trend(series, slen):\n",
        "    sum = 0.0\n",
        "    for i in range(slen):\n",
        "        sum += float(series[i+slen] - series[i]) / slen\n",
        "    return sum / slen\n",
        "\n",
        "def initial_seasonal_components(series, slen):\n",
        "    seasonals = {}\n",
        "    season_averages = []\n",
        "    n_seasons = int(len(series)/slen)\n",
        "    # compute season averages\n",
        "    for j in range(n_seasons):\n",
        "        season_averages.append(sum(series[slen*j:slen*j+slen])/float(slen))\n",
        "    # compute initial values\n",
        "    for i in range(slen):\n",
        "        sum_of_vals_over_avg = 0.0\n",
        "        for j in range(n_seasons):\n",
        "            sum_of_vals_over_avg += series[slen*j+i]-season_averages[j]\n",
        "        seasonals[i] = sum_of_vals_over_avg/n_seasons\n",
        "    return seasonals\n",
        "\n",
        "def triple_exponential_smoothing(df,cols, slen, alpha, beta, gamma, n_preds):\n",
        "    for col in cols:\n",
        "      result = []\n",
        "      seasonals = initial_seasonal_components(df[col], slen)\n",
        "      for i in range(len(df[col])+n_preds):\n",
        "          if i == 0: # initial values\n",
        "              smooth = df[col][0]\n",
        "              trend = initial_trend(df[col], slen)\n",
        "              result.append(df[col][0])\n",
        "              continue\n",
        "          if i >= len(df[col]): # we are forecasting\n",
        "              m = i - len(df[col]) + 1\n",
        "              result.append((smooth + m*trend) + seasonals[i%slen])\n",
        "          else:\n",
        "              val = df[col][i]\n",
        "              last_smooth, smooth = smooth, alpha*(val-seasonals[i%slen]) + (1-alpha)*(smooth+trend)\n",
        "              trend = beta * (smooth-last_smooth) + (1-beta)*trend\n",
        "              seasonals[i%slen] = gamma*(val-smooth) + (1-gamma)*seasonals[i%slen]\n",
        "              result.append(smooth+trend+seasonals[i%slen])\n",
        "      df[col+\"_TES\"] = result\n",
        "    #print(seasonals)\n",
        "    return df\n",
        "\n",
        "df_out= transform.triple_exponential_smoothing(df.copy(),[\"Close\"], 12, .2,.2,.2,0); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_TES</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.119995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>301.784944</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>307.510139</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>314.721120</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>317.535470</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...     Close_1   Close_TES\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...  300.359985  310.119995\n",
              "2019-01-03  309.399994  297.380005  ...  317.690002  301.784944\n",
              "2019-01-04  318.000000  302.730011  ...  334.959991  307.510139\n",
              "2019-01-07  336.739990  317.750000  ...  335.350006  314.721120\n",
              "2019-01-08  344.010010  327.019989  ...  338.529999  317.535470\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jt3R7PQYHYB2",
        "colab_type": "text"
      },
      "source": [
        "#### **(7) Decomposing**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5iAycWZ3JHhQ",
        "colab_type": "text"
      },
      "source": [
        "Decomposition procedures are used in time series to describe the trend and seasonal factors in a time series. More extensive decompositions might also include long-run cycles, holiday effects, day of week effects and so on. Here, we’ll only consider trend and seasonal decompositions. A naive decomposition makes use of moving averages, other decomposition methods are available that make use of LOESS."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oJHN_0s6H5pF",
        "colab_type": "text"
      },
      "source": [
        "(i) Naive Decomposition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7HiyUd6U790e",
        "colab_type": "text"
      },
      "source": [
        "The base trend takes historical information into account and established moving averages; it does not have to be linear. To estimate the seasonal component for each season, simply average the detrended values for that season. If the seasonal variation looks constant, we should use the additive model. If the magintude is increasing as a function of time, we will use multiplicative. Here because it is predictive in nature we are using a one sided moving average, as opposed to a two-sided centred average. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XOHRlj4s-SMA",
        "colab_type": "code",
        "outputId": "9d05e2e9-dd66-4e86-d3eb-89ccf37b26c0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "import statsmodels.api as sm\n",
        "\n",
        "def naive_dec(df, columns, freq=2):\n",
        "  for col in columns:\n",
        "    decomposition = sm.tsa.seasonal_decompose(df[col], model='additive', freq = freq, two_sided=False)\n",
        "    df[col+\"_NDDT\" ] = decomposition.trend\n",
        "    df[col+\"_NDDT\"] = decomposition.seasonal\n",
        "    df[col+\"_NDDT\"] = decomposition.resid\n",
        "  return df\n",
        "\n",
        "df_out = transform.naive_dec(df.copy(), [\"Close\",\"Open\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_NDDT</th>\n",
              "      <th>Close_NDDS</th>\n",
              "      <th>Close_NDDR</th>\n",
              "      <th>Open_NDDT</th>\n",
              "      <th>Open_NDDS</th>\n",
              "      <th>Open_NDDR</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>NaN</td>\n",
              "      <td>-0.417818</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>-0.45177</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.417818</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.45177</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>307.132492</td>\n",
              "      <td>-0.417818</td>\n",
              "      <td>10.975329</td>\n",
              "      <td>306.525002</td>\n",
              "      <td>-0.45177</td>\n",
              "      <td>-0.073231</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>317.674995</td>\n",
              "      <td>0.417818</td>\n",
              "      <td>16.867178</td>\n",
              "      <td>310.180000</td>\n",
              "      <td>0.45177</td>\n",
              "      <td>11.088231</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>330.739998</td>\n",
              "      <td>-0.417818</td>\n",
              "      <td>5.027827</td>\n",
              "      <td>322.849998</td>\n",
              "      <td>-0.45177</td>\n",
              "      <td>19.561763</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Open_NDDS  Open_NDDR\n",
              "Date                                ...                      \n",
              "2019-01-02  315.130005  298.799988  ...   -0.45177        NaN\n",
              "2019-01-03  309.399994  297.380005  ...    0.45177        NaN\n",
              "2019-01-04  318.000000  302.730011  ...   -0.45177  -0.073231\n",
              "2019-01-07  336.739990  317.750000  ...    0.45177  11.088231\n",
              "2019-01-08  344.010010  327.019989  ...   -0.45177  19.561763\n",
              "\n",
              "[5 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDqzoHLFIMW7",
        "colab_type": "text"
      },
      "source": [
        "#### **(8) Filtering**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xy3tnLLLe6K1",
        "colab_type": "text"
      },
      "source": [
        "It is often useful to either low-pass filter (smooth) time series in order to reveal low-frequency features and trends, or to high-pass filter (detrend) time series in order to isolate high frequency transients (e.g. storms). Low pass filters use historical values, high-pass filters detrends with low-pass filters, so also indirectly uses historical values.\n",
        "\n",
        "There are a few filters available, closely associated with decompositions and smoothing functions. The Hodrick-Prescott filter separates a time-series $yt$ into a trend $τt$ and a cyclical component $ζt$. The Christiano-Fitzgerald filter is a generalization of Baxter-King filter and can be seen as weighted moving average."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hvmf2zLJJpzH",
        "colab_type": "text"
      },
      "source": [
        "(i) Baxter-King Bandpass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2MQ3fUokKotb",
        "colab_type": "text"
      },
      "source": [
        "The Baxter-King filter is intended to explicitly deal with the periodicity of the business cycle. By applying their band-pass filter to a series, they produce a new series that does not contain fluctuations at higher or lower than those of the business cycle. The parameters are arbitarily chosen. This method uses a centred moving average that has to be changed to a lagged moving average before it can be used as an input feature. The maximum period of oscillation should be used as the point to truncate the dataset, as that part of the time series does not incorporate all the required datapoints."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "srPrsCq5-SJB",
        "colab_type": "code",
        "outputId": "619245e2-3ee6-4b6d-e97b-24286975d2ac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "import statsmodels.api as sm\n",
        "\n",
        "def bkb(df, cols):\n",
        "  for col in cols:\n",
        "    df[col+\"_BPF\"] = sm.tsa.filters.bkfilter(df[[col]].values, 2, 10, len(df)-1)\n",
        "  return df\n",
        "\n",
        "df_out = transform.bkb(df.copy(), [\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_BPF</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>125.901279</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>55.188438</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>19.933900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>-1.973850</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>-25.055888</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...     Close_1   Close_BPF\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...  300.359985  125.901279\n",
              "2019-01-03  309.399994  297.380005  ...  317.690002   55.188438\n",
              "2019-01-04  318.000000  302.730011  ...  334.959991   19.933900\n",
              "2019-01-07  336.739990  317.750000  ...  335.350006   -1.973850\n",
              "2019-01-08  344.010010  327.019989  ...  338.529999  -25.055888\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqXtTLHPfeKg",
        "colab_type": "text"
      },
      "source": [
        "(ii) Butter Lowpass (IIR Filter Design)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ETiOoBNif44f",
        "colab_type": "text"
      },
      "source": [
        "The Butterworth filter is a type of signal processing filter designed to have a frequency response as flat as possible in the passban. Like other filtersm the first few values have to be disregarded for accurate downstream prediction. Instead of disregarding these values on a per case basis, they can be diregarded in one chunk once the database of transformed features have been developed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kSbKKwqnfhpB",
        "colab_type": "code",
        "outputId": "14259c4e-44f2-4625-a76a-61eb770b86da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from scipy import signal, integrate\n",
        "def butter_lowpass(cutoff, fs=20, order=5):\n",
        "    nyq = 0.5 * fs\n",
        "    normal_cutoff = cutoff / nyq\n",
        "    b, a = signal.butter(order, normal_cutoff, btype='low', analog=False)\n",
        "    return b, a\n",
        "    \n",
        "def butter_lowpass_filter(df,cols, cutoff, fs=20, order=5):\n",
        "    b, a = butter_lowpass(cutoff, fs, order=order)\n",
        "    for col in cols:\n",
        "      df[col+\"_BUTTER\"] = signal.lfilter(b, a, df[col])\n",
        "    return df\n",
        "\n",
        "df_out = transform.butter_lowpass_filter(df.copy(),[\"Close\"],4); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_BUTTER</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6.803915</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>47.313429</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>147.951197</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>278.468513</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>361.542710</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...     Close_1  Close_BUTTER\n",
              "Date                                ...                          \n",
              "2019-01-02  315.130005  298.799988  ...  300.359985      6.803915\n",
              "2019-01-03  309.399994  297.380005  ...  317.690002     47.313429\n",
              "2019-01-04  318.000000  302.730011  ...  334.959991    147.951197\n",
              "2019-01-07  336.739990  317.750000  ...  335.350006    278.468513\n",
              "2019-01-08  344.010010  327.019989  ...  338.529999    361.542710\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iWXwxaDiibax",
        "colab_type": "text"
      },
      "source": [
        "(iii) Hilbert Transform Angle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBGtibY4GN3r",
        "colab_type": "text"
      },
      "source": [
        "The Hilbert transform is a time-domain to time-domain transformation which shifts the phase of a signal by 90 degrees. It is also a centred measure and would be difficult to use in a time series prediction setting, unless it is recalculated on a per step basis or transformed to be based on historical values only."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IDYHXqOTiSy9",
        "colab_type": "code",
        "outputId": "d0685a90-93f6-4208-d10c-96e3c285920a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from scipy import signal\n",
        "import numpy as np\n",
        "\n",
        "def instantaneous_phases(df,cols):\n",
        "    for col in cols:\n",
        "      df[col+\"_HILLB\"] = np.unwrap(np.angle(signal.hilbert(df[col], axis=0)), axis=0)\n",
        "    return df\n",
        "\n",
        "df_out = transform.instantaneous_phases(df.copy(), [\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_HILLB</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.375961</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.177702</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>0.105221</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>0.092816</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>0.105906</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...     Close_1  Close_HILLB\n",
              "Date                                ...                         \n",
              "2019-01-02  315.130005  298.799988  ...  300.359985     0.375961\n",
              "2019-01-03  309.399994  297.380005  ...  317.690002     0.177702\n",
              "2019-01-04  318.000000  302.730011  ...  334.959991     0.105221\n",
              "2019-01-07  336.739990  317.750000  ...  335.350006     0.092816\n",
              "2019-01-08  344.010010  327.019989  ...  338.529999     0.105906\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rw9AWm4fxMXO",
        "colab_type": "text"
      },
      "source": [
        "(iiiv) Unscented Kalman Filter\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Ba3dwAxxIin",
        "colab_type": "text"
      },
      "source": [
        "The Kalman filter is better suited for estimating things that change over time. The most tangible example is tracking moving objects. A Kalman filter will be very close to the actual trajectory because it says the most recent measurement is more important than the older ones. The Unscented Kalman Filter (UKF) is a model based-techniques that recursively estimates the states (and with some modifications also parameters) of a nonlinear, dynamic, discrete-time system. The UKF is based on the typical prediction-correction style methods. The Kalman Smoother incorporates future values, the Filter doesn't and can be used for online prediction. The normal Kalman filter is a forward filter in the sense that it makes forecast of the current state using only current and past observations, whereas the smoother is based on computing a suitable linear combination of two filters, which are ran in forward and backward directions. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Vi7ickulD2H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1nHEPr9hw-tg",
        "colab_type": "code",
        "outputId": "f38f1725-52b4-4c87-e826-28daf304f948",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from pykalman import UnscentedKalmanFilter\n",
        "\n",
        "def kalman_feat(df, cols):\n",
        "  for col in cols:\n",
        "    ukf = UnscentedKalmanFilter(lambda x, w: x + np.sin(w), lambda x, v: x + v, observation_covariance=0.1)\n",
        "    (filtered_state_means, filtered_state_covariances) = ukf.filter(df[col])\n",
        "    (smoothed_state_means, smoothed_state_covariances) = ukf.smooth(df[col])\n",
        "    df[col+\"_UKFSMOOTH\"] = smoothed_state_means.flatten()\n",
        "    df[col+\"_UKFFILTER\"] = filtered_state_means.flatten()\n",
        "  return df \n",
        "\n",
        "df_out = transform.kalman_feat(df.copy(), [\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_UKFSMOOTH</th>\n",
              "      <th>Close_UKFFILTER</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>286.062007</td>\n",
              "      <td>281.927268</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>300.831896</td>\n",
              "      <td>296.785326</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>317.134269</td>\n",
              "      <td>313.553309</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>331.631951</td>\n",
              "      <td>330.720643</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>335.322139</td>\n",
              "      <td>334.433186</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Close_UKFSMOOTH  Close_UKFFILTER\n",
              "Date                                ...                                  \n",
              "2019-01-02  315.130005  298.799988  ...       286.062007       281.927268\n",
              "2019-01-03  309.399994  297.380005  ...       300.831896       296.785326\n",
              "2019-01-04  318.000000  302.730011  ...       317.134269       313.553309\n",
              "2019-01-07  336.739990  317.750000  ...       331.631951       330.720643\n",
              "2019-01-08  344.010010  327.019989  ...       335.322139       334.433186\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jC_r_sZTgKxM",
        "colab_type": "text"
      },
      "source": [
        "#### **(9) Spectral Analysis**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZHtFFAsCGv4b",
        "colab_type": "text"
      },
      "source": [
        "There are a range of functions for spectral analysis. You can use periodograms and the welch method to estimate the power spectral density. You can also use the welch method to estimate the cross power spectral density. Other techniques include spectograms, Lomb-Scargle periodograms and, short time fourier transform."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zSfNGaZuQwx4",
        "colab_type": "text"
      },
      "source": [
        "(i) Periodogram"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jv_QubqtWGws",
        "colab_type": "text"
      },
      "source": [
        "This returns an array of sample frequencies and the power spectrum of x, or the power spectral density of x."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ip_ogGyTUmkL",
        "colab_type": "code",
        "outputId": "441e9d62-d9ad-4a24-9be1-5b1870d21c7d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from scipy import signal\n",
        "def perd_feat(df, cols):\n",
        "  for col in cols:\n",
        "    sig = signal.periodogram(df[col],fs=1, return_onesided=False)\n",
        "    df[col+\"_FREQ\"] = sig[0]\n",
        "    df[col+\"_POWER\"] = sig[1]\n",
        "  return df\n",
        "\n",
        "df_out = transform.perd_feat(df.copy(),[\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_FREQ</th>\n",
              "      <th>Close_POWER</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>3.154257e-26</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.003984</td>\n",
              "      <td>2.577125e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>0.007968</td>\n",
              "      <td>1.961317e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>0.011952</td>\n",
              "      <td>2.486299e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>0.015936</td>\n",
              "      <td>1.925430e+03</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Close_FREQ   Close_POWER\n",
              "Date                                ...                          \n",
              "2019-01-02  315.130005  298.799988  ...    0.000000  3.154257e-26\n",
              "2019-01-03  309.399994  297.380005  ...    0.003984  2.577125e+05\n",
              "2019-01-04  318.000000  302.730011  ...    0.007968  1.961317e+04\n",
              "2019-01-07  336.739990  317.750000  ...    0.011952  2.486299e+04\n",
              "2019-01-08  344.010010  327.019989  ...    0.015936  1.925430e+03\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yrT2Uz-GLz_h",
        "colab_type": "text"
      },
      "source": [
        "(ii) Fast Fourier Transform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3aufHL_NeVX",
        "colab_type": "text"
      },
      "source": [
        "The FFT, or fast fourier transform is an algorithm that essentially uses convolution techniques to efficiently find the magnitude and location of the tones that make up the signal of interest. We can often play with the FFT spectrum, by adding and removing successive tones (which is akin to selectively filtering particular tones that make up the signal), in order to obtain a smoothed version of the underlying signal. This takes the entire signal into account, and as a result has to be recalculated on a running basis to avoid peaking into the future. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vV7_YupNLdTX",
        "colab_type": "code",
        "outputId": "72a6b1f3-7c92-4d92-812f-c2f8c0f7eba3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "def fft_feat(df, cols):\n",
        "  for col in cols:\n",
        "    fft_df = np.fft.fft(np.asarray(df[col].tolist()))\n",
        "    fft_df = pd.DataFrame({'fft':fft_df})\n",
        "    df[col+'_FFTABS'] = fft_df['fft'].apply(lambda x: np.abs(x)).values\n",
        "    df[col+'_FFTANGLE'] = fft_df['fft'].apply(lambda x: np.angle(x)).values\n",
        "  return df \n",
        "\n",
        "df_out = transform.fft_feat(df.copy(), [\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_FFTABS</th>\n",
              "      <th>Close_FFTANGLE</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>68511.279968</td>\n",
              "      <td>1.908297e-17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>8042.750048</td>\n",
              "      <td>1.841344e-02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>2218.762153</td>\n",
              "      <td>3.468148e-01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>2498.121511</td>\n",
              "      <td>1.576258e+00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>695.185569</td>\n",
              "      <td>-4.840852e-01</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Close_FFTABS  Close_FFTANGLE\n",
              "Date                                ...                              \n",
              "2019-01-02  315.130005  298.799988  ...  68511.279968    1.908297e-17\n",
              "2019-01-03  309.399994  297.380005  ...   8042.750048    1.841344e-02\n",
              "2019-01-04  318.000000  302.730011  ...   2218.762153    3.468148e-01\n",
              "2019-01-07  336.739990  317.750000  ...   2498.121511    1.576258e+00\n",
              "2019-01-08  344.010010  327.019989  ...    695.185569   -4.840852e-01\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vaRlVD27WsZE",
        "colab_type": "text"
      },
      "source": [
        "#### **(10) Waveforms**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tjoxns0uJVeT",
        "colab_type": "text"
      },
      "source": [
        "The waveform of a signal is the shape of its graph as a function of time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iWe0YR25Wt--",
        "colab_type": "text"
      },
      "source": [
        "(i) Continuous Wave Radar"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eG-OWzi4I5k0",
        "colab_type": "code",
        "outputId": "934ee858-2672-4a61-8af9-42d9c7b548cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from scipy import signal\n",
        "def harmonicradar_cw(df, cols, fs,fc):\n",
        "    for col in cols:\n",
        "      ttxt = f'CW: {fc} Hz'\n",
        "      #%% input\n",
        "      t = df[col]\n",
        "      tx = np.sin(2*np.pi*fc*t)\n",
        "      _,Pxx = signal.welch(tx,fs)\n",
        "      #%% diode\n",
        "      d = (signal.square(2*np.pi*fc*t))\n",
        "      d[d<0] = 0.\n",
        "      #%% output of diode\n",
        "      rx = tx * d\n",
        "      df[col+\"_HARRAD\"] = rx.values\n",
        "    return df\n",
        "\n",
        "df_out = transform.harmonicradar_cw(df.copy(), [\"Close\"],0.3,0.2); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_HARRAD</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.150220</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.437099</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>-0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>-0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>0.425786</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...     Close_1  Close_HARRAD\n",
              "Date                                ...                          \n",
              "2019-01-02  315.130005  298.799988  ...  300.359985      0.150220\n",
              "2019-01-03  309.399994  297.380005  ...  317.690002      0.437099\n",
              "2019-01-04  318.000000  302.730011  ...  334.959991     -0.000000\n",
              "2019-01-07  336.739990  317.750000  ...  335.350006     -0.000000\n",
              "2019-01-08  344.010010  327.019989  ...  338.529999      0.425786\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3K5hGPl5hZUh",
        "colab_type": "text"
      },
      "source": [
        "(ii) Saw Tooth"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QiTd5lXihct7",
        "colab_type": "text"
      },
      "source": [
        "Return a periodic sawtooth or triangle waveform."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Su3-xPB0hRUr",
        "colab_type": "code",
        "outputId": "5241d533-d6a7-469f-d03f-6ba65ec92ec9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "def saw(df, cols):\n",
        "  for col in cols:\n",
        "    df[col+\" SAW\"] = signal.sawtooth(df[col])\n",
        "  return df\n",
        "\n",
        "df_out = transform.saw(df.copy(),[\"Close\",\"Open\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close SAW</th>\n",
              "      <th>Open SAW</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>-0.285740</td>\n",
              "      <td>0.434658</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.607553</td>\n",
              "      <td>0.721135</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>0.123869</td>\n",
              "      <td>0.402825</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>-0.378923</td>\n",
              "      <td>-0.593343</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>-0.254778</td>\n",
              "      <td>-0.150754</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low        Open  ...     Close_1  Close SAW  Open SAW\n",
              "Date                                            ...                                 \n",
              "2019-01-02  315.130005  298.799988  306.100006  ...  300.359985  -0.285740  0.434658\n",
              "2019-01-03  309.399994  297.380005  307.000000  ...  317.690002   0.607553  0.721135\n",
              "2019-01-04  318.000000  302.730011  306.000000  ...  334.959991   0.123869  0.402825\n",
              "2019-01-07  336.739990  317.750000  321.720001  ...  335.350006  -0.378923 -0.593343\n",
              "2019-01-08  344.010010  327.019989  341.959991  ...  338.529999  -0.254778 -0.150754\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "me3nleaTNsa8",
        "colab_type": "text"
      },
      "source": [
        "##### **(9) Modifications**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wFoXkxj03ov-",
        "colab_type": "text"
      },
      "source": [
        "A range of modification usually applied ot images, these values would have to be recalculate for each time-series. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QDTmUAmHRCvf",
        "colab_type": "text"
      },
      "source": [
        "(i) Various Techniques"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rm1xfEnaLdPy",
        "colab_type": "code",
        "outputId": "a9babe57-1788-49ce-ded5-6f5a14c8f49d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from tsaug import *\n",
        "def modify(df, cols):\n",
        "  for col in cols:\n",
        "    series = df[col].values\n",
        "    df[col+\"_magnify\"], _ = magnify(series, series)\n",
        "    df[col+\"_affine\"], _ = affine(series, series)\n",
        "    df[col+\"_crop\"], _ = crop(series, series)\n",
        "    df[col+\"_cross_sum\"], _ = cross_sum(series, series)\n",
        "    df[col+\"_resample\"], _ = resample(series, series)\n",
        "    df[col+\"_trend\"], _ = trend(series, series)\n",
        "\n",
        "    df[col+\"_random_affine\"], _ = random_time_warp(series, series)\n",
        "    df[col+\"_random_crop\"], _ = random_crop(series, series)\n",
        "    df[col+\"_random_cross_sum\"], _ = random_cross_sum(series, series)\n",
        "    df[col+\"_random_sidetrack\"], _ = random_sidetrack(series, series)\n",
        "    df[col+\"_random_time_warp\"], _ = random_time_warp(series, series)\n",
        "    df[col+\"_random_magnify\"], _ = random_magnify(series, series)\n",
        "    df[col+\"_random_jitter\"], _ = random_jitter(series, series)\n",
        "    df[col+\"_random_trend\"], _ = random_trend(series, series)\n",
        "  return df\n",
        "\n",
        "df_out = transform.modify(df.copy(),[\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_magnify</th>\n",
              "      <th>Close_affine</th>\n",
              "      <th>Close_crop</th>\n",
              "      <th>Close_cross_sum</th>\n",
              "      <th>Close_resample</th>\n",
              "      <th>Close_trend</th>\n",
              "      <th>Close_random_affine</th>\n",
              "      <th>Close_random_crop</th>\n",
              "      <th>Close_random_cross_sum</th>\n",
              "      <th>Close_random_sidetrack</th>\n",
              "      <th>Close_random_time_warp</th>\n",
              "      <th>Close_random_magnify</th>\n",
              "      <th>Close_random_jitter</th>\n",
              "      <th>Close_random_trend</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>1860.719971</td>\n",
              "      <td>272.737686</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>305.452808</td>\n",
              "      <td>397.898581</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>306.785424</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>1802.159912</td>\n",
              "      <td>299.909458</td>\n",
              "      <td>303.758285</td>\n",
              "      <td>335.345326</td>\n",
              "      <td>297.198462</td>\n",
              "      <td>387.227141</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>303.327624</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>1906.140015</td>\n",
              "      <td>301.105842</td>\n",
              "      <td>305.764026</td>\n",
              "      <td>338.453679</td>\n",
              "      <td>318.834848</td>\n",
              "      <td>403.641836</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>301.446488</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>2009.759949</td>\n",
              "      <td>280.498279</td>\n",
              "      <td>317.344213</td>\n",
              "      <td>344.738161</td>\n",
              "      <td>333.805513</td>\n",
              "      <td>419.992790</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>308.015894</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>2012.100037</td>\n",
              "      <td>216.958366</td>\n",
              "      <td>329.026975</td>\n",
              "      <td>347.150089</td>\n",
              "      <td>339.894193</td>\n",
              "      <td>419.460233</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Close_random_jitter  Close_random_trend\n",
              "Date                                ...                                         \n",
              "2019-01-02  315.130005  298.799988  ...           305.452808          397.898581\n",
              "2019-01-03  309.399994  297.380005  ...           297.198462          387.227141\n",
              "2019-01-04  318.000000  302.730011  ...           318.834848          403.641836\n",
              "2019-01-07  336.739990  317.750000  ...           333.805513          419.992790\n",
              "2019-01-08  344.010010  327.019989  ...           339.894193          419.460233\n",
              "\n",
              "[5 rows x 21 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ql_2y4TQkp-3",
        "colab_type": "text"
      },
      "source": [
        "#### **(11) Rolling**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V4qNje1SLuxu",
        "colab_type": "text"
      },
      "source": [
        "Features that are calculated on a rolling basis over fixed window size."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eyQUbB74kvQI",
        "colab_type": "text"
      },
      "source": [
        "(i) Mean, Standard Deviation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WtR8RHQJk1Wv",
        "colab_type": "code",
        "outputId": "36ec4e02-7d92-4bea-d21b-e62ddcc52fb1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "def multiple_rolling(df, windows = [1,2], functions=[\"mean\",\"std\"], columns=None):\n",
        "  windows = [1+a for a in windows]\n",
        "  if not columns:\n",
        "    columns = df.columns.to_list()\n",
        "  rolling_dfs = (df[columns].rolling(i)                                    # 1. Create window\n",
        "                  .agg(functions)                                # 1. Aggregate\n",
        "                  .rename({col: '{0}_{1:d}'.format(col, i)\n",
        "                                for col in columns}, axis=1)  # 2. Rename columns\n",
        "                for i in windows)                                # For each window\n",
        "  df_out = pd.concat((df, *rolling_dfs), axis=1)\n",
        "  da = df_out.iloc[:,len(df.columns):]\n",
        "  da = [col[0] + \"_\" + col[1] for col in  da.columns.to_list()]\n",
        "  df_out.columns = df.columns.to_list() + da \n",
        "\n",
        "  return  df_out                      # 3. Concatenate dataframes\n",
        "\n",
        "df_out = transform.multiple_rolling(df, columns=[\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_2_mean</th>\n",
              "      <th>Close_2_std</th>\n",
              "      <th>Close_3_mean</th>\n",
              "      <th>Close_3_std</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>305.239990</td>\n",
              "      <td>6.901369</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>309.024994</td>\n",
              "      <td>12.254173</td>\n",
              "      <td>309.389994</td>\n",
              "      <td>8.688041</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>326.324997</td>\n",
              "      <td>12.211726</td>\n",
              "      <td>317.669993</td>\n",
              "      <td>17.300012</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>335.154999</td>\n",
              "      <td>0.275782</td>\n",
              "      <td>329.333333</td>\n",
              "      <td>10.085306</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Close_3_mean  Close_3_std\n",
              "Date                                ...                           \n",
              "2019-01-02  315.130005  298.799988  ...           NaN          NaN\n",
              "2019-01-03  309.399994  297.380005  ...           NaN          NaN\n",
              "2019-01-04  318.000000  302.730011  ...    309.389994     8.688041\n",
              "2019-01-07  336.739990  317.750000  ...    317.669993    17.300012\n",
              "2019-01-08  344.010010  327.019989  ...    329.333333    10.085306\n",
              "\n",
              "[5 rows x 11 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gMiwZhKPliap",
        "colab_type": "text"
      },
      "source": [
        "#### **(12) Lagging**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vbePK7eL7oW",
        "colab_type": "text"
      },
      "source": [
        "Lagged values from existing features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xholFpghlsJd",
        "colab_type": "text"
      },
      "source": [
        "(i) Single Steps"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5ysGNGyxLc8x",
        "colab_type": "code",
        "outputId": "8d2aef8f-9bdf-4262-8ad5-b77b8082596a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "def multiple_lags(df, start=1, end=3,columns=None):\n",
        "  if not columns:\n",
        "    columns = df.columns.to_list()\n",
        "  lags = range(start, end+1)  # Just two lags for demonstration.\n",
        "\n",
        "  df = df.assign(**{\n",
        "      '{}_t_{}'.format(col, t): df[col].shift(t)\n",
        "      for t in lags\n",
        "      for col in columns\n",
        "  })\n",
        "  return df\n",
        "\n",
        "df_out = transform.multiple_lags(df, start=1, end=3, columns=[\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_t_1</th>\n",
              "      <th>Close_t_2</th>\n",
              "      <th>Close_t_3</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.119995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>300.359985</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...   Close_t_2   Close_t_3\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...         NaN         NaN\n",
              "2019-01-03  309.399994  297.380005  ...         NaN         NaN\n",
              "2019-01-04  318.000000  302.730011  ...  310.119995         NaN\n",
              "2019-01-07  336.739990  317.750000  ...  300.359985  310.119995\n",
              "2019-01-08  344.010010  327.019989  ...  317.690002  300.359985\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iL7iDIYmn-NB",
        "colab_type": "text"
      },
      "source": [
        "#### **(13) Forecast Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LGYuxaIiRchZ",
        "colab_type": "text"
      },
      "source": [
        "There are a range of time series model that can be implemented like AR, MA, ARMA, ARIMA, SARIMA, SARIMAX, VAR, VARMA, VARMAX, SES, and HWES. The models can be divided into autoregressive models and smoothing models. In an autoregression model, we forecast the variable of interest using a linear combination of past values of the variable. Each method might requre specific tuning and parameters to suit your prediction task. You need to drop a certain amount of historical data that you use during the fitting stage. Models that take seasonaity into account need more training data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aiwrmNd-oIGu",
        "colab_type": "text"
      },
      "source": [
        "(i) Prophet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_RfRxhs9RNDV",
        "colab_type": "text"
      },
      "source": [
        "Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality. You can apply additive models to your training data but also interactive models like deep learning models. The problem is that because these models have learned from future observations, there would this be a need to recalculate the time series on a running basis, or to only include the predicted as opposed to fitted values in future training and test sets. In this example, I train on 150 data points to illustrate how the remaining or so 100 datapoints can be used in a new prediction problem. You can plot with ```df[\"PROPHET\"].plot()``` to see the effect."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5nq7aBa6Pdt",
        "colab_type": "text"
      },
      "source": [
        "You can apply additive models to your training data but also interactive models like deep learning models. The problem is that these models have learned from future observations, there would this be a need to recalculate the time series on a running basis, or to only include the predicted as opposed to fitted values in future training and test sets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puLg0W9KLcvX",
        "colab_type": "code",
        "outputId": "ff298f1f-be09-47d1-a79d-a7fba662e6f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        }
      },
      "source": [
        "from fbprophet import Prophet\n",
        "\n",
        "def prophet_feat(df, cols,date, freq,train_size=150):\n",
        "  def prophet_dataframe(df): \n",
        "    df.columns = ['ds','y']\n",
        "    return df\n",
        "\n",
        "  def original_dataframe(df, freq, name):\n",
        "    prophet_pred = pd.DataFrame({\"Date\" : df['ds'], name : df[\"yhat\"]})\n",
        "    prophet_pred = prophet_pred.set_index(\"Date\")\n",
        "    #prophet_pred.index.freq = pd.tseries.frequencies.to_offset(freq)\n",
        "    return prophet_pred[name].values\n",
        "\n",
        "  for col in cols:\n",
        "    model = Prophet(daily_seasonality=True)\n",
        "    fb = model.fit(prophet_dataframe(df[[date, col]].head(train_size)))\n",
        "    forecast_len = len(df) - train_size\n",
        "    future = model.make_future_dataframe(periods=forecast_len,freq=freq)\n",
        "    future_pred = model.predict(future)\n",
        "    df[col+\"_PROPHET\"] = list(original_dataframe(future_pred,freq,col))\n",
        "  return df\n",
        "\n",
        "df_out  = transform.prophet_feat(df.copy().reset_index(),[\"Close\",\"Open\"],\"Date\", \"D\"); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:fbprophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.\n",
            "INFO:fbprophet:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_PROPHET</th>\n",
              "      <th>Open_PROPHET</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-01-02</td>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>334.137987</td>\n",
              "      <td>332.311005</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2019-01-03</td>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>330.850212</td>\n",
              "      <td>329.519669</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2019-01-04</td>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>328.162444</td>\n",
              "      <td>326.904573</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2019-01-07</td>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>329.410732</td>\n",
              "      <td>327.373098</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2019-01-08</td>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>328.671778</td>\n",
              "      <td>326.535886</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        Date        High         Low  ...     Close_1  Close_PROPHET  Open_PROPHET\n",
              "0 2019-01-02  315.130005  298.799988  ...  300.359985     334.137987    332.311005\n",
              "1 2019-01-03  309.399994  297.380005  ...  317.690002     330.850212    329.519669\n",
              "2 2019-01-04  318.000000  302.730011  ...  334.959991     328.162444    326.904573\n",
              "3 2019-01-07  336.739990  317.750000  ...  335.350006     329.410732    327.373098\n",
              "4 2019-01-08  344.010010  327.019989  ...  338.529999     328.671778    326.535886\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UpFnjnDC-JGd",
        "colab_type": "text"
      },
      "source": [
        "<a name=\"interaction\"></a>\n",
        "\n",
        "## **(2) Interaction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhxCZABeECK1",
        "colab_type": "text"
      },
      "source": [
        "Interactions are defined as methods that require more than one feature to create an additional feature. Here we include normalising and discretising techniques that are non-feature specific. Almost all of these method can be applied to cross-section method. The only methods that are time specific is the technical features in the speciality section and the autoregression model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0cpNd_I8K2_H",
        "colab_type": "text"
      },
      "source": [
        "#### **(1) Regression**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y8Bb0eyc0YWM",
        "colab_type": "text"
      },
      "source": [
        "Regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the 'outcome variable') and one or more independent variables. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ajJZKMuK82o",
        "colab_type": "text"
      },
      "source": [
        "(i) Lowess Smoother"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "poBfhmZx0iFl",
        "colab_type": "text"
      },
      "source": [
        "The lowess smoother is a robust locally weighted regression. The function fits a nonparametric regression curve to a scatterplot."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lu8K4hziK64M",
        "colab_type": "code",
        "outputId": "21d963ca-c816-4389-c274-4c8597e3557d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from math import ceil\n",
        "import numpy as np\n",
        "from scipy import linalg\n",
        "import math\n",
        "\n",
        "def lowess(df, cols, y, f=2. / 3., iter=3):\n",
        "    for col in cols:\n",
        "      n = len(df[col])\n",
        "      r = int(ceil(f * n))\n",
        "      h = [np.sort(np.abs(df[col] - df[col][i]))[r] for i in range(n)]\n",
        "      w = np.clip(np.abs((df[col][:, None] - df[col][None, :]) / h), 0.0, 1.0)\n",
        "      w = (1 - w ** 3) ** 3\n",
        "      yest = np.zeros(n)\n",
        "      delta = np.ones(n)\n",
        "      for iteration in range(iter):\n",
        "          for i in range(n):\n",
        "              weights = delta * w[:, i]\n",
        "              b = np.array([np.sum(weights * y), np.sum(weights * y * df[col])])\n",
        "              A = np.array([[np.sum(weights), np.sum(weights * df[col])],\n",
        "                            [np.sum(weights * df[col]), np.sum(weights * df[col] * df[col])]])\n",
        "              beta = linalg.solve(A, b)\n",
        "              yest[i] = beta[0] + beta[1] * df[col][i]\n",
        "\n",
        "          residuals = y - yest\n",
        "          s = np.median(np.abs(residuals))\n",
        "          delta = np.clip(residuals / (6.0 * s), -1, 1)\n",
        "          delta = (1 - delta ** 2) ** 2\n",
        "      df[col+\"_LOWESS\"] = yest\n",
        "\n",
        "    return df\n",
        "\n",
        "df_out = interact.lowess(df.copy(), [\"Open\",\"Volume\"], df[\"Close\"], f=0.25, iter=3); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Open_LOWESS</th>\n",
              "      <th>Volume_LOWESS</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>306.754232</td>\n",
              "      <td>262.428115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>307.634264</td>\n",
              "      <td>280.663698</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>306.657412</td>\n",
              "      <td>282.080236</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>322.512505</td>\n",
              "      <td>277.757062</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>342.495142</td>\n",
              "      <td>280.819244</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Open_LOWESS  Volume_LOWESS\n",
              "Date                                ...                            \n",
              "2019-01-02  315.130005  298.799988  ...   306.754232     262.428115\n",
              "2019-01-03  309.399994  297.380005  ...   307.634264     280.663698\n",
              "2019-01-04  318.000000  302.730011  ...   306.657412     282.080236\n",
              "2019-01-07  336.739990  317.750000  ...   322.512505     277.757062\n",
              "2019-01-08  344.010010  327.019989  ...   342.495142     280.819244\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qoZXyzuTcsxe",
        "colab_type": "text"
      },
      "source": [
        "Autoregression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Un2fLOWUKHa",
        "colab_type": "text"
      },
      "source": [
        "Autoregression is a time series model that uses observations from previous time steps as input to a regression equation to predict the value at the next time step"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sGW-D92RcwPo",
        "colab_type": "code",
        "outputId": "db73d5c5-b200-4bbd-f0e1-3c30bfa34c3c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from statsmodels.tsa.ar_model import AR\n",
        "from timeit import default_timer as timer\n",
        "def autoregression(df, drop=None, settings={\"autoreg_lag\":4}):\n",
        "\n",
        "    autoreg_lag = settings[\"autoreg_lag\"]\n",
        "    if drop:\n",
        "      keep = df[drop]\n",
        "      df = df.drop([drop],axis=1).values\n",
        "\n",
        "    n_channels = df.shape[0]\n",
        "    t = timer()\n",
        "    channels_regg = np.zeros((n_channels, autoreg_lag + 1))\n",
        "    for i in range(0, n_channels):\n",
        "        fitted_model = AR(df.values[i, :]).fit(autoreg_lag)\n",
        "        # TODO: This is not the same as Matlab's for some reasons!\n",
        "        # kk = ARMAResults(fitted_model)\n",
        "        # autore_vals, dummy1, dummy2 = arburg(x[i, :], autoreg_lag) # This looks like Matlab's but slow\n",
        "        channels_regg[i, 0: len(fitted_model.params)] = np.real(fitted_model.params)\n",
        "\n",
        "    for i in range(channels_regg.shape[1]):\n",
        "      df[\"LAG_\"+str(i+1)] = channels_regg[:,i]\n",
        "    \n",
        "    if drop:\n",
        "      df = pd.concat((keep,df),axis=1)\n",
        "\n",
        "    t = timer() - t\n",
        "    return df\n",
        "\n",
        "df_out = interact.autoregression(df.copy()); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>LAG_1</th>\n",
              "      <th>LAG_2</th>\n",
              "      <th>LAG_3</th>\n",
              "      <th>LAG_4</th>\n",
              "      <th>LAG_5</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>61.823258</td>\n",
              "      <td>-0.984277</td>\n",
              "      <td>-1.002847</td>\n",
              "      <td>18472.758360</td>\n",
              "      <td>19482.405117</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>37.823878</td>\n",
              "      <td>-0.995340</td>\n",
              "      <td>-1.000778</td>\n",
              "      <td>11248.044227</td>\n",
              "      <td>11702.731091</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>38.360199</td>\n",
              "      <td>-0.979941</td>\n",
              "      <td>-1.003694</td>\n",
              "      <td>11612.754350</td>\n",
              "      <td>12198.587534</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>35.229735</td>\n",
              "      <td>-0.976048</td>\n",
              "      <td>-1.001914</td>\n",
              "      <td>11194.216220</td>\n",
              "      <td>11863.312809</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>31.112372</td>\n",
              "      <td>-0.995743</td>\n",
              "      <td>-1.008963</td>\n",
              "      <td>10174.332643</td>\n",
              "      <td>10702.995789</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...         LAG_4         LAG_5\n",
              "Date                                ...                            \n",
              "2019-01-02  315.130005  298.799988  ...  18472.758360  19482.405117\n",
              "2019-01-03  309.399994  297.380005  ...  11248.044227  11702.731091\n",
              "2019-01-04  318.000000  302.730011  ...  11612.754350  12198.587534\n",
              "2019-01-07  336.739990  317.750000  ...  11194.216220  11863.312809\n",
              "2019-01-08  344.010010  327.019989  ...  10174.332643  10702.995789\n",
              "\n",
              "[5 rows x 12 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9YQsVZl2-5ZV",
        "colab_type": "text"
      },
      "source": [
        "####  **(2) Operator**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1dCS1KyYUbDq",
        "colab_type": "text"
      },
      "source": [
        "Looking at interaction between different features. Here the methods employed are multiplication and division."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jCUrkift_Cvd",
        "colab_type": "text"
      },
      "source": [
        "(i) Multiplication and Division"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eAB47GRx0Rma",
        "colab_type": "code",
        "outputId": "59def3f6-906a-42c0-e436-111e2a896f13",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "def muldiv(df, feature_list):\n",
        "  for feat in feature_list:\n",
        "    for feat_two in feature_list:\n",
        "      if feat==feat_two:\n",
        "        continue\n",
        "      else:\n",
        "       df[feat+\"/\"+feat_two] = df[feat]/(df[feat_two]-df[feat_two].min()) #zero division guard\n",
        "       df[feat+\"_X_\"+feat_two] = df[feat]*(df[feat_two])\n",
        "\n",
        "  return df\n",
        "\n",
        "df_out = interact.muldiv(df.copy(), [\"Close\",\"Open\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close/Open</th>\n",
              "      <th>Close_X_Open</th>\n",
              "      <th>Open/Close</th>\n",
              "      <th>Open_X_Close</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>2.480960</td>\n",
              "      <td>94927.732398</td>\n",
              "      <td>2.333969</td>\n",
              "      <td>94927.732398</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>2.385703</td>\n",
              "      <td>92210.515503</td>\n",
              "      <td>2.529039</td>\n",
              "      <td>92210.515503</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>2.543555</td>\n",
              "      <td>97213.140747</td>\n",
              "      <td>2.205882</td>\n",
              "      <td>97213.140747</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>2.382022</td>\n",
              "      <td>107763.328860</td>\n",
              "      <td>2.062440</td>\n",
              "      <td>107763.328860</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>2.084732</td>\n",
              "      <td>114676.285222</td>\n",
              "      <td>2.186725</td>\n",
              "      <td>114676.285222</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  Open/Close   Open_X_Close\n",
              "Date                                ...                           \n",
              "2019-01-02  315.130005  298.799988  ...    2.333969   94927.732398\n",
              "2019-01-03  309.399994  297.380005  ...    2.529039   92210.515503\n",
              "2019-01-04  318.000000  302.730011  ...    2.205882   97213.140747\n",
              "2019-01-07  336.739990  317.750000  ...    2.062440  107763.328860\n",
              "2019-01-08  344.010010  327.019989  ...    2.186725  114676.285222\n",
              "\n",
              "[5 rows x 11 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bCVACKWz91WS",
        "colab_type": "text"
      },
      "source": [
        "#### **(3) Discretising**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sQh4QqxEUwyf",
        "colab_type": "text"
      },
      "source": [
        "In statistics and machine learning, discretization refers to the process of converting or partitioning continuous attributes, features or variables to discretized or nominal attributes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uro31izJzAZf",
        "colab_type": "text"
      },
      "source": [
        "(i) Decision Tree Discretiser\n",
        "\n",
        "The first method that will be applies here is a supersived discretiser. Discretisation with Decision Trees consists of using a decision tree to identify the optimal splitting points that would determine the bins or contiguous intervals."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZATSQF5pztkz",
        "colab_type": "code",
        "outputId": "b060880d-2cf0-4966-c704-b5ffd854bfa3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from sklearn.tree import DecisionTreeRegressor\n",
        "\n",
        "def decision_tree_disc(df, cols, depth=4 ):\n",
        "  for col in cols:\n",
        "    df[col +\"_m1\"] = df[col].shift(1)\n",
        "    df = df.iloc[1:,:]\n",
        "    tree_model = DecisionTreeRegressor(max_depth=depth,random_state=0)\n",
        "    tree_model.fit(df[col +\"_m1\"].to_frame(), df[col])\n",
        "    df[col+\"_Disc\"] = tree_model.predict(df[col +\"_m1\"].to_frame())\n",
        "  return df\n",
        "\n",
        "df_out = interact.decision_tree_disc(df.copy(), [\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>Close_m1</th>\n",
              "      <th>Close_Disc</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.503125</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.503125</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>310.503125</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>336.546668</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-09</th>\n",
              "      <td>343.500000</td>\n",
              "      <td>331.470001</td>\n",
              "      <td>335.500000</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>5432900</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>344.970001</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>336.546668</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...    Close_m1  Close_Disc\n",
              "Date                                ...                        \n",
              "2019-01-03  309.399994  297.380005  ...  310.119995  310.503125\n",
              "2019-01-04  318.000000  302.730011  ...  300.359985  310.503125\n",
              "2019-01-07  336.739990  317.750000  ...  317.690002  310.503125\n",
              "2019-01-08  344.010010  327.019989  ...  334.959991  336.546668\n",
              "2019-01-09  343.500000  331.470001  ...  335.350006  336.546668\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "htKpCbhrC8g6",
        "colab_type": "text"
      },
      "source": [
        "#### **(4) Normalising**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTEvJ_HkVO1E",
        "colab_type": "text"
      },
      "source": [
        "Normalising normally pertains to the scaling of data. There are many method available, interacting normalising methods makes use of all the feature's attributes to do the scaling."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k1Ci3advDNnK",
        "colab_type": "text"
      },
      "source": [
        "(i) Quantile Normalisation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DY1DI_qlWCp6",
        "colab_type": "text"
      },
      "source": [
        "In statistics, quantile normalization is a technique for making two distributions identical in statistical properties."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Z1N3g7aDJtN",
        "colab_type": "code",
        "outputId": "b6a6182c-b805-491f-c071-267493cbd4b1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 257
        }
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "def quantile_normalize(df, drop):\n",
        "\n",
        "    if drop:\n",
        "      keep = df[drop]\n",
        "      df = df.drop(drop,axis=1)\n",
        "\n",
        "    #compute rank\n",
        "    dic = {}\n",
        "    for col in df:\n",
        "      dic.update({col : sorted(df[col])})\n",
        "    sorted_df = pd.DataFrame(dic)\n",
        "    rank = sorted_df.mean(axis = 1).tolist()\n",
        "    #sort\n",
        "    for col in df:\n",
        "        t = np.searchsorted(np.sort(df[col]), df[col])\n",
        "        df[col] = [rank[i] for i in t]\n",
        "    \n",
        "    if drop:\n",
        "      df = pd.concat((keep,df),axis=1)\n",
        "    return df\n",
        "\n",
        "df_out = interact.quantile_normalize(df.copy(), drop=[\"Close\"]); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>310.119995</td>\n",
              "      <td>1.764424e+06</td>\n",
              "      <td>1.719071e+06</td>\n",
              "      <td>1.735323e+06</td>\n",
              "      <td>1.943365e+06</td>\n",
              "      <td>1.772574e+06</td>\n",
              "      <td>1.674051e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>1.719071e+06</td>\n",
              "      <td>1.701086e+06</td>\n",
              "      <td>1.784325e+06</td>\n",
              "      <td>1.161071e+06</td>\n",
              "      <td>1.674051e+06</td>\n",
              "      <td>1.943365e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>1.806060e+06</td>\n",
              "      <td>1.772574e+06</td>\n",
              "      <td>1.734705e+06</td>\n",
              "      <td>1.232563e+06</td>\n",
              "      <td>1.967032e+06</td>\n",
              "      <td>2.148862e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>2.098010e+06</td>\n",
              "      <td>2.024788e+06</td>\n",
              "      <td>2.011387e+06</td>\n",
              "      <td>1.258749e+06</td>\n",
              "      <td>2.173329e+06</td>\n",
              "      <td>2.173329e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>2.356382e+06</td>\n",
              "      <td>2.056858e+06</td>\n",
              "      <td>2.459069e+06</td>\n",
              "      <td>1.168288e+06</td>\n",
              "      <td>2.177680e+06</td>\n",
              "      <td>2.356382e+06</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                 Close          High  ...     Adj Close       Close_1\n",
              "Date                                  ...                            \n",
              "2019-01-02  310.119995  1.764424e+06  ...  1.772574e+06  1.674051e+06\n",
              "2019-01-03  300.359985  1.719071e+06  ...  1.674051e+06  1.943365e+06\n",
              "2019-01-04  317.690002  1.806060e+06  ...  1.967032e+06  2.148862e+06\n",
              "2019-01-07  334.959991  2.098010e+06  ...  2.173329e+06  2.173329e+06\n",
              "2019-01-08  335.350006  2.356382e+06  ...  2.177680e+06  2.356382e+06\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QEzHO1H8Gr_O",
        "colab_type": "text"
      },
      "source": [
        "#### **(5) Distance**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmZQMl6pXfjP",
        "colab_type": "text"
      },
      "source": [
        "There are multiple types of distance functions like Euclidean, Mahalanobis, and Minkowski distance. Here we are using a contrived example in a location based haversine distance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Woh33jqBGzU_",
        "colab_type": "text"
      },
      "source": [
        "(i) Haversine Distance\n",
        "\n",
        "The Haversine (or great circle) distance is the angular distance between two points on the surface of a sphere."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hIK2PsunzRyO",
        "colab_type": "code",
        "outputId": "8c91464f-ec7e-49f7-d1e0-26d0f67e457e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 257
        }
      },
      "source": [
        "from math import sin, cos, sqrt, atan2, radians\n",
        "def haversine_distance(row, lon=\"Open\", lat=\"Close\"):\n",
        "    c_lat,c_long = radians(52.5200), radians(13.4050)\n",
        "    R = 6373.0\n",
        "    long = radians(row['Open'])\n",
        "    lat = radians(row['Close'])\n",
        "    \n",
        "    dlon = long - c_long\n",
        "    dlat = lat - c_lat\n",
        "    a = sin(dlat / 2)**2 + cos(lat) * cos(c_lat) * sin(dlon / 2)**2\n",
        "    c = 2 * atan2(sqrt(a), sqrt(1 - a))\n",
        "    \n",
        "    return R * c\n",
        "\n",
        "df_out['distance_central'] = df.apply(interact.haversine_distance,axis=1); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>distance_central</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>310.119995</td>\n",
              "      <td>1.764424e+06</td>\n",
              "      <td>1.719071e+06</td>\n",
              "      <td>1.735323e+06</td>\n",
              "      <td>1.943365e+06</td>\n",
              "      <td>1.772574e+06</td>\n",
              "      <td>1.674051e+06</td>\n",
              "      <td>13025.079841</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>1.719071e+06</td>\n",
              "      <td>1.701086e+06</td>\n",
              "      <td>1.784325e+06</td>\n",
              "      <td>1.161071e+06</td>\n",
              "      <td>1.674051e+06</td>\n",
              "      <td>1.943365e+06</td>\n",
              "      <td>13811.329860</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>1.806060e+06</td>\n",
              "      <td>1.772574e+06</td>\n",
              "      <td>1.734705e+06</td>\n",
              "      <td>1.232563e+06</td>\n",
              "      <td>1.967032e+06</td>\n",
              "      <td>2.148862e+06</td>\n",
              "      <td>12366.485485</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>2.098010e+06</td>\n",
              "      <td>2.024788e+06</td>\n",
              "      <td>2.011387e+06</td>\n",
              "      <td>1.258749e+06</td>\n",
              "      <td>2.173329e+06</td>\n",
              "      <td>2.173329e+06</td>\n",
              "      <td>9972.978859</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>2.356382e+06</td>\n",
              "      <td>2.056858e+06</td>\n",
              "      <td>2.459069e+06</td>\n",
              "      <td>1.168288e+06</td>\n",
              "      <td>2.177680e+06</td>\n",
              "      <td>2.356382e+06</td>\n",
              "      <td>9110.110034</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                 Close          High  ...       Close_1  distance_central\n",
              "Date                                  ...                                \n",
              "2019-01-02  310.119995  1.764424e+06  ...  1.674051e+06      13025.079841\n",
              "2019-01-03  300.359985  1.719071e+06  ...  1.943365e+06      13811.329860\n",
              "2019-01-04  317.690002  1.806060e+06  ...  2.148862e+06      12366.485485\n",
              "2019-01-07  334.959991  2.098010e+06  ...  2.173329e+06       9972.978859\n",
              "2019-01-08  335.350006  2.356382e+06  ...  2.356382e+06       9110.110034\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hETa-DCpHLpF",
        "colab_type": "text"
      },
      "source": [
        "#### **(6) Speciality**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A8r4xktjHW8O",
        "colab_type": "text"
      },
      "source": [
        "(i) Technical Features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lpZrhX3iYQ37",
        "colab_type": "text"
      },
      "source": [
        "Technical indicators are heuristic or mathematical calculations based on the price, volume, or open interest of a security or contract used by traders who follow technical analysis. By analyzing historical data, technical analysts use indicators to predict future price movements."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QdseCrXSHLWR",
        "colab_type": "code",
        "outputId": "f6c73638-5c0b-491d-f441-798c48435492",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "import ta\n",
        "\n",
        "def tech(df):\n",
        "  return ta.add_all_ta_features(df, open=\"Open\", high=\"High\", low=\"Low\", close=\"Close\", volume=\"Volume\")\n",
        "  \n",
        "df_out = interact.tech(df.copy()); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>volume_adi</th>\n",
              "      <th>volume_obv</th>\n",
              "      <th>volume_cmf</th>\n",
              "      <th>volume_fi</th>\n",
              "      <th>momentum_mfi</th>\n",
              "      <th>volume_em</th>\n",
              "      <th>volume_sma_em</th>\n",
              "      <th>volume_vpt</th>\n",
              "      <th>volume_nvi</th>\n",
              "      <th>volatility_atr</th>\n",
              "      <th>volatility_bbm</th>\n",
              "      <th>volatility_bbh</th>\n",
              "      <th>volatility_bbl</th>\n",
              "      <th>volatility_bbw</th>\n",
              "      <th>volatility_bbp</th>\n",
              "      <th>volatility_bbhi</th>\n",
              "      <th>volatility_bbli</th>\n",
              "      <th>volatility_kcc</th>\n",
              "      <th>volatility_kch</th>\n",
              "      <th>volatility_kcl</th>\n",
              "      <th>volatility_kcw</th>\n",
              "      <th>volatility_kcp</th>\n",
              "      <th>volatility_kchi</th>\n",
              "      <th>volatility_kcli</th>\n",
              "      <th>volatility_dcl</th>\n",
              "      <th>volatility_dch</th>\n",
              "      <th>volatility_dchi</th>\n",
              "      <th>volatility_dcli</th>\n",
              "      <th>trend_macd</th>\n",
              "      <th>trend_macd_signal</th>\n",
              "      <th>trend_macd_diff</th>\n",
              "      <th>trend_sma_fast</th>\n",
              "      <th>trend_sma_slow</th>\n",
              "      <th>trend_ema_fast</th>\n",
              "      <th>trend_ema_slow</th>\n",
              "      <th>trend_adx</th>\n",
              "      <th>trend_adx_pos</th>\n",
              "      <th>trend_adx_neg</th>\n",
              "      <th>trend_vortex_ind_pos</th>\n",
              "      <th>trend_vortex_ind_neg</th>\n",
              "      <th>trend_vortex_ind_diff</th>\n",
              "      <th>trend_trix</th>\n",
              "      <th>trend_mass_index</th>\n",
              "      <th>trend_cci</th>\n",
              "      <th>trend_dpo</th>\n",
              "      <th>trend_kst</th>\n",
              "      <th>trend_kst_sig</th>\n",
              "      <th>trend_kst_diff</th>\n",
              "      <th>trend_ichimoku_a</th>\n",
              "      <th>trend_ichimoku_b</th>\n",
              "      <th>trend_visual_ichimoku_a</th>\n",
              "      <th>trend_visual_ichimoku_b</th>\n",
              "      <th>trend_aroon_up</th>\n",
              "      <th>trend_aroon_down</th>\n",
              "      <th>trend_aroon_ind</th>\n",
              "      <th>trend_psar</th>\n",
              "      <th>trend_psar_up</th>\n",
              "      <th>trend_psar_down</th>\n",
              "      <th>trend_psar_up_indicator</th>\n",
              "      <th>trend_psar_down_indicator</th>\n",
              "      <th>momentum_rsi</th>\n",
              "      <th>momentum_tsi</th>\n",
              "      <th>momentum_uo</th>\n",
              "      <th>momentum_stoch</th>\n",
              "      <th>momentum_stoch_signal</th>\n",
              "      <th>momentum_wr</th>\n",
              "      <th>momentum_ao</th>\n",
              "      <th>momentum_kama</th>\n",
              "      <th>momentum_roc</th>\n",
              "      <th>others_dr</th>\n",
              "      <th>others_dlr</th>\n",
              "      <th>others_cr</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>4.504939e+06</td>\n",
              "      <td>11658600</td>\n",
              "      <td>0.386405</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.598233e+06</td>\n",
              "      <td>1000.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>308.016663</td>\n",
              "      <td>324.346680</td>\n",
              "      <td>291.686646</td>\n",
              "      <td>10.603334</td>\n",
              "      <td>0.564401</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>-37.166688</td>\n",
              "      <td>136.165005</td>\n",
              "      <td>136.165005</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>306.964996</td>\n",
              "      <td>306.964996</td>\n",
              "      <td>270.626566</td>\n",
              "      <td>269.84809</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>100.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>69.320242</td>\n",
              "      <td>69.320242</td>\n",
              "      <td>-30.679758</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13.616501</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>9.933466e+05</td>\n",
              "      <td>4693400</td>\n",
              "      <td>0.053337</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>-616.944582</td>\n",
              "      <td>-616.944582</td>\n",
              "      <td>1.368287e+06</td>\n",
              "      <td>968.528280</td>\n",
              "      <td>0.0</td>\n",
              "      <td>305.239990</td>\n",
              "      <td>315.000000</td>\n",
              "      <td>295.479980</td>\n",
              "      <td>6.394974</td>\n",
              "      <td>0.250000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>305.198329</td>\n",
              "      <td>319.373332</td>\n",
              "      <td>291.023326</td>\n",
              "      <td>9.289044</td>\n",
              "      <td>0.329335</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>-66.666667</td>\n",
              "      <td>-32.286684</td>\n",
              "      <td>118.286472</td>\n",
              "      <td>127.225739</td>\n",
              "      <td>-8.939267</td>\n",
              "      <td>306.255005</td>\n",
              "      <td>306.255005</td>\n",
              "      <td>270.626566</td>\n",
              "      <td>269.84809</td>\n",
              "      <td>4.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>-4.0</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>-100.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>16.788622</td>\n",
              "      <td>43.054432</td>\n",
              "      <td>-83.211378</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>-3.147172</td>\n",
              "      <td>-3.197760</td>\n",
              "      <td>-3.147172</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>8.087230e+06</td>\n",
              "      <td>12087500</td>\n",
              "      <td>0.310833</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1440.449366</td>\n",
              "      <td>411.752392</td>\n",
              "      <td>2.074142e+05</td>\n",
              "      <td>968.528280</td>\n",
              "      <td>0.0</td>\n",
              "      <td>309.389994</td>\n",
              "      <td>323.577505</td>\n",
              "      <td>295.202483</td>\n",
              "      <td>9.171280</td>\n",
              "      <td>0.792511</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>307.734443</td>\n",
              "      <td>322.274441</td>\n",
              "      <td>293.194444</td>\n",
              "      <td>9.449705</td>\n",
              "      <td>0.842351</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>94.729246</td>\n",
              "      <td>-36.436688</td>\n",
              "      <td>133.490552</td>\n",
              "      <td>129.314010</td>\n",
              "      <td>4.176542</td>\n",
              "      <td>307.690002</td>\n",
              "      <td>307.690002</td>\n",
              "      <td>270.626566</td>\n",
              "      <td>269.84809</td>\n",
              "      <td>12.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>65.661705</td>\n",
              "      <td>-96.130530</td>\n",
              "      <td>NaN</td>\n",
              "      <td>98.496617</td>\n",
              "      <td>61.535160</td>\n",
              "      <td>-1.503383</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.769749</td>\n",
              "      <td>5.609437</td>\n",
              "      <td>2.440993</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>1.422283e+07</td>\n",
              "      <td>19638700</td>\n",
              "      <td>0.423688</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4245.031758</td>\n",
              "      <td>1689.512181</td>\n",
              "      <td>8.371128e+05</td>\n",
              "      <td>968.528280</td>\n",
              "      <td>0.0</td>\n",
              "      <td>315.782494</td>\n",
              "      <td>341.107040</td>\n",
              "      <td>290.457947</td>\n",
              "      <td>16.039234</td>\n",
              "      <td>0.878635</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>313.254997</td>\n",
              "      <td>328.907494</td>\n",
              "      <td>297.602501</td>\n",
              "      <td>9.993454</td>\n",
              "      <td>1.193340</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>133.333333</td>\n",
              "      <td>-42.829187</td>\n",
              "      <td>156.910306</td>\n",
              "      <td>136.213084</td>\n",
              "      <td>20.697222</td>\n",
              "      <td>317.059998</td>\n",
              "      <td>317.059998</td>\n",
              "      <td>270.626566</td>\n",
              "      <td>269.84809</td>\n",
              "      <td>16.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>79.856418</td>\n",
              "      <td>-89.413985</td>\n",
              "      <td>NaN</td>\n",
              "      <td>95.477644</td>\n",
              "      <td>70.254294</td>\n",
              "      <td>-4.522356</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.436113</td>\n",
              "      <td>5.293502</td>\n",
              "      <td>8.009802</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>1.408671e+07</td>\n",
              "      <td>26647200</td>\n",
              "      <td>0.347155</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2004.816212</td>\n",
              "      <td>1768.338188</td>\n",
              "      <td>4.186522e+05</td>\n",
              "      <td>969.655997</td>\n",
              "      <td>0.0</td>\n",
              "      <td>319.695996</td>\n",
              "      <td>347.229869</td>\n",
              "      <td>292.162123</td>\n",
              "      <td>17.225035</td>\n",
              "      <td>0.784268</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>317.695998</td>\n",
              "      <td>333.615999</td>\n",
              "      <td>301.775997</td>\n",
              "      <td>10.022160</td>\n",
              "      <td>1.054460</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>99.069779</td>\n",
              "      <td>-46.742689</td>\n",
              "      <td>171.247933</td>\n",
              "      <td>143.220054</td>\n",
              "      <td>28.027879</td>\n",
              "      <td>320.695007</td>\n",
              "      <td>320.695007</td>\n",
              "      <td>270.626566</td>\n",
              "      <td>269.84809</td>\n",
              "      <td>20.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>299.741604</td>\n",
              "      <td>299.741604</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>80.056917</td>\n",
              "      <td>-84.102885</td>\n",
              "      <td>NaN</td>\n",
              "      <td>81.428259</td>\n",
              "      <td>91.800840</td>\n",
              "      <td>-18.571741</td>\n",
              "      <td>0.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.116436</td>\n",
              "      <td>0.116368</td>\n",
              "      <td>8.135564</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  others_dlr  others_cr\n",
              "Date                                ...                       \n",
              "2019-01-02  315.130005  298.799988  ...         NaN   0.000000\n",
              "2019-01-03  309.399994  297.380005  ...   -3.197760  -3.147172\n",
              "2019-01-04  318.000000  302.730011  ...    5.609437   2.440993\n",
              "2019-01-07  336.739990  317.750000  ...    5.293502   8.009802\n",
              "2019-01-08  344.010010  327.019989  ...    0.116368   8.135564\n",
              "\n",
              "[5 rows x 79 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IrqjEJlSHzzJ",
        "colab_type": "text"
      },
      "source": [
        "#### **(7) Genetic**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r7-y0VhXYxTT",
        "colab_type": "text"
      },
      "source": [
        "Genetic programming has shown promise in constructing feature by osing original features to form high-level ones that can help algorithms achieve better performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6TFa3_GH-n8",
        "colab_type": "text"
      },
      "source": [
        "(i) Symbolic Transformer\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oH-VyUNwYcIJ",
        "colab_type": "text"
      },
      "source": [
        "A symbolic transformer is a supervised transformer that begins by building a population of naive random formulas to represent a relationship."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N_QZ0I09sE3o",
        "colab_type": "code",
        "outputId": "d86fe4d5-eb59-4d4b-9a72-1bdf8bf1d2dd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...   Adj Close     Close_1\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...  310.119995  300.359985\n",
              "2019-01-03  309.399994  297.380005  ...  300.359985  317.690002\n",
              "2019-01-04  318.000000  302.730011  ...  317.690002  334.959991\n",
              "2019-01-07  336.739990  317.750000  ...  334.959991  335.350006\n",
              "2019-01-08  344.010010  327.019989  ...  335.350006  338.529999\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cmLW12YIH1yT",
        "colab_type": "code",
        "outputId": "a4581567-e279-4e6b-9d8c-748beba0da95",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 679
        }
      },
      "source": [
        "from gplearn.genetic import SymbolicTransformer\n",
        "\n",
        "def genetic_feat(df, num_gen=20, num_comp=10):\n",
        "  function_set = ['add', 'sub', 'mul', 'div',\n",
        "                  'sqrt', 'log', 'abs', 'neg', 'inv','tan']\n",
        "\n",
        "  gp = SymbolicTransformer(generations=num_gen, population_size=200,\n",
        "                          hall_of_fame=100, n_components=num_comp,\n",
        "                          function_set=function_set,\n",
        "                          parsimony_coefficient=0.0005,\n",
        "                          max_samples=0.9, verbose=1,\n",
        "                          random_state=0, n_jobs=6)\n",
        "\n",
        "  gen_feats = gp.fit_transform(df.drop(\"Close_1\", axis=1), df[\"Close_1\"]); df.iloc[:,:8]\n",
        "  gen_feats = pd.DataFrame(gen_feats, columns=[\"gen_\"+str(a) for a in range(gen_feats.shape[1])])\n",
        "  gen_feats.index = df.index\n",
        "  return pd.concat((df,gen_feats),axis=1)\n",
        "\n",
        "df_out = interact.genetic_feat(df.copy()); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "    |   Population Average    |             Best Individual              |\n",
            "---- ------------------------- ------------------------------------------ ----------\n",
            " Gen   Length          Fitness   Length          Fitness      OOB Fitness  Time Left\n",
            "   0     9.83         0.514851       11         0.989145         0.933562     58.61s\n",
            "   1     4.43         0.937462        2          0.98989         0.954176     11.72s\n",
            "   2     2.50         0.942791        2         0.990619         0.949961     14.28s\n",
            "   3     1.95         0.977673        2         0.990191         0.954205     11.47s\n",
            "   4     1.52         0.977366        1          0.99006         0.948208     12.47s\n",
            "   5     1.27         0.982232        1         0.990122         0.960944      9.77s\n",
            "   6     1.15         0.982199        1         0.990382         0.946597     10.69s\n",
            "   7     1.15          0.98508        1         0.989952         0.944719      9.77s\n",
            "   8     1.06         0.981903        1         0.991222         0.950846      7.64s\n",
            "   9     1.13         0.982097        1         0.989826         0.974097      7.96s\n",
            "  10     1.13         0.977621        1         0.989617         0.961456      6.19s\n",
            "  11     1.03         0.987014        1         0.989921          0.95797      6.48s\n",
            "  12     1.02         0.982356        1         0.990526         0.958263      4.80s\n",
            "  13     1.01         0.986901        1         0.991101         0.911745      4.80s\n",
            "  14     1.00         0.987087        1         0.989856         0.961373      3.39s\n",
            "  15     1.02         0.982253        1         0.990471         0.950419      3.23s\n",
            "  16     1.00         0.986992        1         0.990015         0.953431      2.03s\n",
            "  17     1.11         0.982442        1         0.990359         0.943364      1.62s\n",
            "  18     1.06         0.982249        1         0.990034         0.957424      0.67s\n",
            "  19     1.00         0.987054        1         0.990107         0.946241      0.00s\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>gen_0</th>\n",
              "      <th>gen_1</th>\n",
              "      <th>gen_2</th>\n",
              "      <th>gen_3</th>\n",
              "      <th>gen_4</th>\n",
              "      <th>gen_5</th>\n",
              "      <th>gen_6</th>\n",
              "      <th>gen_7</th>\n",
              "      <th>gen_8</th>\n",
              "      <th>gen_9</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>315.130005</td>\n",
              "      <td>298.799988</td>\n",
              "      <td>306.100006</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>11658600</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>310.119995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>300.359985</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>317.690002</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>334.959991</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>335.350006</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...       gen_8       gen_9\n",
              "Date                                ...                        \n",
              "2019-01-02  315.130005  298.799988  ...  310.119995  310.119995\n",
              "2019-01-03  309.399994  297.380005  ...  300.359985  300.359985\n",
              "2019-01-04  318.000000  302.730011  ...  317.690002  317.690002\n",
              "2019-01-07  336.739990  317.750000  ...  334.959991  334.959991\n",
              "2019-01-08  344.010010  327.019989  ...  335.350006  335.350006\n",
              "\n",
              "[5 rows x 17 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zrgqcQ_HfPvK",
        "colab_type": "text"
      },
      "source": [
        "<a name=\"mapping\"></a>\n",
        "\n",
        "## **(3) Mapping**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "51Q7kjqSZRB8",
        "colab_type": "text"
      },
      "source": [
        "Methods that help with the summarisation of features by remapping them to achieve some aim like the maximisation of variability or class separability. These methods tend to be unsupervised, but can also take an supervised form."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dCdVu99rhFZb",
        "colab_type": "text"
      },
      "source": [
        "#### **(1) Eigen Decomposition**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_txhdjEWZnuN",
        "colab_type": "text"
      },
      "source": [
        "Eigendecomposition or sometimes spectral decomposition is the factorization of a matrix into a canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors. Some examples are LDA and PCA."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZCZ3ILsuhhVH",
        "colab_type": "text"
      },
      "source": [
        "(i) Principal Component Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r43-u9PfaQ_o",
        "colab_type": "text"
      },
      "source": [
        "Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9s75knCdgv4p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def pca_feature(df, memory_issues=False,mem_iss_component=False,variance_or_components=0.80,n_components=5 ,drop_cols=None, non_linear=True):\n",
        "    \n",
        "  if non_linear:\n",
        "    pca = KernelPCA(n_components = n_components, kernel='rbf', fit_inverse_transform=True, random_state = 33, remove_zero_eig= True)\n",
        "  else:\n",
        "    if memory_issues:\n",
        "      if not mem_iss_component:\n",
        "        raise ValueError(\"If you have memory issues, you have to preselect mem_iss_component\")\n",
        "      pca = IncrementalPCA(mem_iss_component)\n",
        "    else:\n",
        "      if variance_or_components>1:\n",
        "        pca = PCA(n_components=variance_or_components) \n",
        "      else: # automted selection based on variance\n",
        "        pca = PCA(n_components=variance_or_components,svd_solver=\"full\") \n",
        "  if drop_cols:\n",
        "    X_pca = pca.fit_transform(df.drop(drop_cols,axis=1))\n",
        "    return pd.concat((df[drop_cols],pd.DataFrame(X_pca, columns=[\"PCA_\"+str(i+1) for i in range(X_pca.shape[1])],index=df.index)),axis=1)\n",
        "\n",
        "  else:\n",
        "    X_pca = pca.fit_transform(df)\n",
        "    return pd.DataFrame(X_pca, columns=[\"PCA_\"+str(i+1) for i in range(X_pca.shape[1])],index=df.index)\n",
        "\n",
        "\n",
        "  return df\n",
        "\n",
        "df_out = mapper.pca_feature(df.copy(), variance_or_components=0.9, n_components=8,non_linear=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zd_PvZBPXwx0",
        "colab_type": "text"
      },
      "source": [
        "#### **(2) Cross Decomposition**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLqNBrxkaaOm",
        "colab_type": "text"
      },
      "source": [
        "These families of algorithms are useful to find linear relations between two multivariate datasets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tukydcMUXz9Y",
        "colab_type": "text"
      },
      "source": [
        "(1) Canonical Correlation Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rZDUQfG4akmY",
        "colab_type": "text"
      },
      "source": [
        "Canonical-correlation analysis (CCA) is a way of inferring information from cross-covariance matrices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xwjWJC9rX4yB",
        "colab_type": "code",
        "outputId": "a87c796f-d012-4215-bf96-69875a3c387b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from sklearn.cross_decomposition import CCA\n",
        "\n",
        "def cross_lag(df, drop=None, lags=1, components=4 ):\n",
        "\n",
        "  if drop:\n",
        "    keep = df[drop]\n",
        "    df = df.drop([drop],axis=1)\n",
        "\n",
        "  df_2 = df.shift(lags)\n",
        "  df = df.iloc[lags:,:]\n",
        "  df_2 = df_2.dropna().reset_index(drop=True)\n",
        "\n",
        "  cca = CCA(n_components=components)\n",
        "  cca.fit(df_2, df)\n",
        "\n",
        "  X_c, df_2 = cca.transform(df_2, df)\n",
        "  df_2 = pd.DataFrame(df_2, index=df.index)\n",
        "  df_2 = df.add_prefix('crd_')\n",
        "\n",
        "  if drop:\n",
        "    df = pd.concat([keep,df,df_2],axis=1)\n",
        "  else:\n",
        "    df = pd.concat([df,df_2],axis=1)\n",
        "  return df\n",
        "\n",
        "df_out = mapper.cross_lag(df.copy()); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>crd_High</th>\n",
              "      <th>crd_Low</th>\n",
              "      <th>crd_Open</th>\n",
              "      <th>crd_Close</th>\n",
              "      <th>crd_Volume</th>\n",
              "      <th>crd_Adj Close</th>\n",
              "      <th>crd_Close_1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-09</th>\n",
              "      <td>343.500000</td>\n",
              "      <td>331.470001</td>\n",
              "      <td>335.500000</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>5432900</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>344.970001</td>\n",
              "      <td>343.500000</td>\n",
              "      <td>331.470001</td>\n",
              "      <td>335.500000</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>5432900</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>344.970001</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  crd_Adj Close  crd_Close_1\n",
              "Date                                ...                            \n",
              "2019-01-03  309.399994  297.380005  ...     300.359985   317.690002\n",
              "2019-01-04  318.000000  302.730011  ...     317.690002   334.959991\n",
              "2019-01-07  336.739990  317.750000  ...     334.959991   335.350006\n",
              "2019-01-08  344.010010  327.019989  ...     335.350006   338.529999\n",
              "2019-01-09  343.500000  331.470001  ...     338.529999   344.970001\n",
              "\n",
              "[5 rows x 14 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Na8Y6ySbpC4",
        "colab_type": "text"
      },
      "source": [
        "#### **(3) Kernel Approximation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GMxcX-6za3Eb",
        "colab_type": "text"
      },
      "source": [
        "Functions that approximate the feature mappings that correspond to certain kernels, as they are used for example in support vector machines."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NpRXZNmgcj-6",
        "colab_type": "text"
      },
      "source": [
        "(i) Additive Chi2 Kernel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5UmCbyqwbFEy",
        "colab_type": "text"
      },
      "source": [
        "Computes the additive chi-squared kernel between observations in X and Y The chi-squared kernel is computed between each pair of rows in X and Y. X and Y have to be non-negative."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yV8JAfmXbuXO",
        "colab_type": "code",
        "outputId": "5de64517-4b9b-4d4e-9d06-5c03ba86010b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "from sklearn.kernel_approximation import AdditiveChi2Sampler\n",
        "\n",
        "def a_chi(df, drop=None, lags=1, sample_steps=2 ):\n",
        "\n",
        "  if drop:\n",
        "    keep = df[drop]\n",
        "    df = df.drop([drop],axis=1)\n",
        "\n",
        "  df_2 = df.shift(lags)\n",
        "  df = df.iloc[lags:,:]\n",
        "  df_2 = df_2.dropna().reset_index(drop=True)\n",
        "\n",
        "  chi2sampler = AdditiveChi2Sampler(sample_steps=sample_steps)\n",
        "\n",
        "  df_2 = chi2sampler.fit_transform(df_2, df[\"Close\"])\n",
        "\n",
        "  df_2 = pd.DataFrame(df_2, index=df.index)\n",
        "  df_2 = df.add_prefix('achi_')\n",
        "\n",
        "  if drop:\n",
        "    df = pd.concat([keep,df,df_2],axis=1)\n",
        "  else:\n",
        "    df = pd.concat([df,df_2],axis=1)\n",
        "  return df\n",
        "\n",
        "df_out = mapper.a_chi(df.copy()); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_1</th>\n",
              "      <th>achi_High</th>\n",
              "      <th>achi_Low</th>\n",
              "      <th>achi_Open</th>\n",
              "      <th>achi_Close</th>\n",
              "      <th>achi_Volume</th>\n",
              "      <th>achi_Adj Close</th>\n",
              "      <th>achi_Close_1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>309.399994</td>\n",
              "      <td>297.380005</td>\n",
              "      <td>307.000000</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>6965200</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>317.690002</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>318.000000</td>\n",
              "      <td>302.730011</td>\n",
              "      <td>306.000000</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>7394100</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>334.959991</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>336.739990</td>\n",
              "      <td>317.750000</td>\n",
              "      <td>321.720001</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>7551200</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>335.350006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>344.010010</td>\n",
              "      <td>327.019989</td>\n",
              "      <td>341.959991</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>7008500</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>338.529999</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-09</th>\n",
              "      <td>343.500000</td>\n",
              "      <td>331.470001</td>\n",
              "      <td>335.500000</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>5432900</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>344.970001</td>\n",
              "      <td>343.500000</td>\n",
              "      <td>331.470001</td>\n",
              "      <td>335.500000</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>5432900</td>\n",
              "      <td>338.529999</td>\n",
              "      <td>344.970001</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  High         Low  ...  achi_Adj Close  achi_Close_1\n",
              "Date                                ...                              \n",
              "2019-01-03  309.399994  297.380005  ...      300.359985    317.690002\n",
              "2019-01-04  318.000000  302.730011  ...      317.690002    334.959991\n",
              "2019-01-07  336.739990  317.750000  ...      334.959991    335.350006\n",
              "2019-01-08  344.010010  327.019989  ...      335.350006    338.529999\n",
              "2019-01-09  343.500000  331.470001  ...      338.529999    344.970001\n",
              "\n",
              "[5 rows x 14 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJI_GMWN6Dip",
        "colab_type": "text"
      },
      "source": [
        "#### **(4) Autoencoder**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xXBDBCP3bRy6",
        "colab_type": "text"
      },
      "source": [
        "An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore noise."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t0hs4BRB6GBx",
        "colab_type": "text"
      },
      "source": [
        "(i) Feed Forward"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2m-iM_-ybrYz",
        "colab_type": "text"
      },
      "source": [
        "The simplest form of an autoencoder is a feedforward, non-recurrent neural network similar to single layer perceptrons that participate in multilayer perceptrons"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PZsnIaEACfNS",
        "colab_type": "code",
        "outputId": "b69693be-3670-4ad2-ba35-85c8ba20172e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 257
        }
      },
      "source": [
        "from sklearn.preprocessing import minmax_scale\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "def encoder_dataset(df, drop=None, dimesions=20):\n",
        "\n",
        "  if drop:\n",
        "    train_scaled = minmax_scale(df.drop(drop,axis=1).values, axis = 0)\n",
        "  else:\n",
        "    train_scaled = minmax_scale(df.values, axis = 0)\n",
        "\n",
        "  # define the number of encoding dimensions\n",
        "  encoding_dim = dimesions\n",
        "  # define the number of features\n",
        "  ncol = train_scaled.shape[1]\n",
        "  input_dim = tf.keras.Input(shape = (ncol, ))\n",
        "\n",
        "  # Encoder Layers\n",
        "  encoded1 = tf.keras.layers.Dense(3000, activation = 'relu')(input_dim)\n",
        "  encoded2 = tf.keras.layers.Dense(2750, activation = 'relu')(encoded1)\n",
        "  encoded3 = tf.keras.layers.Dense(2500, activation = 'relu')(encoded2)\n",
        "  encoded4 = tf.keras.layers.Dense(750, activation = 'relu')(encoded3)\n",
        "  encoded5 = tf.keras.layers.Dense(500, activation = 'relu')(encoded4)\n",
        "  encoded6 = tf.keras.layers.Dense(250, activation = 'relu')(encoded5)\n",
        "  encoded7 = tf.keras.layers.Dense(encoding_dim, activation = 'relu')(encoded6)\n",
        "\n",
        "  encoder = tf.keras.Model(inputs = input_dim, outputs = encoded7)\n",
        "  encoded_input = tf.keras.Input(shape = (encoding_dim, ))\n",
        "\n",
        "  encoded_train = pd.DataFrame(encoder.predict(train_scaled),index=df.index)\n",
        "  encoded_train = encoded_train.add_prefix('encoded_')\n",
        "  if drop:\n",
        "    encoded_train = pd.concat((df[drop],encoded_train),axis=1)\n",
        "\n",
        "  return encoded_train\n",
        "\n",
        "df_out = mapper.encoder_dataset(df.copy(), [\"Close_1\"], 15); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close_1</th>\n",
              "      <th>encoded_0</th>\n",
              "      <th>encoded_1</th>\n",
              "      <th>encoded_2</th>\n",
              "      <th>encoded_3</th>\n",
              "      <th>encoded_4</th>\n",
              "      <th>encoded_5</th>\n",
              "      <th>encoded_6</th>\n",
              "      <th>encoded_7</th>\n",
              "      <th>encoded_8</th>\n",
              "      <th>encoded_9</th>\n",
              "      <th>encoded_10</th>\n",
              "      <th>encoded_11</th>\n",
              "      <th>encoded_12</th>\n",
              "      <th>encoded_13</th>\n",
              "      <th>encoded_14</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002014</td>\n",
              "      <td>0.012228</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002723</td>\n",
              "      <td>0.012987</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000363</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.005032</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.006323</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.001496</td>\n",
              "      <td>0.010318</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003512</td>\n",
              "      <td>0.011826</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003393</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.006712</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002074</td>\n",
              "      <td>0.010756</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003784</td>\n",
              "      <td>0.013213</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003459</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.006907</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002236</td>\n",
              "      <td>0.012071</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.004357</td>\n",
              "      <td>0.014771</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003676</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.007873</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>338.529999</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002041</td>\n",
              "      <td>0.012700</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.005110</td>\n",
              "      <td>0.014761</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.004150</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.008607</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Close_1  encoded_0  ...  encoded_13  encoded_14\n",
              "Date                               ...                        \n",
              "2019-01-02  300.359985        0.0  ...         0.0    0.006323\n",
              "2019-01-03  317.690002        0.0  ...         0.0    0.006712\n",
              "2019-01-04  334.959991        0.0  ...         0.0    0.006907\n",
              "2019-01-07  335.350006        0.0  ...         0.0    0.007873\n",
              "2019-01-08  338.529999        0.0  ...         0.0    0.008607\n",
              "\n",
              "[5 rows x 16 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SBlttJLkaX5q",
        "colab_type": "code",
        "outputId": "91d9d6db-878f-41d3-929b-328d6d4b2abd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 257
        }
      },
      "source": [
        "df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close_1</th>\n",
              "      <th>encoded_0</th>\n",
              "      <th>encoded_1</th>\n",
              "      <th>encoded_2</th>\n",
              "      <th>encoded_3</th>\n",
              "      <th>encoded_4</th>\n",
              "      <th>encoded_5</th>\n",
              "      <th>encoded_6</th>\n",
              "      <th>encoded_7</th>\n",
              "      <th>encoded_8</th>\n",
              "      <th>encoded_9</th>\n",
              "      <th>encoded_10</th>\n",
              "      <th>encoded_11</th>\n",
              "      <th>encoded_12</th>\n",
              "      <th>encoded_13</th>\n",
              "      <th>encoded_14</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002014</td>\n",
              "      <td>0.012228</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002723</td>\n",
              "      <td>0.012987</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000363</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.005032</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.006323</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.001496</td>\n",
              "      <td>0.010318</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003512</td>\n",
              "      <td>0.011826</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003393</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.006712</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002074</td>\n",
              "      <td>0.010756</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003784</td>\n",
              "      <td>0.013213</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003459</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.006907</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002236</td>\n",
              "      <td>0.012071</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.004357</td>\n",
              "      <td>0.014771</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.003676</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.007873</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>338.529999</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.002041</td>\n",
              "      <td>0.012700</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.005110</td>\n",
              "      <td>0.014761</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.004150</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.008607</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Close_1  encoded_0  ...  encoded_13  encoded_14\n",
              "Date                               ...                        \n",
              "2019-01-02  300.359985        0.0  ...         0.0    0.006323\n",
              "2019-01-03  317.690002        0.0  ...         0.0    0.006712\n",
              "2019-01-04  334.959991        0.0  ...         0.0    0.006907\n",
              "2019-01-07  335.350006        0.0  ...         0.0    0.007873\n",
              "2019-01-08  338.529999        0.0  ...         0.0    0.008607\n",
              "\n",
              "[5 rows x 16 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OsCtrfqtHLeo",
        "colab_type": "text"
      },
      "source": [
        "#### **(5) Manifold Learning**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XqEai0r_b_v0",
        "colab_type": "text"
      },
      "source": [
        "Manifold Learning can be thought of as an attempt to generalize linear frameworks like PCA to be sensitive to non-linear structure in data. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RKrwQtrcKNt5",
        "colab_type": "text"
      },
      "source": [
        "(i) Local Linear Embedding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5j6fZx20cIvs",
        "colab_type": "text"
      },
      "source": [
        "Locally Linear Embedding is a method of non-linear dimensionality reduction. It tries to reduce these n-Dimensions while trying to preserve the geometric features of the original non-linear feature structure."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ErmbqARHLO9g",
        "colab_type": "code",
        "outputId": "857cb3fc-877f-4481-a527-9d75adebf9b0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        }
      },
      "source": [
        "from sklearn.manifold import LocallyLinearEmbedding\n",
        "\n",
        "def lle_feat(df, drop=None, components=4):\n",
        "\n",
        "  if drop:\n",
        "    keep = df[drop]\n",
        "    df = df.drop(drop, axis=1)\n",
        "\n",
        "  embedding = LocallyLinearEmbedding(n_components=components)\n",
        "  em = embedding.fit_transform(df)\n",
        "  df = pd.DataFrame(em,index=df.index)\n",
        "  df = df.add_prefix('lle_')\n",
        "  if drop:\n",
        "    df = pd.concat((keep,df),axis=1)\n",
        "  return df\n",
        "\n",
        "df_out = mapper.lle_feat(df.copy(),[\"Close_1\"],4); df_out.head()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close_1</th>\n",
              "      <th>lle_0</th>\n",
              "      <th>lle_1</th>\n",
              "      <th>lle_2</th>\n",
              "      <th>lle_3</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>0.000015</td>\n",
              "      <td>-1.373606e-01</td>\n",
              "      <td>2.484890e-07</td>\n",
              "      <td>-0.000001</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>-0.205008</td>\n",
              "      <td>1.611563e-06</td>\n",
              "      <td>-6.554339e-08</td>\n",
              "      <td>0.000011</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>0.000010</td>\n",
              "      <td>-2.294303e-07</td>\n",
              "      <td>-1.054092e-01</td>\n",
              "      <td>-0.000006</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>0.000005</td>\n",
              "      <td>-2.036527e-07</td>\n",
              "      <td>-1.054093e-01</td>\n",
              "      <td>-0.000005</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>338.529999</td>\n",
              "      <td>-0.205007</td>\n",
              "      <td>1.607294e-06</td>\n",
              "      <td>-6.490422e-08</td>\n",
              "      <td>0.000011</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Close_1     lle_0         lle_1         lle_2     lle_3\n",
              "Date                                                                  \n",
              "2019-01-02  300.359985  0.000015 -1.373606e-01  2.484890e-07 -0.000001\n",
              "2019-01-03  317.690002 -0.205008  1.611563e-06 -6.554339e-08  0.000011\n",
              "2019-01-04  334.959991  0.000010 -2.294303e-07 -1.054092e-01 -0.000006\n",
              "2019-01-07  335.350006  0.000005 -2.036527e-07 -1.054093e-01 -0.000005\n",
              "2019-01-08  338.529999 -0.205007  1.607294e-06 -6.490422e-08  0.000011"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dFlObKCMUuP6",
        "colab_type": "text"
      },
      "source": [
        "#### **(6) Clustering**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BI9NrEcrdJvm",
        "colab_type": "text"
      },
      "source": [
        "Most clustering techniques start with a bottom up approach: each observation starts in its own cluster, and clusters are successively merged together with some measure. Although these clustering techniques are typically used for observations, it can also be used for feature dimensionality reduction; especially hierarchical clustering techniques. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JmxHmqk6WPZ0",
        "colab_type": "text"
      },
      "source": [
        "(i) Feature Agglomeration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H8oxiwg2c9oQ",
        "colab_type": "text"
      },
      "source": [
        "Feature agglomerative uses clustering to group together features that look very similar, thus decreasing the number of features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rBAdnJeHK6wc",
        "colab_type": "code",
        "outputId": "aab16b79-7c0a-4338-927c-ba4da7459b45",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        }
      },
      "source": [
        "import numpy as np\n",
        "from sklearn import datasets, cluster\n",
        "\n",
        "def feature_agg(df, drop=None, components=4):\n",
        "\n",
        "  if drop:\n",
        "    keep = df[drop]\n",
        "    df = df.drop(drop, axis=1)\n",
        "\n",
        "  components = min(df.shape[1]-1,components)\n",
        "  agglo = cluster.FeatureAgglomeration(n_clusters=components)\n",
        "  agglo.fit(df)\n",
        "  df = pd.DataFrame(agglo.transform(df),index=df.index)\n",
        "  df = df.add_prefix('feagg_')\n",
        "\n",
        "  if drop:\n",
        "    return pd.concat((keep,df),axis=1)\n",
        "  else:\n",
        "    return df\n",
        "\n",
        "\n",
        "df_out = mapper.feature_agg(df.copy(),[\"Close_1\"],4 ); df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close_1</th>\n",
              "      <th>feagg_0</th>\n",
              "      <th>feagg_1</th>\n",
              "      <th>feagg_2</th>\n",
              "      <th>feagg_3</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>302.449997</td>\n",
              "      <td>11658600.0</td>\n",
              "      <td>310.119995</td>\n",
              "      <td>315.130005</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>302.190002</td>\n",
              "      <td>6965200.0</td>\n",
              "      <td>300.359985</td>\n",
              "      <td>309.399994</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>304.365005</td>\n",
              "      <td>7394100.0</td>\n",
              "      <td>317.690002</td>\n",
              "      <td>318.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>319.735001</td>\n",
              "      <td>7551200.0</td>\n",
              "      <td>334.959991</td>\n",
              "      <td>336.739990</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>338.529999</td>\n",
              "      <td>334.489990</td>\n",
              "      <td>7008500.0</td>\n",
              "      <td>335.350006</td>\n",
              "      <td>344.010010</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Close_1     feagg_0     feagg_1     feagg_2     feagg_3\n",
              "Date                                                                  \n",
              "2019-01-02  300.359985  302.449997  11658600.0  310.119995  315.130005\n",
              "2019-01-03  317.690002  302.190002   6965200.0  300.359985  309.399994\n",
              "2019-01-04  334.959991  304.365005   7394100.0  317.690002  318.000000\n",
              "2019-01-07  335.350006  319.735001   7551200.0  334.959991  336.739990\n",
              "2019-01-08  338.529999  334.489990   7008500.0  335.350006  344.010010"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTqf6PZLdyiv",
        "colab_type": "text"
      },
      "source": [
        "#### **(7) Neigbouring**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z9qy9wdid7vc",
        "colab_type": "text"
      },
      "source": [
        "Neighbouring points can be calculated using distance metrics like Hamming, Manhattan, Minkowski distance. The principle behind nearest neighbor methods is to find a predefined number of training samples closest in distance to the new point, and predict the label from these."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hrL_dyag-iVK",
        "colab_type": "text"
      },
      "source": [
        "(i) Nearest Neighbours"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S77Yz3acezul",
        "colab_type": "text"
      },
      "source": [
        "Unsupervised learner for implementing neighbor searches."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h_CfSx8wdVdf",
        "colab_type": "code",
        "outputId": "34f833ca-f57e-4302-cd45-04b14c5c0411",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        }
      },
      "source": [
        "from sklearn.neighbors import NearestNeighbors\n",
        "\n",
        "def neigh_feat(df, drop, neighbors=6):\n",
        "  \n",
        "  if drop:\n",
        "    keep = df[drop]\n",
        "    df = df.drop(drop, axis=1)\n",
        "\n",
        "  components = min(df.shape[0]-1,neighbors)\n",
        "  neigh = NearestNeighbors(n_neighbors=neighbors)\n",
        "  neigh.fit(df)\n",
        "  neigh = neigh.kneighbors()[0]\n",
        "  df = pd.DataFrame(neigh, index=df.index)\n",
        "  df = df.add_prefix('neigh_')\n",
        "\n",
        "  if drop:\n",
        "    return pd.concat((keep,df),axis=1)\n",
        "  else:\n",
        "    return df\n",
        "\n",
        "  return df\n",
        "\n",
        "df_out = mapper.neigh_feat(df.copy(),[\"Close_1\"],4 ); df_out.head()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close_1</th>\n",
              "      <th>neigh_0</th>\n",
              "      <th>neigh_1</th>\n",
              "      <th>neigh_2</th>\n",
              "      <th>neigh_3</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-02</th>\n",
              "      <td>300.359985</td>\n",
              "      <td>5104.020265</td>\n",
              "      <td>142000.028615</td>\n",
              "      <td>204900.092408</td>\n",
              "      <td>225900.055502</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-03</th>\n",
              "      <td>317.690002</td>\n",
              "      <td>21800.029313</td>\n",
              "      <td>43100.234741</td>\n",
              "      <td>43300.066365</td>\n",
              "      <td>57000.043646</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-04</th>\n",
              "      <td>334.959991</td>\n",
              "      <td>1215.311717</td>\n",
              "      <td>1800.873884</td>\n",
              "      <td>34700.093746</td>\n",
              "      <td>39300.622102</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-07</th>\n",
              "      <td>335.350006</td>\n",
              "      <td>36800.561715</td>\n",
              "      <td>47100.109142</td>\n",
              "      <td>61200.003858</td>\n",
              "      <td>67900.373738</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>338.529999</td>\n",
              "      <td>295.808619</td>\n",
              "      <td>19801.476534</td>\n",
              "      <td>39900.396729</td>\n",
              "      <td>43300.066365</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Close_1       neigh_0  ...        neigh_2        neigh_3\n",
              "Date                                  ...                              \n",
              "2019-01-02  300.359985   5104.020265  ...  204900.092408  225900.055502\n",
              "2019-01-03  317.690002  21800.029313  ...   43300.066365   57000.043646\n",
              "2019-01-04  334.959991   1215.311717  ...   34700.093746   39300.622102\n",
              "2019-01-07  335.350006  36800.561715  ...   61200.003858   67900.373738\n",
              "2019-01-08  338.529999    295.808619  ...   39900.396729   43300.066365\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MxSVUu-skUIT",
        "colab_type": "text"
      },
      "source": [
        "<a name=\"extraction\"></a>\n",
        "\n",
        "## **(4) Extraction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23jx5IDV1x_n",
        "colab_type": "text"
      },
      "source": [
        "When working with extraction, you have decide the size of the time series history to take into account when calculating a collection of walk-forward feature values. To facilitate our extraction, we use an excellent package called TSfresh, and also some of their default features. For completeness, we also include 12 or so custom features to be added to the extraction pipeline.\n",
        "\n",
        "The *time series* methods in the transformation section and the interaction section are similar to the methods we will uncover in the extraction section, however, for transformation and interaction methods the output is an entire new time series, whereas extraction methods takes as input multiple constructed time series and extracts a singular value from each time series to reconstruct an entirely new time series. \n",
        "\n",
        "Some methods naturally fit better in one format over another, e.g., lags are too expensive for extraction; time series decomposition only has to be performed once, because it has a low level of 'leakage' so is better suited to transformation; and forecast methods attempt to predict multiple future training samples, so won't work with extraction that only delivers one value per time series. Furthermore all non time-series (cross-sectional) transformation and extraction techniques can not make use of extraction as it is solely a time-series method. \n",
        "\n",
        "Lastly, when we want to double apply specific functions we can apply it as a transformation/interaction then all the extraction methods can be applied to this feature as well. For example, if we calculate a smoothing function (transformation) then all other extraction functions (median, entropy, linearity etc.) can now be applied to that smoothing function, including the application of the smoothing function itself, e.g., a double smooth, double lag, double filter etc. So separating these methods out give us great flexibility."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PEaT3REtUW7b",
        "colab_type": "text"
      },
      "source": [
        "Decorator"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dwIX4pS8UXn9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def set_property(key, value):\n",
        "    \"\"\"\n",
        "    This method returns a decorator that sets the property key of the function to value\n",
        "    \"\"\"\n",
        "    def decorate_func(func):\n",
        "        setattr(func, key, value)\n",
        "        if func.__doc__ and key == \"fctype\":\n",
        "            func.__doc__ = func.__doc__ + \"\\n\\n    *This function is of type: \" + value + \"*\\n\"\n",
        "        return func\n",
        "    return decorate_func"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gS0E7xQeqP9L",
        "colab_type": "text"
      },
      "source": [
        "#### **(1) Energy**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HR82DmId2uKI",
        "colab_type": "text"
      },
      "source": [
        "You can calculate the linear, non-linear and absolute energy of a time series. In signal processing, the energy $E_S$ of a continuous-time signal $x(t)$ is defined as the area under the squared magnitude of the considered signal. Mathematically, $E_{s}=\\langle x(t), x(t)\\rangle=\\int_{-\\infty}^{\\infty}|x(t)|^{2} d t$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NeXp4Do0rTcf",
        "colab_type": "text"
      },
      "source": [
        "(i) Absolute Energy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U6KqGxe4fp0v",
        "colab_type": "text"
      },
      "source": [
        "Returns the absolute energy of the time series which is the sum over the squared values"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9rJmdTb21zn0",
        "colab_type": "code",
        "outputId": "285d18bb-6762-4520-8db7-d42531c3ab71",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def abs_energy(x):\n",
        "\n",
        "    if not isinstance(x, (np.ndarray, pd.Series)):\n",
        "        x = np.asarray(x)\n",
        "    return np.dot(x, x)\n",
        "\n",
        "extract.abs_energy(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "19385194.419412717"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hp_Pzpa_rEKg",
        "colab_type": "text"
      },
      "source": [
        "#### **(2) Distance**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QnxdkZbH4ujr",
        "colab_type": "text"
      },
      "source": [
        "Here we widely define distance measures as those that take a difference between attributes or series of datapoints."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c339phBVqgF6",
        "colab_type": "text"
      },
      "source": [
        "(i) Complexity-Invariant Distance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j-AQKB0qfwYQ",
        "colab_type": "text"
      },
      "source": [
        "This function calculator is an estimate for a time series complexity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E3G6C1fsqhnK",
        "colab_type": "code",
        "outputId": "1478d639-c998-40db-e563-f1b677cad6dd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def cid_ce(x, normalize):\n",
        "\n",
        "    if not isinstance(x, (np.ndarray, pd.Series)):\n",
        "        x = np.asarray(x)\n",
        "    if normalize:\n",
        "        s = np.std(x)\n",
        "        if s!=0:\n",
        "            x = (x - np.mean(x))/s\n",
        "        else:\n",
        "            return 0.0\n",
        "\n",
        "    x = np.diff(x)\n",
        "    return np.sqrt(np.dot(x, x))\n",
        "\n",
        "extract.cid_ce(df[\"Close\"], True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2.579588431640294"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZUkYNHlFre23",
        "colab_type": "text"
      },
      "source": [
        "#### **(3) Differencing**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YYa8SQyc5Vbk",
        "colab_type": "text"
      },
      "source": [
        "Many alternatives to differecing exists, one can for example take the difference of every other value, take the squared diffence, take the fractional difference, or like our example, take the mean absolute difference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HEmfYHnKrkVj",
        "colab_type": "text"
      },
      "source": [
        "(i) Mean Absolute Change"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QR-3EQWvf8Oa",
        "colab_type": "text"
      },
      "source": [
        "Returns the mean over the absolute differences between subsequent time series values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q2qbmClcqzKy",
        "colab_type": "code",
        "outputId": "809eef33-1ed6-489b-8c23-d4fe27083032",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def mean_abs_change(x):\n",
        "    return np.mean(np.abs(np.diff(x)))\n",
        "\n",
        "extract.mean_abs_change(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5.739680297851564"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2U6pdg6qr8Uw",
        "colab_type": "text"
      },
      "source": [
        "#### **(4) Derivative**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4loV0zfe5_oy",
        "colab_type": "text"
      },
      "source": [
        "Features where the emphasis is on the rate of change. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_cMCQmOQr3P5",
        "colab_type": "text"
      },
      "source": [
        "(i) Mean Central Second Derivative"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tv15umKtgFKL",
        "colab_type": "text"
      },
      "source": [
        "Returns the mean value of a central approximation of the second derivative"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b5oxxYEerr2A",
        "colab_type": "code",
        "outputId": "6ff1e32c-120d-434e-a2f1-cf74da2ec006",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def _roll(a, shift):\n",
        "    if not isinstance(a, np.ndarray):\n",
        "        a = np.asarray(a)\n",
        "    idx = shift % len(a)\n",
        "    return np.concatenate([a[-idx:], a[:-idx]])\n",
        "\n",
        "def mean_second_derivative_central(x):\n",
        "\n",
        "    diff = (_roll(x, 1) - 2 * np.array(x) + _roll(x, -1)) / 2.0\n",
        "    return np.mean(diff[1:-1])\n",
        "\n",
        "extract.mean_second_derivative_central(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "-0.011887515883847835"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sz0GMiTfsfgx",
        "colab_type": "text"
      },
      "source": [
        "#### **(5) Volatility**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rp0993qA6W39",
        "colab_type": "text"
      },
      "source": [
        "Volatility is a statistical measure of the dispersion of a time-series."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U2FgQiScskgS",
        "colab_type": "text"
      },
      "source": [
        "(i) Variance Larger than Standard Deviation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N_7UBIOosIGE",
        "colab_type": "code",
        "outputId": "27fb8985-03c4-4d42-df66-e9e7d809fb85",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def variance_larger_than_standard_deviation(x):\n",
        "\n",
        "    y = np.var(x)\n",
        "    return y > np.sqrt(y)\n",
        "\n",
        "extract.variance_larger_than_standard_deviation(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbwNU9wlRMlX",
        "colab_type": "text"
      },
      "source": [
        "(ii) Variability Index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BqmryncaRWCe",
        "colab_type": "text"
      },
      "source": [
        "Variability Index is a way to measure how smooth or 'variable' a time series is."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LTVo7t3mRLzm",
        "colab_type": "code",
        "outputId": "4f1a32b2-e650-45cd-c84a-d2615be5f581",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "var_index_param = {\"Volume\":df[\"Volume\"].values, \"Open\": df[\"Open\"].values}\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def var_index(time,param=var_index_param):\n",
        "    final = []\n",
        "    keys = []\n",
        "    for key, magnitude in param.items():\n",
        "      w = 1.0 / np.power(np.subtract(time[1:], time[:-1]), 2)\n",
        "      w_mean = np.mean(w)\n",
        "\n",
        "      N = len(time)\n",
        "      sigma2 = np.var(magnitude)\n",
        "\n",
        "      S1 = sum(w * (magnitude[1:] - magnitude[:-1]) ** 2)\n",
        "      S2 = sum(w)\n",
        "\n",
        "      eta_e = (w_mean * np.power(time[N - 1] -\n",
        "                time[0], 2) * S1 / (sigma2 * S2 * N ** 2))\n",
        "      final.append(eta_e)\n",
        "      keys.append(key)\n",
        "    return {\"Interact__{}\".format(k): eta_e for eta_e, k in zip(final,keys) }\n",
        "\n",
        "extract.var_index(df[\"Close\"].values,var_index_param)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Interact__Open': 0.00396022538846289,\n",
              " 'Interact__Volume': 0.20550155114176533}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_9Ufe3hs1ep",
        "colab_type": "text"
      },
      "source": [
        "#### **(6) Shape**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S-f-y-mL61ST",
        "colab_type": "text"
      },
      "source": [
        "Features that emphasises a particular shape not ordinarily considered as a distribution statistic. Extends to derivations of the original time series too For example a feature looking at the sinusoidal shape of an autocorrelation plot."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Es5z7i1ls3Ys",
        "colab_type": "text"
      },
      "source": [
        "(i) Symmetrical"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nv4cDPFL6gsi",
        "colab_type": "text"
      },
      "source": [
        "Boolean variable denoting if the distribution of x looks symmetric."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cyyjgBP4sr99",
        "colab_type": "code",
        "outputId": "4d0d8ab0-0af2-4a53-aa80-ac9514e6650d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def symmetry_looking(x, param=[{\"r\": 0.2}]):\n",
        "\n",
        "    if not isinstance(x, (np.ndarray, pd.Series)):\n",
        "        x = np.asarray(x)\n",
        "    mean_median_difference = np.abs(np.mean(x) - np.median(x))\n",
        "    max_min_difference = np.max(x) - np.min(x)\n",
        "    return [(\"r_{}\".format(r[\"r\"]), mean_median_difference < (r[\"r\"] * max_min_difference))\n",
        "            for r in param]\n",
        "            \n",
        "extract.symmetry_looking(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('r_0.2', True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C8Iifk7Ct3rC",
        "colab_type": "text"
      },
      "source": [
        "#### **(7) Occurence**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wcB7hMyozWNr",
        "colab_type": "text"
      },
      "source": [
        "Looking at the occurance, and reocurance of defined values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dhswQ1VOuDKl",
        "colab_type": "text"
      },
      "source": [
        "(i) Has Duplicate Max"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "buub5ZZ_tHmj",
        "colab_type": "code",
        "outputId": "2701c86c-b47e-4078-b57a-8f5f390aa8a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def has_duplicate_max(x):\n",
        "    \"\"\"\n",
        "    Checks if the maximum value of x is observed more than once\n",
        "\n",
        "    :param x: the time series to calculate the feature of\n",
        "    :type x: numpy.ndarray\n",
        "    :return: the value of this feature\n",
        "    :return type: bool\n",
        "    \"\"\"\n",
        "    if not isinstance(x, (np.ndarray, pd.Series)):\n",
        "        x = np.asarray(x)\n",
        "    return np.sum(x == np.max(x)) >= 2\n",
        "\n",
        "extract.has_duplicate_max(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IDvTaGA8ufg4",
        "colab_type": "text"
      },
      "source": [
        "#### **(8) Autocorrelation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32KuYtOC8ILF",
        "colab_type": "text"
      },
      "source": [
        "Autocorrelation, also known as serial correlation, is the correlation of a signal with a delayed copy of itself as a function of delay. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pd3TQKWxuiYC",
        "colab_type": "text"
      },
      "source": [
        "(i) Partial Autocorrelation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ys-cKmTs8MZK",
        "colab_type": "text"
      },
      "source": [
        "Partial autocorrelation is a summary of the relationship between an observation in a time series with observations at prior time steps with the relationships of intervening observations removed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8IcrHJTDuehQ",
        "colab_type": "code",
        "outputId": "5a26a570-aa4d-47c0-a9a6-dd846c089c26",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "from statsmodels.tsa.stattools import acf, adfuller, pacf\n",
        "\n",
        "def partial_autocorrelation(x, param=[{\"lag\": 1}]):\n",
        "\n",
        "    # Check the difference between demanded lags by param and possible lags to calculate (depends on len(x))\n",
        "    max_demanded_lag = max([lag[\"lag\"] for lag in param])\n",
        "    n = len(x)\n",
        "\n",
        "    # Check if list is too short to make calculations\n",
        "    if n <= 1:\n",
        "        pacf_coeffs = [np.nan] * (max_demanded_lag + 1)\n",
        "    else:\n",
        "        if (n <= max_demanded_lag):\n",
        "            max_lag = n - 1\n",
        "        else:\n",
        "            max_lag = max_demanded_lag\n",
        "        pacf_coeffs = list(pacf(x, method=\"ld\", nlags=max_lag))\n",
        "        pacf_coeffs = pacf_coeffs + [np.nan] * max(0, (max_demanded_lag - max_lag))\n",
        "\n",
        "    return [(\"lag_{}\".format(lag[\"lag\"]), pacf_coeffs[lag[\"lag\"]]) for lag in param]\n",
        "\n",
        "extract.partial_autocorrelation(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('lag_1', 0.9749503955091001)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tOMi6j_svQNk",
        "colab_type": "text"
      },
      "source": [
        "#### **(9) Stochasticity**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GjUmaNT7JjMu",
        "colab_type": "text"
      },
      "source": [
        "Stochastic refers to a randomly determined process. Any features trying to capture stocahsticity by degree or type are included under this branch."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3JCAb11svSKF",
        "colab_type": "text"
      },
      "source": [
        "(i) Augmented Dickey Fuller"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EVo06ozs8Vc2",
        "colab_type": "text"
      },
      "source": [
        "The Augmented Dickey-Fuller test is a hypothesis test which checks whether a unit root is present in a time series sample."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x7U2k3iNuPBe",
        "colab_type": "code",
        "outputId": "ff7d5e72-f9b6-4e65-8a37-65c02bc849e4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def augmented_dickey_fuller(x, param=[{\"attr\": \"teststat\"}]):\n",
        "\n",
        "    res = None\n",
        "    try:\n",
        "        res = adfuller(x)\n",
        "    except LinAlgError:\n",
        "        res = np.NaN, np.NaN, np.NaN\n",
        "    except ValueError: # occurs if sample size is too small\n",
        "        res = np.NaN, np.NaN, np.NaN\n",
        "    except MissingDataError: # is thrown for e.g. inf or nan in the data\n",
        "        res = np.NaN, np.NaN, np.NaN\n",
        "\n",
        "    return [('attr_\"{}\"'.format(config[\"attr\"]),\n",
        "                  res[0] if config[\"attr\"] == \"teststat\"\n",
        "             else res[1] if config[\"attr\"] == \"pvalue\"\n",
        "             else res[2] if config[\"attr\"] == \"usedlag\" else np.NaN)\n",
        "            for config in param]\n",
        "\n",
        "extract.augmented_dickey_fuller(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('attr_\"teststat\"', 0.04838643101451489)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b0gZYnfbwQZT",
        "colab_type": "text"
      },
      "source": [
        "#### **(10) Averages**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cSrwkSgYwaeh",
        "colab_type": "text"
      },
      "source": [
        "(i) Median of Magnitudes Skew"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OAydVohLUAOy",
        "colab_type": "code",
        "outputId": "31bb305c-18f8-4526-ce45-e34871b6a076",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "@set_property(\"fctype\", \"simple\")\n",
        "@set_property(\"custom\", True)\n",
        "def gskew(x):\n",
        "    interpolation=\"nearest\"\n",
        "    median_mag = np.median(x)\n",
        "    F_3_value = np.percentile(x, 3, interpolation=interpolation)\n",
        "    F_97_value = np.percentile(x, 97, interpolation=interpolation)\n",
        "\n",
        "    skew = (np.median(x[x <= F_3_value]) +\n",
        "            np.median(x[x >= F_97_value]) - 2 * median_mag)\n",
        "\n",
        "    return skew\n",
        "\n",
        "extract.gskew(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "84.21998596191395"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x4xkIWITJG-E",
        "colab_type": "text"
      },
      "source": [
        "(ii) Stetson Mean"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdJlbcSpJYMF",
        "colab_type": "text"
      },
      "source": [
        "An iteratively weighted mean used in the Stetson variability index"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hxHjSQkAJE1b",
        "colab_type": "code",
        "outputId": "9bff3a45-6701-4805-9cfc-5c520cc4897e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "stestson_param = {\"weight\":100., \"alpha\":2., \"beta\":2., \"tol\":1.e-6, \"nmax\":20}\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def stetson_mean(x, param=stestson_param):\n",
        "    \n",
        "    weight= stestson_param[\"weight\"]\n",
        "    alpha= stestson_param[\"alpha\"]\n",
        "    beta = stestson_param[\"beta\"]\n",
        "    tol= stestson_param[\"tol\"]\n",
        "    nmax= stestson_param[\"nmax\"]\n",
        "    \n",
        "    \n",
        "    mu = np.median(x)\n",
        "    for i in range(nmax):\n",
        "        resid = x - mu\n",
        "        resid_err = np.abs(resid) * np.sqrt(weight)\n",
        "        weight1 = weight / (1. + (resid_err / alpha)**beta)\n",
        "        weight1 /= weight1.mean()\n",
        "        diff = np.mean(x * weight1) - mu\n",
        "        mu += diff\n",
        "        if (np.abs(diff) < tol*np.abs(mu) or np.abs(diff) < tol):\n",
        "            break\n",
        "\n",
        "    return mu\n",
        "\n",
        "extract.stetson_mean(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "259.7271079387519"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4p9OXv2CwmBo",
        "colab_type": "text"
      },
      "source": [
        "#### **(11) Size**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyGCz_8xwoHY",
        "colab_type": "text"
      },
      "source": [
        "(i) Lenght"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ol0a4oiXwg1e",
        "colab_type": "code",
        "outputId": "2bc76458-03ab-4a20-9827-4278f3cf07d9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def length(x):\n",
        "    return len(x)\n",
        "    \n",
        "extract.length(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "251"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1aeY3c-I9mMj",
        "colab_type": "text"
      },
      "source": [
        "#### **(12) Count**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fJ6PH-2q9qVJ",
        "colab_type": "text"
      },
      "source": [
        "(i) Count Above Mean"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8KHym6Pv9uFa",
        "colab_type": "text"
      },
      "source": [
        "Returns the number of values in x that are higher than the mean of x"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WIKY_BYH9wR5",
        "colab_type": "code",
        "outputId": "61619c45-cd70-4133-85f3-555de7095e1a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def count_above_mean(x):\n",
        "    m = np.mean(x)\n",
        "    return np.where(x > m)[0].size\n",
        "\n",
        "extract.count_above_mean(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "110"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aqs9XcJFxGd1",
        "colab_type": "text"
      },
      "source": [
        "#### **(13) Streaks**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k69I0pV6xJui",
        "colab_type": "text"
      },
      "source": [
        "(i) Longest Strike Below Mean"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZEFYOgw8d5N",
        "colab_type": "text"
      },
      "source": [
        "Returns the length of the longest consecutive subsequence in x that is smaller than the mean of x"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q5AhNHBXw-f8",
        "colab_type": "code",
        "outputId": "4f2da68a-e287-46cc-a970-4f4cc1adce04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "import itertools\n",
        "def get_length_sequences_where(x):\n",
        "\n",
        "    if len(x) == 0:\n",
        "        return [0]\n",
        "    else:\n",
        "        res = [len(list(group)) for value, group in itertools.groupby(x) if value == 1]\n",
        "        return res if len(res) > 0 else [0]\n",
        "\n",
        "def longest_strike_below_mean(x):\n",
        "\n",
        "    if not isinstance(x, (np.ndarray, pd.Series)):\n",
        "        x = np.asarray(x)\n",
        "    return np.max(get_length_sequences_where(x <= np.mean(x))) if x.size > 0 else 0\n",
        "\n",
        "extract.longest_strike_below_mean(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "130"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QxObz12SVv-j",
        "colab_type": "text"
      },
      "source": [
        "(ii) Wozniak"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "srsbklR-WS4S",
        "colab_type": "text"
      },
      "source": [
        "This is an astronomical feature, we count the number of three consecutive data points that are brighter or fainter than $2σ$ and normalize the number by $N−2$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eFimxbQjQcIs",
        "colab_type": "code",
        "outputId": "8fdb0401-428a-49ec-e412-f6f6fc639970",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        }
      },
      "source": [
        "woz_param = [{\"consecutiveStar\": n} for n in [2, 4]]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def wozniak(magnitude, param=woz_param):\n",
        "\n",
        "    iters = []\n",
        "    for consecutiveStar in [stars[\"consecutiveStar\"] for stars in param]:\n",
        "      N = len(magnitude)\n",
        "      if N < consecutiveStar:\n",
        "          return 0\n",
        "      sigma = np.std(magnitude)\n",
        "      m = np.mean(magnitude)\n",
        "      count = 0\n",
        "\n",
        "      for i in range(N - consecutiveStar + 1):\n",
        "          flag = 0\n",
        "          for j in range(consecutiveStar):\n",
        "              if(magnitude[i + j] > m + 2 * sigma or\n",
        "                  magnitude[i + j] < m - 2 * sigma):\n",
        "                  flag = 1\n",
        "              else:\n",
        "                  flag = 0\n",
        "                  break\n",
        "          if flag:\n",
        "              count = count + 1\n",
        "      iters.append(count * 1.0 / (N - consecutiveStar + 1))\n",
        "\n",
        "    return [(\"consecutiveStar_{}\".format(config[\"consecutiveStar\"]), iters[en] )  for en, config in enumerate(param)]\n",
        "\n",
        "extract.wozniak(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('consecutiveStar_2', 0.036), ('consecutiveStar_4', 0.028225806451612902)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6MtonX1ryger",
        "colab_type": "text"
      },
      "source": [
        "####  **(14) Location**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "plrQxIhVykWl",
        "colab_type": "text"
      },
      "source": [
        "(i) Last location of Maximum"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6Xns3td8xJg",
        "colab_type": "text"
      },
      "source": [
        "Returns the relative last location of the maximum value of x.\n",
        "last_location_of_minimum(x),"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gTAK1vTqyidL",
        "colab_type": "code",
        "outputId": "3772f3e6-a14f-45b9-eb9d-252f64725372",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def last_location_of_maximum(x):\n",
        "\n",
        "    x = np.asarray(x)\n",
        "    return 1.0 - np.argmax(x[::-1]) / len(x) if len(x) > 0 else np.NaN\n",
        "\n",
        "extract.last_location_of_maximum(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9920318725099602"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5PNG551zktJ",
        "colab_type": "text"
      },
      "source": [
        "#### **(15) Model Coefficients**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R07AVQGl-J6X",
        "colab_type": "text"
      },
      "source": [
        "Any coefficient that are obtained from a model that might help in the prediction problem. For example here we might include coefficients of polynomial $h(x)$, which has been fitted to the deterministic dynamics of Langevin model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qwvn54dNzoUX",
        "colab_type": "text"
      },
      "source": [
        "(i) FFT Coefficient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NXgH5Coc81PQ",
        "colab_type": "text"
      },
      "source": [
        "Calculates the fourier coefficients of the one-dimensional discrete Fourier Transform for real input."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ihw0nVm6zAgB",
        "colab_type": "code",
        "outputId": "9aa79def-142d-4b61-b754-26d228df732f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def fft_coefficient(x, param = [{\"coeff\": 10, \"attr\": \"real\"}]):\n",
        "\n",
        "    assert min([config[\"coeff\"] for config in param]) >= 0, \"Coefficients must be positive or zero.\"\n",
        "    assert set([config[\"attr\"] for config in param]) <= set([\"imag\", \"real\", \"abs\", \"angle\"]), \\\n",
        "        'Attribute must be \"real\", \"imag\", \"angle\" or \"abs\"'\n",
        "\n",
        "    fft = np.fft.rfft(x)\n",
        "\n",
        "    def complex_agg(x, agg):\n",
        "        if agg == \"real\":\n",
        "            return x.real\n",
        "        elif agg == \"imag\":\n",
        "            return x.imag\n",
        "        elif agg == \"abs\":\n",
        "            return np.abs(x)\n",
        "        elif agg == \"angle\":\n",
        "            return np.angle(x, deg=True)\n",
        "\n",
        "    res = [complex_agg(fft[config[\"coeff\"]], config[\"attr\"]) if config[\"coeff\"] < len(fft)\n",
        "           else np.NaN for config in param]\n",
        "    index = [('coeff_{}__attr_\"{}\"'.format(config[\"coeff\"], config[\"attr\"]),res[0]) for config in param]\n",
        "    return index\n",
        "\n",
        "extract.fft_coefficient(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('coeff_10__attr_\"real\"', -245.47932447962748)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "plFmRWC7_ZVr",
        "colab_type": "text"
      },
      "source": [
        "(ii) AR Coefficient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "muhTOCEU_rOc",
        "colab_type": "text"
      },
      "source": [
        "This feature calculator fits the unconditional maximum likelihood of an autoregressive AR(k) process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lDZCUM2L_cBz",
        "colab_type": "code",
        "outputId": "aee117c9-52e9-40ea-ad62-cfe8f27bd8d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "from statsmodels.tsa.ar_model import AR\n",
        "\n",
        "def ar_coefficient(x, param=[{\"coeff\": 5, \"k\": 5}]):\n",
        "\n",
        "    calculated_ar_params = {}\n",
        "\n",
        "    x_as_list = list(x)\n",
        "    calculated_AR = AR(x_as_list)\n",
        "\n",
        "    res = {}\n",
        "\n",
        "    for parameter_combination in param:\n",
        "        k = parameter_combination[\"k\"]\n",
        "        p = parameter_combination[\"coeff\"]\n",
        "\n",
        "        column_name = \"k_{}__coeff_{}\".format(k, p)\n",
        "\n",
        "        if k not in calculated_ar_params:\n",
        "            try:\n",
        "                calculated_ar_params[k] = calculated_AR.fit(maxlag=k, solver=\"mle\").params\n",
        "            except (LinAlgError, ValueError):\n",
        "                calculated_ar_params[k] = [np.NaN]*k\n",
        "\n",
        "        mod = calculated_ar_params[k]\n",
        "\n",
        "        if p <= k:\n",
        "            try:\n",
        "                res[column_name] = mod[p]\n",
        "            except IndexError:\n",
        "                res[column_name] = 0\n",
        "        else:\n",
        "            res[column_name] = np.NaN\n",
        "\n",
        "    return [(key, value) for key, value in res.items()]\n",
        "\n",
        "extract.ar_coefficient(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('k_5__coeff_5', -0.055200871031580195)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "spXFlQUm0ipH",
        "colab_type": "text"
      },
      "source": [
        "#### **(16) Quantiles**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KjuPxGXD9NP0",
        "colab_type": "text"
      },
      "source": [
        "This includes finding normal quantile values in the series, but also quantile derived measures like change quantiles and index max quantiles. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ZBHs1ol0lS8",
        "colab_type": "text"
      },
      "source": [
        "(i) Index Mass Quantile"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "flkZUrHK87k4",
        "colab_type": "text"
      },
      "source": [
        "The relative index $i$ where $q\\%$ of the mass of the time series $x$ lie left of $i$\n",
        "."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5-guME2D0CFu",
        "colab_type": "code",
        "outputId": "2c19bf23-7686-4821-c028-5f34394dbac3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def index_mass_quantile(x, param=[{\"q\": 0.3}]):\n",
        "\n",
        "    x = np.asarray(x)\n",
        "    abs_x = np.abs(x)\n",
        "    s = sum(abs_x)\n",
        "\n",
        "    if s == 0:\n",
        "        # all values in x are zero or it has length 0\n",
        "        return [(\"q_{}\".format(config[\"q\"]), np.NaN) for config in param]\n",
        "    else:\n",
        "        # at least one value is not zero\n",
        "        mass_centralized = np.cumsum(abs_x) / s\n",
        "        return [(\"q_{}\".format(config[\"q\"]), (np.argmax(mass_centralized >= config[\"q\"])+1)/len(x)) for config in param]\n",
        "\n",
        "extract.index_mass_quantile(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('q_0.3', 0.2749003984063745)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZ1apEEC1cde",
        "colab_type": "text"
      },
      "source": [
        "#### **(17) Peaks**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bkA_Lova1ktL",
        "colab_type": "text"
      },
      "source": [
        "(i) Number of CWT Peaks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dNJPwtel-2Ff",
        "colab_type": "text"
      },
      "source": [
        "This feature calculator searches for different peaks in x."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qnQez2HK01sE",
        "colab_type": "code",
        "outputId": "8de6f510-7a12-4fc5-d324-1c6c3fd2544a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "from scipy.signal import cwt, find_peaks_cwt, ricker, welch\n",
        "\n",
        "cwt_param = [ka for ka in [2,6,9]]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def number_cwt_peaks(x, param=cwt_param):\n",
        "\n",
        "    return [(\"CWTPeak_{}\".format(n), len(find_peaks_cwt(vector=x, widths=np.array(list(range(1, n + 1))), wavelet=ricker))) for n in param]\n",
        "\n",
        "extract.number_cwt_peaks(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('CWTPeak_2', 37), ('CWTPeak_6', 23), ('CWTPeak_9', 12)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DxehUR8F2POM",
        "colab_type": "text"
      },
      "source": [
        "#### **(18) Density**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qlULvI8U_BkZ",
        "colab_type": "text"
      },
      "source": [
        "The density, and more specifically the power spectral density of the signal describes the power present in the signal as a function of frequency, per unit frequency."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xp4pmdte2SKJ",
        "colab_type": "text"
      },
      "source": [
        "(i) Cross Power Spectral Density"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8fDye5ll-6rk",
        "colab_type": "text"
      },
      "source": [
        "This feature calculator estimates the cross power spectral density of the time series $x$ at different frequencies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bl_4n3YB15Xa",
        "colab_type": "code",
        "outputId": "40244dcc-9189-434c-ada0-c9b14df54e8e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def spkt_welch_density(x, param=[{\"coeff\": 5}]):\n",
        "    freq, pxx = welch(x, nperseg=min(len(x), 256))\n",
        "    coeff = [config[\"coeff\"] for config in param]\n",
        "    indices = [\"coeff_{}\".format(i) for i in coeff]\n",
        "\n",
        "    if len(pxx) <= np.max(coeff):  # There are fewer data points in the time series than requested coefficients\n",
        "\n",
        "        # filter coefficients that are not contained in pxx\n",
        "        reduced_coeff = [coefficient for coefficient in coeff if len(pxx) > coefficient]\n",
        "        not_calculated_coefficients = [coefficient for coefficient in coeff\n",
        "                                       if coefficient not in reduced_coeff]\n",
        "\n",
        "        # Fill up the rest of the requested coefficients with np.NaNs\n",
        "        return zip(indices, list(pxx[reduced_coeff]) + [np.NaN] * len(not_calculated_coefficients))\n",
        "    else:\n",
        "        return pxx[coeff].ravel()[0]\n",
        "\n",
        "extract.spkt_welch_density(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "9997.022760604385"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GeO2VGEv3mZk",
        "colab_type": "text"
      },
      "source": [
        "#### **(19) Linearity**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZzAopxlO_1Ho",
        "colab_type": "text"
      },
      "source": [
        "Any measure of linearity that might make use of something like the linear least-squares regression for the values of the time series. This can be against the time series minus one and many other alternatives."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXcuvwY832nm",
        "colab_type": "text"
      },
      "source": [
        "(i) Linear Trend Time Wise"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BiVFMeMvAmJS",
        "colab_type": "text"
      },
      "source": [
        "Calculate a linear least-squares regression for the values of the time series versus the sequence from 0 to length of the time series minus one."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AMYFrAU_3SUZ",
        "colab_type": "code",
        "outputId": "64d23a28-9ca1-42d4-de6a-795ca49e9f12",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "from scipy.stats import linregress\n",
        "\n",
        "#-> In Package\n",
        "def linear_trend_timewise(x, param= [{\"attr\": \"pvalue\"}]):\n",
        "\n",
        "    ix = x.index\n",
        "\n",
        "    # Get differences between each timestamp and the first timestamp in seconds.\n",
        "    # Then convert to hours and reshape for linear regression\n",
        "    times_seconds = (ix - ix[0]).total_seconds()\n",
        "    times_hours = np.asarray(times_seconds / float(3600))\n",
        "\n",
        "    linReg = linregress(times_hours, x.values)\n",
        "\n",
        "    return [(\"attr_\\\"{}\\\"\".format(config[\"attr\"]), getattr(linReg, config[\"attr\"]))\n",
        "            for config in param]\n",
        "\n",
        "extract.linear_trend_timewise(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('attr_\"pvalue\"', 0.010927064557243151)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Aset52xnA5KG",
        "colab_type": "text"
      },
      "source": [
        "#### **(20) Non-Linearity**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I74aZWndBEeo",
        "colab_type": "text"
      },
      "source": [
        "(i) Schreiber Non-Linearity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V9GMNlcwBGbI",
        "colab_type": "code",
        "outputId": "adb9923b-4200-4392-b2bd-ddb9d9eea9fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def c3(x, lag=3):\n",
        "    if not isinstance(x, (np.ndarray, pd.Series)):\n",
        "        x = np.asarray(x)\n",
        "    n = x.size\n",
        "    if 2 * lag >= n:\n",
        "        return 0\n",
        "    else:\n",
        "        return np.mean((_roll(x, 2 * -lag) * _roll(x, -lag) * x)[0:(n - 2 * lag)])\n",
        "\n",
        "extract.c3(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "21764331.61192791"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EVLZqSY54MCz",
        "colab_type": "text"
      },
      "source": [
        "#### **(21) Entropy**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ab9wJQCMBvwL",
        "colab_type": "text"
      },
      "source": [
        "Any feature looking at the complexity of a time series. This is typically used in medical signal disciplines (EEG, EMG). There are multile types of measures like spectral entropy, permutation entropy, sample entropy, approximate entropy, Lempel-Ziv complexity and other. This includes entropy measures and there derivations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CL0ojbve4OAZ",
        "colab_type": "text"
      },
      "source": [
        "(i) Binned Entropy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oGIf7itZFsua",
        "colab_type": "text"
      },
      "source": [
        "Bins the values of x into max_bins equidistant bins."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "srwAHzSn4E3l",
        "colab_type": "code",
        "outputId": "73aedaa1-21fd-4be2-fedd-16a616568d76",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def binned_entropy(x, max_bins=10):\n",
        "    if not isinstance(x, (np.ndarray, pd.Series)):\n",
        "        x = np.asarray(x)\n",
        "    hist, bin_edges = np.histogram(x, bins=max_bins)\n",
        "    probs = hist / x.size\n",
        "    return - np.sum(p * np.math.log(p) for p in probs if p != 0)\n",
        "\n",
        "extract.binned_entropy(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2.0153356708276275"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DWzpodZDHgeK",
        "colab_type": "text"
      },
      "source": [
        "(ii) SVD Entropy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0SKIOuI9FoDc",
        "colab_type": "text"
      },
      "source": [
        "SVD entropy is an indicator of the number of eigenvectors that are needed for an adequate explanation of the data set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rTKMIdBvHjU6",
        "colab_type": "code",
        "outputId": "9e4fa761-d686-4771-ac94-87506f5297ee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "svd_param = [{\"Tau\": ta, \"DE\": de}\n",
        "                      for ta in [4] \n",
        "                      for de in [3,6]]\n",
        "                      \n",
        "def _embed_seq(X,Tau,D):\n",
        "  N =len(X)\n",
        "  if D * Tau > N:\n",
        "      print(\"Cannot build such a matrix, because D * Tau > N\")\n",
        "      exit()\n",
        "  if Tau<1:\n",
        "      print(\"Tau has to be at least 1\")\n",
        "      exit()\n",
        "  Y= np.zeros((N - (D - 1) * Tau, D))\n",
        "\n",
        "  for i in range(0, N - (D - 1) * Tau):\n",
        "      for j in range(0, D):\n",
        "          Y[i][j] = X[i + j * Tau]\n",
        "  return Y                     \n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def svd_entropy(epochs, param=svd_param):\n",
        "    axis=0\n",
        "    \n",
        "    final = []\n",
        "    for par in param:\n",
        "\n",
        "      def svd_entropy_1d(X, Tau, DE):\n",
        "          Y = _embed_seq(X, Tau, DE)\n",
        "          W = np.linalg.svd(Y, compute_uv=0)\n",
        "          W /= sum(W)  # normalize singular values\n",
        "          return -1 * np.sum(W * np.log(W))\n",
        "\n",
        "      Tau = par[\"Tau\"]\n",
        "      DE = par[\"DE\"]\n",
        "\n",
        "      final.append(np.apply_along_axis(svd_entropy_1d, axis, epochs, Tau, DE).ravel()[0])\n",
        "\n",
        "\n",
        "    return [(\"Tau_\\\"{}\\\"__De_{}\\\"\".format(par[\"Tau\"], par[\"DE\"]), final[en]) for en, par in enumerate(param)]\n",
        "\n",
        "extract.svd_entropy(df[\"Close\"].values)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Tau_\"4\"__De_3\"', 0.2503978615193672),\n",
              " ('Tau_\"4\"__De_6\"', 0.4997955153438164)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_MaNmr6B8rA",
        "colab_type": "text"
      },
      "source": [
        "(iii) Hjort"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3sQSeeBrGGf8",
        "colab_type": "text"
      },
      "source": [
        "The Complexity parameter represents the change in frequency. The parameter compares the signal's similarity to a pure sine wave, where the value converges to 1 if the signal is more similar. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fxpnnMU7B-ln",
        "colab_type": "code",
        "outputId": "0c68540f-0499-4838-bb2a-471e554ba45e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "def _hjorth_mobility(epochs):\n",
        "    diff = np.diff(epochs, axis=0)\n",
        "    sigma0 = np.std(epochs, axis=0)\n",
        "    sigma1 = np.std(diff, axis=0)\n",
        "    return np.divide(sigma1, sigma0)\n",
        "\n",
        "@set_property(\"fctype\", \"simple\")\n",
        "@set_property(\"custom\", True)\n",
        "def hjorth_complexity(epochs):\n",
        "    diff1 = np.diff(epochs, axis=0)\n",
        "    diff2 = np.diff(diff1, axis=0)\n",
        "    sigma1 = np.std(diff1, axis=0)\n",
        "    sigma2 = np.std(diff2, axis=0)\n",
        "    return np.divide(np.divide(sigma2, sigma1), _hjorth_mobility(epochs))\n",
        "\n",
        "extract.hjorth_complexity(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "8.311414626256186"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sox3PU6u5D63",
        "colab_type": "text"
      },
      "source": [
        "#### **(22) Fixed Points**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CXnwjuw8GmPG",
        "colab_type": "text"
      },
      "source": [
        "Fixed points and equilibria as identified from fitted models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FlRIYyU5JvG",
        "colab_type": "text"
      },
      "source": [
        "(i) Langevin Fixed Points"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DC5MXnLCGNuU",
        "colab_type": "text"
      },
      "source": [
        "Largest fixed point of dynamics $max\\ {h(x)=0}$ estimated from polynomial $h(x)$ which has been fitted to the deterministic dynamics of Langevin model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NfQCSl964bGP",
        "colab_type": "code",
        "outputId": "c32bbb3b-03e4-4aad-bffa-ab1d633e3a7a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def _estimate_friedrich_coefficients(x, m, r):\n",
        "    assert m > 0, \"Order of polynomial need to be positive integer, found {}\".format(m)\n",
        "    df = pd.DataFrame({'signal': x[:-1], 'delta': np.diff(x)})\n",
        "    try:\n",
        "        df['quantiles'] = pd.qcut(df.signal, r)\n",
        "    except ValueError:\n",
        "        return [np.NaN] * (m + 1)\n",
        "\n",
        "    quantiles = df.groupby('quantiles')\n",
        "\n",
        "    result = pd.DataFrame({'x_mean': quantiles.signal.mean(), 'y_mean': quantiles.delta.mean()})\n",
        "    result.dropna(inplace=True)\n",
        "\n",
        "    try:\n",
        "        return np.polyfit(result.x_mean, result.y_mean, deg=m)\n",
        "    except (np.linalg.LinAlgError, ValueError):\n",
        "        return [np.NaN] * (m + 1)\n",
        "\n",
        "\n",
        "def max_langevin_fixed_point(x, r=3, m=30):\n",
        "    coeff = _estimate_friedrich_coefficients(x, m, r)\n",
        "\n",
        "    try:\n",
        "        max_fixed_point = np.max(np.real(np.roots(coeff)))\n",
        "    except (np.linalg.LinAlgError, ValueError):\n",
        "        return np.nan\n",
        "\n",
        "    return max_fixed_point\n",
        "\n",
        "extract.max_langevin_fixed_point(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "334.4064112915032"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rrkyEFjm9oXe",
        "colab_type": "text"
      },
      "source": [
        "#### **(23) Amplitude**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ILxCeYNhJ55y",
        "colab_type": "text"
      },
      "source": [
        "Features derived from peaked values in either the positive or negative direction. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IfsdsD7C8-aA",
        "colab_type": "text"
      },
      "source": [
        "(i) Willison Amplitude"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtiR44yuH2IC",
        "colab_type": "text"
      },
      "source": [
        "This feature is defined as the amount of times that the change in the signal amplitude exceeds a threshold."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WehSArJz5ic3",
        "colab_type": "code",
        "outputId": "41888e57-dc5d-4940-8bf6-9e548933756e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "will_param = [ka for ka in [0.2,3]]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def willison_amplitude(X, param=will_param):\n",
        "  return [(\"Thresh_{}\".format(n),np.sum(np.abs(np.diff(X)) >= n)) for n in param]\n",
        "\n",
        "extract.willison_amplitude(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Thresh_0.2', 249), ('Thresh_3', 151)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0m9aki9KrSe",
        "colab_type": "text"
      },
      "source": [
        "(ii) Percent Amplitude"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxLgtQ3HHt-_",
        "colab_type": "text"
      },
      "source": [
        "Returns the largest distance from the median value, measured\n",
        "    as a percentage of the median"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ATenPUxILztq",
        "colab_type": "code",
        "outputId": "a7f2bd41-e59c-46be-993c-a3e5a908abed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "perc_param = [{\"base\":ba, \"exponent\":exp} for ba in [3,5] for exp in [-0.1,-0.2]]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def percent_amplitude(x, param =perc_param):\n",
        "    final = []\n",
        "    for par in param:\n",
        "      linear_scale_data = par[\"base\"] ** (par[\"exponent\"] * x)\n",
        "      y_max = np.max(linear_scale_data)\n",
        "      y_min = np.min(linear_scale_data)\n",
        "      y_med = np.median(linear_scale_data)\n",
        "      final.append(max(abs((y_max - y_med) / y_med), abs((y_med - y_min) / y_med)))\n",
        "\n",
        "    return [(\"Base_{}__Exp{}\".format(pa[\"base\"],pa[\"exponent\"]),fin) for fin, pa in zip(final,param)]\n",
        "\n",
        "extract.percent_amplitude(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Base_3__Exp-0.1', 7484.575081248652),\n",
              " ('Base_3__Exp-0.2', 56033833.29701077),\n",
              " ('Base_5__Exp-0.1', 473845.0759281438),\n",
              " ('Base_5__Exp-0.2', 224530103671.5002)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2cMJD8dLLd7m",
        "colab_type": "text"
      },
      "source": [
        "#### **(24) Probability**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ytaXpLElIxCd",
        "colab_type": "text"
      },
      "source": [
        "(i) Cadence Probability"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CkSw-heNIjOV",
        "colab_type": "text"
      },
      "source": [
        "Given the observed distribution of time lags cads, compute the probability that the next observation occurs within time minutes of an arbitrary epoch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yst5_jL9LH94",
        "colab_type": "code",
        "outputId": "0dd21b47-eaab-42b2-824c-5ff8ba987bc2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> fixes required\n",
        "import scipy.stats as stats\n",
        "\n",
        "cad_param = [0.1,1000, -234]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def cad_prob(cads, param=cad_param):\n",
        "    return [(\"time_{}\".format(time), stats.percentileofscore(cads, float(time) / (24.0 * 60.0)) / 100.0) for time in param]\n",
        "    \n",
        "extract.cad_prob(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('time_0.1', 0.0), ('time_1000', 0.0), ('time_-234', 0.0)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aJSJ5wSOIe2q",
        "colab_type": "text"
      },
      "source": [
        "#### **(25) Crossings**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gpt88-78KQ3A",
        "colab_type": "text"
      },
      "source": [
        "Calculates the crossing of the series with other defined values or series."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rq8GnCWKIlST",
        "colab_type": "text"
      },
      "source": [
        "(i) Zero Crossing Derivative"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "svHF16XaKg83",
        "colab_type": "text"
      },
      "source": [
        "The positioning of the edge point is located at the zero crossing of the first derivative of the filter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AHSRicaiOj6O",
        "colab_type": "code",
        "outputId": "63920a76-fbf4-4ae3-90ef-516509181615",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "zero_param = [0.01, 8]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def zero_crossing_derivative(epochs, param=zero_param):\n",
        "    diff = np.diff(epochs)\n",
        "    norm = diff-diff.mean()\n",
        "    return [(\"e_{}\".format(e), np.apply_along_axis(lambda epoch: np.sum(((epoch[:-5] <= e) & (epoch[5:] > e))), 0, norm).ravel()[0]) for e in param]\n",
        "\n",
        "extract.zero_crossing_derivative(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('e_0.01', 70), ('e_8', 26)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L9F_PUJsSnS7",
        "colab_type": "text"
      },
      "source": [
        "#### **(26) Fluctuations**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gbT6u-RuLL-K",
        "colab_type": "text"
      },
      "source": [
        "These features are again from medical signal sciences, but under this category we would include values such as fluctuation based entropy measures, fluctuation of correlation dynamics, and co-fluctuations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u2AOVrs_Kst-",
        "colab_type": "text"
      },
      "source": [
        "(i) Detrended Fluctuation Analysis (DFA)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qIKOgCRCKzCn",
        "colab_type": "text"
      },
      "source": [
        "DFA Calculate the Hurst exponent using DFA analysis."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eSCDRBYKRN4_",
        "colab_type": "code",
        "outputId": "e0376159-3cff-4a48-d4cf-613e4b89a942",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "from scipy.stats import kurtosis as _kurt\n",
        "from scipy.stats import skew as _skew\n",
        "import numpy as np\n",
        "\n",
        "@set_property(\"fctype\", \"simple\")\n",
        "@set_property(\"custom\", True)\n",
        "def detrended_fluctuation_analysis(epochs):\n",
        "    def dfa_1d(X, Ave=None, L=None):\n",
        "        X = np.array(X)\n",
        "\n",
        "        if Ave is None:\n",
        "            Ave = np.mean(X)\n",
        "\n",
        "        Y = np.cumsum(X)\n",
        "        Y -= Ave\n",
        "\n",
        "        if L is None:\n",
        "            L = np.floor(len(X) * 1 / (\n",
        "                    2 ** np.array(list(range(1, int(np.log2(len(X))) - 4))))\n",
        "                            )\n",
        "            \n",
        "        F = np.zeros(len(L))  # F(n) of different given box length n\n",
        "\n",
        "        for i in range(0, len(L)):\n",
        "            n = int(L[i])  # for each box length L[i]\n",
        "            if n == 0:\n",
        "                print(\"time series is too short while the box length is too big\")\n",
        "                print(\"abort\")\n",
        "                exit()\n",
        "            for j in range(0, len(X), n):  # for each box\n",
        "                if j + n < len(X):\n",
        "                    c = list(range(j, j + n))\n",
        "                    # coordinates of time in the box\n",
        "                    c = np.vstack([c, np.ones(n)]).T\n",
        "                    # the value of data in the box\n",
        "                    y = Y[j:j + n]\n",
        "                    # add residue in this box\n",
        "                    F[i] += np.linalg.lstsq(c, y, rcond=None)[1]\n",
        "            F[i] /= ((len(X) / n) * n)\n",
        "        F = np.sqrt(F)\n",
        "\n",
        "        stacked = np.vstack([np.log(L), np.ones(len(L))])\n",
        "        stacked_t = stacked.T\n",
        "        Alpha = np.linalg.lstsq(stacked_t, np.log(F), rcond=None)\n",
        "\n",
        "        return Alpha[0][0]\n",
        "\n",
        "    return np.apply_along_axis(dfa_1d, 0, epochs).ravel()[0]\n",
        "\n",
        "extract.detrended_fluctuation_analysis(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.7717928105335718"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xrbnRy4V2Gfs",
        "colab_type": "text"
      },
      "source": [
        "#### **(27) Information**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MEwgOZVpLxpK",
        "colab_type": "text"
      },
      "source": [
        "Closely related to entropy and complexity measures. Any measure that attempts to measure the amount of information from an observable variable is included here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fVZPUPVCS2ku",
        "colab_type": "text"
      },
      "source": [
        "(i) Fisher Information"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QNSr41oRM19U",
        "colab_type": "text"
      },
      "source": [
        "Fisher information is a statistical information concept distinct from, and earlier than, Shannon information in communication theory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZcNrFmGSRgL",
        "colab_type": "code",
        "outputId": "6a1dd10c-57ae-4ac2-b3e1-722b5046d5ff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "def _embed_seq(X, Tau, D):\n",
        "\n",
        "    shape = (X.size - Tau * (D - 1), D)\n",
        "    strides = (X.itemsize, Tau * X.itemsize)\n",
        "    return np.lib.stride_tricks.as_strided(X, shape=shape, strides=strides)\n",
        "\n",
        "fisher_param = [{\"Tau\":ta, \"DE\":de} for ta in [3,15] for de in [10,5]]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def fisher_information(epochs, param=fisher_param):\n",
        "    def fisher_info_1d(a, tau, de):\n",
        "        # taken from pyeeg improvements\n",
        "\n",
        "        mat = _embed_seq(a, tau, de)\n",
        "        W = np.linalg.svd(mat, compute_uv=False)\n",
        "        W /= sum(W)  # normalize singular values\n",
        "        FI_v = (W[1:] - W[:-1]) ** 2 / W[:-1]\n",
        "        return np.sum(FI_v)\n",
        "\n",
        "    return [(\"Tau_{}__DE_{}\".format(par[\"Tau\"], par[\"DE\"]),np.apply_along_axis(fisher_info_1d, 0, epochs, par[\"Tau\"], par[\"DE\"]).ravel()[0]) for par in param]\n",
        "\n",
        "extract.fisher_information(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Tau_3__DE_10', 0.7816375325170446),\n",
              " ('Tau_3__DE_5', 0.8544647848699098),\n",
              " ('Tau_15__DE_10', 0.6420902320134164),\n",
              " ('Tau_15__DE_5', 0.7018600234807363)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6G3VSHx_2N23",
        "colab_type": "text"
      },
      "source": [
        "#### **(28) Fractals**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YzVUSmPKQOV3",
        "colab_type": "text"
      },
      "source": [
        "In mathematics, more specifically in fractal geometry, a fractal dimension is a ratio providing a statistical index of complexity comparing how detail in a pattern (strictly speaking, a fractal pattern) changes with the scale at which it is measured."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CSExVn_gT3WC",
        "colab_type": "text"
      },
      "source": [
        "(i) Highuchi Fractal"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C_yi18rsP8fU",
        "colab_type": "text"
      },
      "source": [
        "Compute a Higuchi Fractal Dimension of a time series"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KxHeqXExTvNU",
        "colab_type": "code",
        "outputId": "c8e66153-085b-4181-b806-ac562d24b76b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "hig_para = [{\"Kmax\": 3},{\"Kmax\": 5}]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def higuchi_fractal_dimension(epochs, param=hig_para):\n",
        "    def hfd_1d(X, Kmax):\n",
        "        \n",
        "        L = []\n",
        "        x = []\n",
        "        N = len(X)\n",
        "        for k in range(1, Kmax):\n",
        "            Lk = []\n",
        "            for m in range(0, k):\n",
        "                Lmk = 0\n",
        "                for i in range(1, int(np.floor((N - m) / k))):\n",
        "                    Lmk += abs(X[m + i * k] - X[m + i * k - k])\n",
        "                Lmk = Lmk * (N - 1) / np.floor((N - m) / float(k)) / k\n",
        "                Lk.append(Lmk)\n",
        "            L.append(np.log(np.mean(Lk)))\n",
        "            x.append([np.log(float(1) / k), 1])\n",
        "\n",
        "        (p, r1, r2, s) = np.linalg.lstsq(x, L, rcond=None)\n",
        "        return p[0]\n",
        "    \n",
        "    return [(\"Kmax_{}\".format(config[\"Kmax\"]), np.apply_along_axis(hfd_1d, 0, epochs, config[\"Kmax\"]).ravel()[0] ) for  config in param]\n",
        "    \n",
        "extract.higuchi_fractal_dimension(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Kmax_3', 0.36712292132178664), ('Kmax_5', 0.4052103522157522)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cNv1HZYcG7jB",
        "colab_type": "text"
      },
      "source": [
        "(ii) Petrosian Fractal"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjR9IAzjP2vn",
        "colab_type": "text"
      },
      "source": [
        "Compute a Petrosian Fractal Dimension of a time series."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LCvrzpL0G_n3",
        "colab_type": "code",
        "outputId": "2f6e4f9a-600b-451c-8b47-1dc467ad5be7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "@set_property(\"fctype\", \"simple\")\n",
        "@set_property(\"custom\", True)\n",
        "def petrosian_fractal_dimension(epochs):\n",
        "    def pfd_1d(X, D=None):\n",
        "        # taken from pyeeg\n",
        "        \"\"\"Compute Petrosian Fractal Dimension of a time series from either two\n",
        "        cases below:\n",
        "            1. X, the time series of type list (default)\n",
        "            2. D, the first order differential sequence of X (if D is provided,\n",
        "               recommended to speed up)\n",
        "        In case 1, D is computed using Numpy's difference function.\n",
        "        To speed up, it is recommended to compute D before calling this function\n",
        "        because D may also be used by other functions whereas computing it here\n",
        "        again will slow down.\n",
        "        \"\"\"\n",
        "        if D is None:\n",
        "            D = np.diff(X)\n",
        "            D = D.tolist()\n",
        "        N_delta = 0  # number of sign changes in derivative of the signal\n",
        "        for i in range(1, len(D)):\n",
        "            if D[i] * D[i - 1] < 0:\n",
        "                N_delta += 1\n",
        "        n = len(X)\n",
        "        return np.log10(n) / (np.log10(n) + np.log10(n / n + 0.4 * N_delta))\n",
        "    return np.apply_along_axis(pfd_1d, 0, epochs).ravel()[0]\n",
        "\n",
        "extract.petrosian_fractal_dimension(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5818872584355564"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wWKUuaz8EfXv",
        "colab_type": "text"
      },
      "source": [
        "#### **(29) Exponent**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPiYNi87EifC",
        "colab_type": "text"
      },
      "source": [
        "(i) Hurst Exponent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZQAyQHZJQjsO",
        "colab_type": "text"
      },
      "source": [
        "The Hurst exponent is used as a measure of long-term memory of time series. It relates to the autocorrelations of the time series, and the rate at which these decrease as the lag between pairs of values increases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s-rUoYfuD8Kd",
        "colab_type": "code",
        "outputId": "3c3a95c7-399c-4727-a710-2bdd43079057",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "@set_property(\"fctype\", \"simple\")\n",
        "@set_property(\"custom\", True)\n",
        "def hurst_exponent(epochs):\n",
        "    def hurst_1d(X):\n",
        "\n",
        "        X = np.array(X)\n",
        "        N = X.size\n",
        "        T = np.arange(1, N + 1)\n",
        "        Y = np.cumsum(X)\n",
        "        Ave_T = Y / T\n",
        "\n",
        "        S_T = np.zeros(N)\n",
        "        R_T = np.zeros(N)\n",
        "        for i in range(N):\n",
        "            S_T[i] = np.std(X[:i + 1])\n",
        "            X_T = Y - T * Ave_T[i]\n",
        "            R_T[i] = np.ptp(X_T[:i + 1])\n",
        "\n",
        "        for i in range(1, len(S_T)):\n",
        "            if np.diff(S_T)[i - 1] != 0:\n",
        "                break\n",
        "        for j in range(1, len(R_T)):\n",
        "            if np.diff(R_T)[j - 1] != 0:\n",
        "                break\n",
        "        k = max(i, j)\n",
        "        assert k < 10, \"rethink it!\"\n",
        "\n",
        "        R_S = R_T[k:] / S_T[k:]\n",
        "        R_S = np.log(R_S)\n",
        "\n",
        "        n = np.log(T)[k:]\n",
        "        A = np.column_stack((n, np.ones(n.size)))\n",
        "        [m, c] = np.linalg.lstsq(A, R_S, rcond=None)[0]\n",
        "        H = m\n",
        "        return H\n",
        "    return np.apply_along_axis(hurst_1d, 0, epochs).ravel()[0]\n",
        "\n",
        "extract.hurst_exponent(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.0019985019409188"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9j3LlrgFH0n",
        "colab_type": "text"
      },
      "source": [
        "(ii) Largest Lyauponov Exponent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MTd4Gyt6QwX8",
        "colab_type": "text"
      },
      "source": [
        "In mathematics the Lyapunov exponent or Lyapunov characteristic exponent of a dynamical system is a quantity that characterizes the rate of separation of infinitesimally close trajectories."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q-8P2fhED8Bs",
        "colab_type": "code",
        "outputId": "f2feca2f-567f-4b0c-8c73-5ee508c178b8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "def _embed_seq(X, Tau, D):\n",
        "    shape = (X.size - Tau * (D - 1), D)\n",
        "    strides = (X.itemsize, Tau * X.itemsize)\n",
        "    return np.lib.stride_tricks.as_strided(X, shape=shape, strides=strides)\n",
        "\n",
        "lyaup_param = [{\"Tau\":4, \"n\":3, \"T\":10, \"fs\":9},{\"Tau\":8, \"n\":7, \"T\":15, \"fs\":6}]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def largest_lyauponov_exponent(epochs, param=lyaup_param):\n",
        "    def LLE_1d(x, tau, n, T, fs):\n",
        "\n",
        "        Em = _embed_seq(x, tau, n)\n",
        "        M = len(Em)\n",
        "        A = np.tile(Em, (len(Em), 1, 1))\n",
        "        B = np.transpose(A, [1, 0, 2])\n",
        "        square_dists = (A - B) ** 2  # square_dists[i,j,k] = (Em[i][k]-Em[j][k])^2\n",
        "        D = np.sqrt(square_dists[:, :, :].sum(axis=2))  # D[i,j] = ||Em[i]-Em[j]||_2\n",
        "\n",
        "        # Exclude elements within T of the diagonal\n",
        "        band = np.tri(D.shape[0], k=T) - np.tri(D.shape[0], k=-T - 1)\n",
        "        band[band == 1] = np.inf\n",
        "        neighbors = (D + band).argmin(axis=0)  # nearest neighbors more than T steps away\n",
        "\n",
        "        # in_bounds[i,j] = (i+j <= M-1 and i+neighbors[j] <= M-1)\n",
        "        inc = np.tile(np.arange(M), (M, 1))\n",
        "        row_inds = (np.tile(np.arange(M), (M, 1)).T + inc)\n",
        "        col_inds = (np.tile(neighbors, (M, 1)) + inc.T)\n",
        "        in_bounds = np.logical_and(row_inds <= M - 1, col_inds <= M - 1)\n",
        "        # Uncomment for old (miscounted) version\n",
        "        # in_bounds = numpy.logical_and(row_inds < M - 1, col_inds < M - 1)\n",
        "        row_inds[~in_bounds] = 0\n",
        "        col_inds[~in_bounds] = 0\n",
        "\n",
        "        # neighbor_dists[i,j] = ||Em[i+j]-Em[i+neighbors[j]]||_2\n",
        "        neighbor_dists = np.ma.MaskedArray(D[row_inds, col_inds], ~in_bounds)\n",
        "        J = (~neighbor_dists.mask).sum(axis=1)  # number of in-bounds indices by row\n",
        "        # Set invalid (zero) values to 1; log(1) = 0 so sum is unchanged\n",
        "\n",
        "        neighbor_dists[neighbor_dists == 0] = 1\n",
        "\n",
        "        # !!! this fixes the divide by zero in log error !!!\n",
        "        neighbor_dists.data[neighbor_dists.data == 0] = 1\n",
        "\n",
        "        d_ij = np.sum(np.log(neighbor_dists.data), axis=1)\n",
        "        mean_d = d_ij[J > 0] / J[J > 0]\n",
        "\n",
        "        x = np.arange(len(mean_d))\n",
        "        X = np.vstack((x, np.ones(len(mean_d)))).T\n",
        "        [m, c] = np.linalg.lstsq(X, mean_d, rcond=None)[0]\n",
        "        Lexp = fs * m\n",
        "        return Lexp\n",
        "\n",
        "    return [(\"Tau_{}__n_{}__T_{}__fs_{}\".format(par[\"Tau\"], par[\"n\"], par[\"T\"], par[\"fs\"]), np.apply_along_axis(LLE_1d, 0, epochs, par[\"Tau\"], par[\"n\"], par[\"T\"], par[\"fs\"]).ravel()[0]) for par in param]\n",
        "  \n",
        "extract.largest_lyauponov_exponent(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Tau_4__n_3__T_10__fs_9', 0.0033118118060813636),\n",
              " ('Tau_8__n_7__T_15__fs_6', 0.007657147410596441)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "70BMBD2kzZSS",
        "colab_type": "text"
      },
      "source": [
        "#### **(30) Spectral Analysis**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ySK4IewsQ3Je",
        "colab_type": "text"
      },
      "source": [
        "Spectral analysis is analysis in terms of a spectrum of frequencies or related quantities such as energies, eigenvalues, etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGOp561zy308",
        "colab_type": "text"
      },
      "source": [
        "(i) Whelch Method"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j7nL_gnYQ-5p",
        "colab_type": "text"
      },
      "source": [
        "The Whelch Method is an approach for spectral density estimation. It is used in physics, engineering, and applied mathematics for estimating the power of a signal at different frequencies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xFyeh6dZUA-Z",
        "colab_type": "code",
        "outputId": "b3fefd39-07ac-4283-b077-790a89a65da1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "from scipy import signal, integrate\n",
        "\n",
        "whelch_param = [100,200]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def whelch_method(data, param=whelch_param):\n",
        "\n",
        "  final = []\n",
        "  for Fs in param:\n",
        "    f, pxx = signal.welch(data, fs=Fs, nperseg=1024)\n",
        "    d = {'psd': pxx, 'freqs': f}\n",
        "    df = pd.DataFrame(data=d)\n",
        "    dfs = df.sort_values(['psd'], ascending=False)\n",
        "    rows = dfs.iloc[:10]\n",
        "    final.append(rows['freqs'].mean())\n",
        "  \n",
        "  return [(\"Fs_{}\".format(pa),fin) for pa, fin in zip(param,final)]\n",
        "\n",
        "extract.whelch_method(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Fs_100', 2.151394422310757), ('Fs_200', 4.302788844621514)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ed0qlVYizbpX",
        "colab_type": "code",
        "outputId": "97ff41e7-036a-48a0-9e26-2cb346bf0260",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        }
      },
      "source": [
        "#-> Basically same as above\n",
        "freq_param = [{\"fs\":50, \"sel\":15},{\"fs\":200, \"sel\":20}]\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def find_freq(serie, param=freq_param):\n",
        "\n",
        "    final = []\n",
        "    for par in param:\n",
        "      fft0 = np.fft.rfft(serie*np.hanning(len(serie)))\n",
        "      freqs = np.fft.rfftfreq(len(serie), d=1.0/par[\"fs\"])\n",
        "      fftmod = np.array([np.sqrt(fft0[i].real**2 + fft0[i].imag**2) for i in range(0, len(fft0))])\n",
        "      d = {'fft': fftmod, 'freq': freqs}\n",
        "      df = pd.DataFrame(d)\n",
        "      hop = df.sort_values(['fft'], ascending=False)\n",
        "      rows = hop.iloc[:par[\"sel\"]]\n",
        "      final.append(rows['freq'].mean())\n",
        "\n",
        "    return [(\"Fs_{}__sel{}\".format(pa[\"fs\"],pa[\"sel\"]),fin) for pa, fin in zip(param,final)]\n",
        "\n",
        "extract.find_freq(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Fs_50__sel15', 1.7397078353253652), ('Fs_200__sel20', 9.52191235059761)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKDXk2__SeJA",
        "colab_type": "text"
      },
      "source": [
        "#### **(31) Percentile**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ovA_F7p69ih",
        "colab_type": "text"
      },
      "source": [
        "(i) Flux Percentile"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uRj4A1aTTCxt",
        "colab_type": "text"
      },
      "source": [
        "Flux (or radiant flux) is the total amount of energy that crosses a unit area per unit time. Flux is an astronomical value, measured in joules per square metre per second (joules/m2/s), or watts per square metre. Here we provide the ratio of flux percentiles."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-5Rhtjsd6-Vi",
        "colab_type": "code",
        "outputId": "55f211af-f698-4a91-ddb0-fd0ec0ef157e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "\n",
        "import math\n",
        "def flux_perc(magnitude):\n",
        "    sorted_data = np.sort(magnitude)\n",
        "    lc_length = len(sorted_data)\n",
        "\n",
        "    F_60_index = int(math.ceil(0.60 * lc_length))\n",
        "    F_40_index = int(math.ceil(0.40 * lc_length))\n",
        "    F_5_index = int(math.ceil(0.05 * lc_length))\n",
        "    F_95_index = int(math.ceil(0.95 * lc_length))\n",
        "\n",
        "    F_40_60 = sorted_data[F_60_index] - sorted_data[F_40_index]\n",
        "    F_5_95 = sorted_data[F_95_index] - sorted_data[F_5_index]\n",
        "    F_mid20 = F_40_60 / F_5_95\n",
        "\n",
        "    return {\"FluxPercentileRatioMid20\": F_mid20}\n",
        "\n",
        "extract.flux_perc(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'FluxPercentileRatioMid20': 0.20485777390978482}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J2y1Hyjv7HPt",
        "colab_type": "text"
      },
      "source": [
        "#### **(32) Range**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NYpH7Lk97I05",
        "colab_type": "text"
      },
      "source": [
        "(i) Range of Cummulative Sum"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zWQfb0Fi7LpK",
        "colab_type": "code",
        "outputId": "ac777546-a6d7-4d59-d71d-678181319af8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "@set_property(\"fctype\", \"simple\")\n",
        "@set_property(\"custom\", True)\n",
        "def range_cum_s(magnitude):\n",
        "    sigma = np.std(magnitude)\n",
        "    N = len(magnitude)\n",
        "    m = np.mean(magnitude)\n",
        "    s = np.cumsum(magnitude - m) * 1.0 / (N * sigma)\n",
        "    R = np.max(s) - np.min(s)\n",
        "    return {\"Rcs\": R}\n",
        "\n",
        "extract.range_cum_s(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Rcs': 0.4111977049788637}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eyZwuKrxKkCu",
        "colab_type": "text"
      },
      "source": [
        "#### **(33) Structural**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BGywykGaKmP3",
        "colab_type": "text"
      },
      "source": [
        "Structural features, potential placeholders for future research."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OMXpnWCjTwhj",
        "colab_type": "text"
      },
      "source": [
        "(i) Structure Function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxo8Ne26Tz2w",
        "colab_type": "text"
      },
      "source": [
        "The structure function of rotation measures (RMs) contains information on electron density and magnetic field fluctuations when used i astronomy. It becomes a custom feature when used with your own unique time series data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AyxclAdkz4QU",
        "colab_type": "code",
        "outputId": "e466b5af-d383-4b31-b0a0-7a4513a0decf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        }
      },
      "source": [
        "from scipy.interpolate import interp1d\n",
        "\n",
        "struct_param = {\"Volume\":df[\"Volume\"].values, \"Open\": df[\"Open\"].values}\n",
        "\n",
        "@set_property(\"fctype\", \"combiner\")\n",
        "@set_property(\"custom\", True)\n",
        "def structure_func(time, param=struct_param):\n",
        "\n",
        "      dict_final = {}\n",
        "      for key, magnitude in param.items():\n",
        "        dict_final[key] = []\n",
        "        Nsf, Np = 100, 100\n",
        "        sf1, sf2, sf3 = np.zeros(Nsf), np.zeros(Nsf), np.zeros(Nsf)\n",
        "        f = interp1d(time, magnitude)\n",
        "\n",
        "        time_int = np.linspace(np.min(time), np.max(time), Np)\n",
        "        mag_int = f(time_int)\n",
        "\n",
        "        for tau in np.arange(1, Nsf):\n",
        "            sf1[tau - 1] = np.mean(\n",
        "                np.power(np.abs(mag_int[0:Np - tau] - mag_int[tau:Np]), 1.0))\n",
        "            sf2[tau - 1] = np.mean(\n",
        "                np.abs(np.power(\n",
        "                    np.abs(mag_int[0:Np - tau] - mag_int[tau:Np]), 2.0)))\n",
        "            sf3[tau - 1] = np.mean(\n",
        "                np.abs(np.power(\n",
        "                    np.abs(mag_int[0:Np - tau] - mag_int[tau:Np]), 3.0)))\n",
        "        sf1_log = np.log10(np.trim_zeros(sf1))\n",
        "        sf2_log = np.log10(np.trim_zeros(sf2))\n",
        "        sf3_log = np.log10(np.trim_zeros(sf3))\n",
        "\n",
        "        if len(sf1_log) and len(sf2_log):\n",
        "            m_21, b_21 = np.polyfit(sf1_log, sf2_log, 1)\n",
        "        else:\n",
        "\n",
        "            m_21 = np.nan\n",
        "\n",
        "        if len(sf1_log) and len(sf3_log):\n",
        "            m_31, b_31 = np.polyfit(sf1_log, sf3_log, 1)\n",
        "        else:\n",
        "\n",
        "            m_31 = np.nan\n",
        "\n",
        "        if len(sf2_log) and len(sf3_log):\n",
        "            m_32, b_32 = np.polyfit(sf2_log, sf3_log, 1)\n",
        "        else:\n",
        "\n",
        "            m_32 = np.nan\n",
        "        dict_final[key].append(m_21)\n",
        "        dict_final[key].append(m_31)\n",
        "        dict_final[key].append(m_32)\n",
        "\n",
        "      return [(\"StructureFunction_{}__m_{}\".format(key, name), li)  for key, lis in dict_final.items() for name, li in zip([21,31,32], lis)]\n",
        "\n",
        "struct_param = {\"Volume\":df[\"Volume\"].values, \"Open\": df[\"Open\"].values}\n",
        "\n",
        "extract.structure_func(df[\"Close\"],struct_param)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('StructureFunction_Volume__m_21', 1.7910066048764215),\n",
              " ('StructureFunction_Volume__m_31', 2.5251433725256502),\n",
              " ('StructureFunction_Volume__m_32', 1.4702466709645565),\n",
              " ('StructureFunction_Open__m_21', 1.9137458024867238),\n",
              " ('StructureFunction_Open__m_31', 2.772674777488756),\n",
              " ('StructureFunction_Open__m_32', 1.4496743181777902)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-67Id-_hw1z0",
        "colab_type": "text"
      },
      "source": [
        "#### **(34) Distribution**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7E-EkSemw4Xy",
        "colab_type": "text"
      },
      "source": [
        "(i) Kurtosis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LQuv8_TSwxef",
        "colab_type": "code",
        "outputId": "e1a76dd8-4d97-47c1-ee11-6a23a8f6af52",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#-> In Package\n",
        "def kurtosis(x):\n",
        "\n",
        "    if not isinstance(x, pd.Series):\n",
        "        x = pd.Series(x)\n",
        "    return pd.Series.kurtosis(x)\n",
        "\n",
        "extract.kurtosis(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0690001861779832"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v-of7QmpIHur",
        "colab_type": "text"
      },
      "source": [
        "(ii) Stetson Kurtosis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "StaW0bhLIFV-",
        "colab_type": "code",
        "outputId": "f0da0d91-d6a4-4418-c559-54059ddafd23",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "@set_property(\"fctype\", \"simple\")\n",
        "@set_property(\"custom\", True)\n",
        "def stetson_k(x):\n",
        "    \"\"\"A robust kurtosis statistic.\"\"\"\n",
        "    n = len(x)\n",
        "    x0 = stetson_mean(x, 1./20**2)\n",
        "    delta_x = np.sqrt(n / (n - 1.)) * (x - x0) / 20\n",
        "    ta = 1. / 0.798 * np.mean(np.abs(delta_x)) / np.sqrt(np.mean(delta_x**2))\n",
        "    return ta\n",
        "  \n",
        "extract.stetson_k(df[\"Close\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9874979862330032"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gf6z7xIvScmO",
        "colab_type": "text"
      },
      "source": [
        "## **(5) Synthesise**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SMT6JCBXSstg",
        "colab_type": "text"
      },
      "source": [
        "Time-Series synthesisation (TSS) happens before the feature extraction step and Cross Sectional Synthesisation (CSS) happens after the feauture extraction step. Currently I will only include a CSS package, in the future, I would further work on developing out this section. This area still has a lot of performance and stability issues. In the future it might be a more viable candididate to improve prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cB-VIu6CKB1X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from lightgbm import LGBMRegressor\n",
        "from sklearn.metrics import mean_squared_error\n",
        "\n",
        "def model(df_final):\n",
        "  model = LGBMRegressor()\n",
        "  test =  df_final.head(int(len(df_final)*0.4))\n",
        "  train = df_final[~df_final.isin(test)].dropna()\n",
        "  model = model.fit(train.drop([\"Close_1\"],axis=1),train[\"Close_1\"])\n",
        "  preds = model.predict(test.drop([\"Close_1\"],axis=1))\n",
        "  test =  df_final.head(int(len(df_final)*0.4))\n",
        "  train = df_final[~df_final.isin(test)].dropna()\n",
        "  model = model.fit(train.drop([\"Close_1\"],axis=1),train[\"Close_1\"])\n",
        "  val = mean_squared_error(test[\"Close_1\"],preds); \n",
        "  return val"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BYVJbLmypKlj",
        "colab_type": "code",
        "outputId": "a93769ed-9580-405f-e1da-110a5620cebb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 248
        }
      },
      "source": [
        "!pip install ctgan"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: ctgan in /usr/local/lib/python3.6/dist-packages (0.2.1)\n",
            "Requirement already satisfied: torchvision<1,>=0.4.2 in /usr/local/lib/python3.6/dist-packages (from ctgan) (0.5.0)\n",
            "Requirement already satisfied: pandas<0.26,>=0.24 in /usr/local/lib/python3.6/dist-packages (from ctgan) (0.25.3)\n",
            "Requirement already satisfied: scikit-learn<0.23,>=0.21 in /usr/local/lib/python3.6/dist-packages (from ctgan) (0.22.2.post1)\n",
            "Requirement already satisfied: torch<2,>=1.0 in /usr/local/lib/python3.6/dist-packages (from ctgan) (1.4.0)\n",
            "Requirement already satisfied: numpy<2,>=1.17.4 in /usr/local/lib/python3.6/dist-packages (from ctgan) (1.18.2)\n",
            "Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision<1,>=0.4.2->ctgan) (7.0.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision<1,>=0.4.2->ctgan) (1.12.0)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas<0.26,>=0.24->ctgan) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas<0.26,>=0.24->ctgan) (2018.9)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn<0.23,>=0.21->ctgan) (0.14.1)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn<0.23,>=0.21->ctgan) (1.4.1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kVs4X1ZYimFm",
        "colab_type": "code",
        "outputId": "bbf2034c-2062-4e2b-d2b2-098fc3934dcf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        }
      },
      "source": [
        "from ctgan import CTGANSynthesizer\n",
        "\n",
        "#discrete_columns = [\"\"]\n",
        "ctgan = CTGANSynthesizer()\n",
        "ctgan.fit(df,epochs=10) #15"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1, Loss G: -0.1186, Loss D: 0.0010\n",
            "Epoch 2, Loss G: -0.1295, Loss D: -0.0539\n",
            "Epoch 3, Loss G: -0.1700, Loss D: -0.0651\n",
            "Epoch 4, Loss G: -0.1700, Loss D: -0.0947\n",
            "Epoch 5, Loss G: -0.2250, Loss D: -0.1316\n",
            "Epoch 6, Loss G: -0.2516, Loss D: -0.0957\n",
            "Epoch 7, Loss G: -0.2617, Loss D: -0.1258\n",
            "Epoch 8, Loss G: -0.3456, Loss D: -0.1716\n",
            "Epoch 9, Loss G: -0.3978, Loss D: -0.2395\n",
            "Epoch 10, Loss G: -0.4347, Loss D: -0.2799\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B29p4L1rqy6W",
        "colab_type": "text"
      },
      "source": [
        "Random Benchmark"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZThcacZxqJ5X",
        "colab_type": "code",
        "outputId": "f515cd17-9def-43b7-e9a8-410833c26474",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "np.random.seed(1)\n",
        "df_in = df.copy()\n",
        "df_in[\"Close_1\"] = np.random.permutation(df_in[\"Close_1\"].values)\n",
        "model(df_in)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3203.182175978968"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 89
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AojSu9JHq-mh",
        "colab_type": "text"
      },
      "source": [
        "Generated Performance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_dXJUk_xqvaa",
        "colab_type": "code",
        "outputId": "996d4fbf-7e77-4ba9-e882-ffa0510c2421",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df_gen = ctgan.sample(len(df_in)*100)\n",
        "model(df_gen)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6870.853975561564"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nCBNK-8dp6wj",
        "colab_type": "text"
      },
      "source": [
        "As expected a cross-sectional technique, does not work well on time-series data, in the future, other methods will be investigated."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uq6543YfXq3a",
        "colab_type": "text"
      },
      "source": [
        "<a name=\"example\"></a>\n",
        "\n",
        "## **(6) Skeleton Example**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ULaw7mnmb3t8",
        "colab_type": "text"
      },
      "source": [
        "Here I will perform tabular agumenting methods on a small dataset single digit features and around 250 instances. This is not necessarily the best sized dataset to highglight the performance of tabular augmentation as some method like extraction would be overkill as it would lead to dimensionality problems. It is also good to know that there are close to infinite number of ways to perform these augmentation methods. In the future, automated augmentation methods can guide the exmperiment process. \n",
        "\n",
        "The approach taken in this skeleton is to develop running models that are tested after each augmentation to highlight what methods might work well on this particular dataset. The metric we will use is mean squared error. In this implementation we do not have special hold-out sets.\n",
        "\n",
        "The above framework of implementation will be consulted, but one still have to be strategic as to when you apply what function, and you have to make sure that you are processing your data with appropriate techniques (drop null values, fill null values) at the appropriate time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6LkaRnrcxEz",
        "colab_type": "text"
      },
      "source": [
        "#### **Validation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5rPhlSqPYW1",
        "colab_type": "text"
      },
      "source": [
        "Develop Model and Define Metric"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f341WXn5R-Z9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from lightgbm import LGBMRegressor\n",
        "from sklearn.metrics import mean_squared_error\n",
        "\n",
        "def model(df_final):\n",
        "  model = LGBMRegressor()\n",
        "  test =  df_final.head(int(len(df_final)*0.4))\n",
        "  train = df_final[~df_final.isin(test)].dropna()\n",
        "  model = model.fit(train.drop([\"Close_1\"],axis=1),train[\"Close_1\"])\n",
        "  preds = model.predict(test.drop([\"Close_1\"],axis=1))\n",
        "  test =  df_final.head(int(len(df_final)*0.4))\n",
        "  train = df_final[~df_final.isin(test)].dropna()\n",
        "  model = model.fit(train.drop([\"Close_1\"],axis=1),train[\"Close_1\"])\n",
        "  val = mean_squared_error(test[\"Close_1\"],preds); \n",
        "  return val"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tX0C1EYePe-F",
        "colab_type": "text"
      },
      "source": [
        "Reload Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1oSrW9hPXwhb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = data_copy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "otnm1ubCSOzH",
        "colab_type": "code",
        "outputId": "37a9c808-0068-45ad-dce7-73bc82611bb5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "model(df)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "302.61676570345287"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wae4Zzx8cX-5",
        "colab_type": "text"
      },
      "source": [
        "**(1) (7) (i) Transformation - Decomposition - Naive**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cimy9XQgSne_",
        "colab_type": "code",
        "outputId": "f30ed027-1554-4fe1-8b5d-76fdc6275bfb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "## If Inferred Seasonality is Too Large Default to Five\n",
        "seasons = transform.infer_seasonality(df[\"Close\"],index=0) \n",
        "df_out = transform.naive_dec(df.copy(), [\"Close\",\"Open\"], freq=5)\n",
        "model(df_out) #improvement"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "274.34477082783525"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQwzuLAbcuuZ",
        "colab_type": "text"
      },
      "source": [
        "**(1) (8) (i) Transformation - Filter - Baxter-King-Bandpass**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CuxT9lAFcJ3r",
        "colab_type": "code",
        "outputId": "954dbb4d-7524-4f1b-f512-7857fb45b4c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df_out = transform.bkb(df_out, [\"Close\",\"Low\"])\n",
        "df_best = df_out.copy()\n",
        "model(df_out) #improvement"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "267.1826850968307"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5phS1vcsemX0",
        "colab_type": "text"
      },
      "source": [
        "**(1) (3) (i) Transformation - Differentiation - Fractional**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lwhn21LOeeLF",
        "colab_type": "code",
        "outputId": "4f39a4f4-374e-4c66-9981-ac76f3e92723",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df_out = transform.fast_fracdiff(df_out, [\"Close_BPF\"],0.5)\n",
        "model(df_out) #null"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "267.7083192402742"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fA4VYBu0e4rc",
        "colab_type": "text"
      },
      "source": [
        "**(1) (1) (i) Transformation - Scaling - Robust Scaler**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2-lzYN_qeBUo",
        "colab_type": "code",
        "outputId": "d6f695c4-c98a-4b29-8eef-d37693a997a3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df_out = df_out.dropna()\n",
        "df_out = transform.robust_scaler(df_out, drop=[\"Close_1\"])\n",
        "model(df_out) #noisy"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "270.96980399571214"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HA7n3-zggWuQ",
        "colab_type": "text"
      },
      "source": [
        "**(2) (2) (i) Interactions - Operator - Multiplication/Division**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBeDOCFbaIhC",
        "colab_type": "code",
        "outputId": "67527670-f6d3-42ca-cddd-9e91222e249a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "df_out.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Close_1</th>\n",
              "      <th>High</th>\n",
              "      <th>Low</th>\n",
              "      <th>Open</th>\n",
              "      <th>Close</th>\n",
              "      <th>Volume</th>\n",
              "      <th>Adj Close</th>\n",
              "      <th>Close_NDDT</th>\n",
              "      <th>Close_NDDS</th>\n",
              "      <th>Close_NDDR</th>\n",
              "      <th>Open_NDDT</th>\n",
              "      <th>Open_NDDS</th>\n",
              "      <th>Open_NDDR</th>\n",
              "      <th>Close_BPF</th>\n",
              "      <th>Low_BPF</th>\n",
              "      <th>Close_BPF_frac</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-01-08</th>\n",
              "      <td>338.529999</td>\n",
              "      <td>1.018413</td>\n",
              "      <td>0.964048</td>\n",
              "      <td>1.096600</td>\n",
              "      <td>1.001175</td>\n",
              "      <td>-0.162616</td>\n",
              "      <td>1.001175</td>\n",
              "      <td>0.832297</td>\n",
              "      <td>0.834964</td>\n",
              "      <td>1.335433</td>\n",
              "      <td>0.758743</td>\n",
              "      <td>0.691596</td>\n",
              "      <td>2.259884</td>\n",
              "      <td>-2.534142</td>\n",
              "      <td>-2.249135</td>\n",
              "      <td>-3.593612</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-09</th>\n",
              "      <td>344.970001</td>\n",
              "      <td>1.012068</td>\n",
              "      <td>1.023302</td>\n",
              "      <td>1.011466</td>\n",
              "      <td>1.042689</td>\n",
              "      <td>-0.501798</td>\n",
              "      <td>1.042689</td>\n",
              "      <td>0.908963</td>\n",
              "      <td>-0.165036</td>\n",
              "      <td>1.111346</td>\n",
              "      <td>0.835786</td>\n",
              "      <td>0.333361</td>\n",
              "      <td>1.129783</td>\n",
              "      <td>-3.081959</td>\n",
              "      <td>-2.776302</td>\n",
              "      <td>-2.523465</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-10</th>\n",
              "      <td>347.260010</td>\n",
              "      <td>1.035581</td>\n",
              "      <td>1.027563</td>\n",
              "      <td>0.996969</td>\n",
              "      <td>1.126762</td>\n",
              "      <td>-0.367576</td>\n",
              "      <td>1.126762</td>\n",
              "      <td>1.029347</td>\n",
              "      <td>2.120026</td>\n",
              "      <td>0.853697</td>\n",
              "      <td>0.907588</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.533777</td>\n",
              "      <td>-2.052768</td>\n",
              "      <td>-2.543449</td>\n",
              "      <td>-0.747382</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-11</th>\n",
              "      <td>334.399994</td>\n",
              "      <td>1.073153</td>\n",
              "      <td>1.120506</td>\n",
              "      <td>1.098313</td>\n",
              "      <td>1.156658</td>\n",
              "      <td>-0.586571</td>\n",
              "      <td>1.156658</td>\n",
              "      <td>1.109144</td>\n",
              "      <td>-5.156051</td>\n",
              "      <td>0.591990</td>\n",
              "      <td>1.002162</td>\n",
              "      <td>-0.666639</td>\n",
              "      <td>0.608516</td>\n",
              "      <td>-0.694642</td>\n",
              "      <td>-0.831670</td>\n",
              "      <td>0.414063</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-01-14</th>\n",
              "      <td>344.429993</td>\n",
              "      <td>0.999627</td>\n",
              "      <td>1.056991</td>\n",
              "      <td>1.102135</td>\n",
              "      <td>0.988773</td>\n",
              "      <td>-0.541752</td>\n",
              "      <td>0.988773</td>\n",
              "      <td>1.107633</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>-0.660350</td>\n",
              "      <td>1.056302</td>\n",
              "      <td>-0.915491</td>\n",
              "      <td>0.263025</td>\n",
              "      <td>-0.645590</td>\n",
              "      <td>-0.116166</td>\n",
              "      <td>-0.118012</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Close_1      High       Low  ...  Close_BPF   Low_BPF  Close_BPF_frac\n",
              "Date                                        ...                                     \n",
              "2019-01-08  338.529999  1.018413  0.964048  ...  -2.534142 -2.249135       -3.593612\n",
              "2019-01-09  344.970001  1.012068  1.023302  ...  -3.081959 -2.776302       -2.523465\n",
              "2019-01-10  347.260010  1.035581  1.027563  ...  -2.052768 -2.543449       -0.747382\n",
              "2019-01-11  334.399994  1.073153  1.120506  ...  -0.694642 -0.831670        0.414063\n",
              "2019-01-14  344.429993  0.999627  1.056991  ...  -0.645590 -0.116166       -0.118012\n",
              "\n",
              "[5 rows x 16 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 98
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m2BYJLaUftZ4",
        "colab_type": "code",
        "outputId": "d45a34e0-1c14-4719-9c40-e86d1af2f335",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df_out = interact.muldiv(df_out, [\"Close\",\"Open_NDDS\",\"Low_BPF\"]) \n",
        "model(df_out) #noisy"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "285.6420643864313"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bcoMw3G-Qeug",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_r = df_out.copy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXul6d1ChTPv",
        "colab_type": "text"
      },
      "source": [
        "**(2) (6) (i) Interactions - Speciality - Technical**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z533EApbRHuj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import ta\n",
        "df = interact.tech(df)\n",
        "df_out = pd.merge(df_out,  df.iloc[:,7:], left_index=True, right_index=True, how=\"left\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XWgQZx_YT_AK",
        "colab_type": "text"
      },
      "source": [
        "**Clean Dataframe and Metric**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SGbYTcLPT9VB",
        "colab_type": "code",
        "outputId": "690a5944-d2ae-4933-8f68-587ecc209252",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "\"\"\"Droping column where missing values are above a threshold\"\"\"\n",
        "df_out = df_out.dropna(thresh = len(df_out)*0.95, axis = \"columns\") \n",
        "df_out = df_out.dropna()\n",
        "df_out = df_out.replace([np.inf, -np.inf], np.nan).ffill().fillna(0)\n",
        "close = df_out[\"Close\"].copy()\n",
        "df_d = df_out.copy()\n",
        "model(df_out) #improve"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "592.52971755184"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fk5kDzp-icy0",
        "colab_type": "text"
      },
      "source": [
        "**(3) (1) (i) Mapping - Eigen Decomposition - PCA**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WsiQCerYidUM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.decomposition import PCA, IncrementalPCA, KernelPCA\n",
        "\n",
        "df_out = transform.robust_scaler(df_out, drop=[\"Close_1\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "otSYyI7k_iyo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_out = df_out.replace([np.inf, -np.inf], np.nan).ffill().fillna(0)\n",
        "df_out = mapper.pca_feature(df_out, drop_cols=[\"Close_1\"], variance_or_components=0.9, n_components=8,non_linear=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w4Offv71XEOR",
        "colab_type": "code",
        "outputId": "c1338499-3ae7-427b-b22f-cf1b87deea04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "model(df_out) #noisy but not too bad given the 10 fold dimensionality reduction"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "687.158330455884"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q2OhlHz4Dpwd",
        "colab_type": "text"
      },
      "source": [
        "**(4) Extracting**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LAcHcXiEzyly",
        "colab_type": "text"
      },
      "source": [
        "Here at first, I show the functions that have been added to the DeltaPy fork of tsfresh. You have to add your own personal adjustments based on the features you would like to construct. I am using self-developed features, but you can also use TSFresh's community functions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cF0HqJIYFbV3",
        "colab_type": "text"
      },
      "source": [
        "*The following files have been appropropriately ammended (Get in contact for advice)*\n",
        "1. https://github.com/firmai/tsfresh/blob/master/tsfresh/feature_extraction/settings.py\n",
        "1. https://github.com/firmai/tsfresh/blob/master/tsfresh/feature_extraction/feature_calculators.py\n",
        "1. https://github.com/firmai/tsfresh/blob/master/tsfresh/feature_extraction/extraction.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2FmwM0xkf1u",
        "colab_type": "text"
      },
      "source": [
        "**(4) (10) (i) Extracting - Averages - GSkew**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v_xQyORhpXRx",
        "colab_type": "code",
        "outputId": "97730dd6-48ec-4494-d72e-745af09fe320",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "extract.gskew(df_out[\"PCA_1\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "-0.7903067336449059"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xVIKcFBkxC1",
        "colab_type": "text"
      },
      "source": [
        "**(4) (21) (ii) Extracting - Entropy - SVD Entropy**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1tqmRU3eqlYV",
        "colab_type": "code",
        "outputId": "175bf079-e0a1-4958-8fc2-9a563f627899",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "svd_param = [{\"Tau\": ta, \"DE\": de}\n",
        "                      for ta in [4] \n",
        "                      for de in [3,6]]\n",
        "\n",
        "extract.svd_entropy(df_out[\"PCA_1\"],svd_param)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Tau_\"4\"__De_3\"', 0.7234823323374294),\n",
              " ('Tau_\"4\"__De_6\"', 1.3014347840145244)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vkz67m91lPoM",
        "colab_type": "text"
      },
      "source": [
        "**(4) (13) (ii) Extracting - Streaks - Wozniak**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4AyS9kvrqp0f",
        "colab_type": "code",
        "outputId": "ffc1dde1-dbd9-4a98-d82a-9863fcf2fb1c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "woz_param = [{\"consecutiveStar\": n} for n in [2, 4]]\n",
        "\n",
        "extract.wozniak(df_out[\"PCA_1\"],woz_param)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('consecutiveStar_2', 0.012658227848101266), ('consecutiveStar_4', 0.0)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ALg-gxqpln5g",
        "colab_type": "text"
      },
      "source": [
        "**(4) (28) (i) Extracting - Fractal - Higuchi**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "b3Y_1liWDWEU",
        "outputId": "a3c2fb44-d06b-494d-d919-4229878a8819",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "hig_param = [{\"Kmax\": 3},{\"Kmax\": 5}]\n",
        "\n",
        "extract.higuchi_fractal_dimension(df_out[\"PCA_1\"],hig_param)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Kmax_3', 0.577913816027104), ('Kmax_5', 0.8176960510304725)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iijWbsll7CKq",
        "colab_type": "text"
      },
      "source": [
        "**(4) (5) (ii) Extracting - Volatility - Variability Index**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "No3oCvpH7JBv",
        "colab_type": "code",
        "outputId": "9aa42f76-5c6f-48c1-f33b-6ee7ff940318",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "var_index_param = {\"Volume\":df[\"Volume\"].values, \"Open\": df[\"Open\"].values}\n",
        "\n",
        "extract.var_index(df[\"Close\"].values,var_index_param)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Interact__Open': 0.00396022538846289,\n",
              " 'Interact__Volume': 0.20550155114176533}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hc1cnA9MGZBw",
        "colab_type": "text"
      },
      "source": [
        "**Time Series Extraction**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TH5tOLc6Gew3",
        "colab_type": "code",
        "outputId": "38700e40-660b-444a-f1ff-2de90f99cf5e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 758
        }
      },
      "source": [
        "!pip install git+git://github.com/firmai/tsfresh.git"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git+git://github.com/firmai/tsfresh.git\n",
            "  Cloning git://github.com/firmai/tsfresh.git to /tmp/pip-req-build-ucpcumqp\n",
            "  Running command git clone -q git://github.com/firmai/tsfresh.git /tmp/pip-req-build-ucpcumqp\n",
            "Requirement already satisfied (use --upgrade to upgrade): tsfresh==0.15.2.dev32+g923288e from git+git://github.com/firmai/tsfresh.git in /usr/local/lib/python3.6/dist-packages\n",
            "Requirement already satisfied: requests>=2.9.1 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (2.21.0)\n",
            "Requirement already satisfied: numpy>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (1.18.2)\n",
            "Requirement already satisfied: pandas!=0.24.*,>=0.20.3 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (0.25.3)\n",
            "Requirement already satisfied: scipy>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (1.4.1)\n",
            "Requirement already satisfied: statsmodels>=0.8.0 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (0.10.2)\n",
            "Requirement already satisfied: patsy>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (0.5.1)\n",
            "Requirement already satisfied: scikit-learn>=0.19.0 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (0.22.2.post1)\n",
            "Requirement already satisfied: tqdm>=4.10.0 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (4.38.0)\n",
            "Requirement already satisfied: dask[dataframe]>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (2.12.0)\n",
            "Requirement already satisfied: distributed>=1.18.3 in /usr/local/lib/python3.6/dist-packages (from tsfresh==0.15.2.dev32+g923288e) (1.25.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.9.1->tsfresh==0.15.2.dev32+g923288e) (2.8)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.9.1->tsfresh==0.15.2.dev32+g923288e) (1.24.3)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.9.1->tsfresh==0.15.2.dev32+g923288e) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.9.1->tsfresh==0.15.2.dev32+g923288e) (2019.11.28)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas!=0.24.*,>=0.20.3->tsfresh==0.15.2.dev32+g923288e) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas!=0.24.*,>=0.20.3->tsfresh==0.15.2.dev32+g923288e) (2.8.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from patsy>=0.4.1->tsfresh==0.15.2.dev32+g923288e) (1.12.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.19.0->tsfresh==0.15.2.dev32+g923288e) (0.14.1)\n",
            "Requirement already satisfied: fsspec>=0.6.0; extra == \"dataframe\" in /usr/local/lib/python3.6/dist-packages (from dask[dataframe]>=0.15.2->tsfresh==0.15.2.dev32+g923288e) (0.7.1)\n",
            "Requirement already satisfied: partd>=0.3.10; extra == \"dataframe\" in /usr/local/lib/python3.6/dist-packages (from dask[dataframe]>=0.15.2->tsfresh==0.15.2.dev32+g923288e) (1.1.0)\n",
            "Requirement already satisfied: toolz>=0.7.3; extra == \"dataframe\" in /usr/local/lib/python3.6/dist-packages (from dask[dataframe]>=0.15.2->tsfresh==0.15.2.dev32+g923288e) (0.10.0)\n",
            "Requirement already satisfied: msgpack in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (1.0.0)\n",
            "Requirement already satisfied: cloudpickle>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (1.3.0)\n",
            "Requirement already satisfied: psutil>=5.0 in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (5.4.8)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (3.13)\n",
            "Requirement already satisfied: tornado>=4.5.1 in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (4.5.3)\n",
            "Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (2.0.0)\n",
            "Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (2.1.0)\n",
            "Requirement already satisfied: tblib in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (1.6.0)\n",
            "Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.6/dist-packages (from distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (7.1.1)\n",
            "Requirement already satisfied: locket in /usr/local/lib/python3.6/dist-packages (from partd>=0.3.10; extra == \"dataframe\"->dask[dataframe]>=0.15.2->tsfresh==0.15.2.dev32+g923288e) (0.2.0)\n",
            "Requirement already satisfied: heapdict in /usr/local/lib/python3.6/dist-packages (from zict>=0.1.3->distributed>=1.18.3->tsfresh==0.15.2.dev32+g923288e) (1.0.1)\n",
            "Building wheels for collected packages: tsfresh\n",
            "  Building wheel for tsfresh (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for tsfresh: filename=tsfresh-0.15.2.dev32+g923288e-py2.py3-none-any.whl size=82188 sha256=1e07b7b113e361faed23c5501e9c3dc05c142d12272ce159d5ef1a168d7e63fc\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-0x0m85b6/wheels/21/fe/93/890784d2b4444e938d4b85281c70cebe0b3e4736f5cfd5a07e\n",
            "Successfully built tsfresh\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kqm-2aAZGcOH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Construct the preferred input dataframe.\n",
        "from tsfresh.utilities.dataframe_functions import roll_time_series\n",
        "df_out[\"ID\"] = 0\n",
        "periods = 30\n",
        "df_out = df_out.reset_index()\n",
        "df_ts = roll_time_series(df_out,\"ID\",\"Date\",None,1,periods)\n",
        "counts = df_ts['ID'].value_counts()\n",
        "df_ts = df_ts[df_ts['ID'].isin(counts[counts > periods].index)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hglxgmkdJqV3",
        "colab_type": "code",
        "outputId": "d55056bf-13a6-4308-f2ba-82b9fa36b8d8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "#Perform extraction\n",
        "from tsfresh.feature_extraction import extract_features, CustomFCParameters\n",
        "settings_dict = CustomFCParameters()\n",
        "settings_dict[\"var_index\"] = {\"PCA_1\":None, \"PCA_2\": None}\n",
        "df_feat = extract_features(df_ts.drop([\"Close_1\"],axis=1),default_fc_parameters=settings_dict,column_id=\"ID\",column_sort=\"Date\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Feature Extraction: 100%|██████████| 5/5 [00:11<00:00,  2.31s/it]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nz6kY2yjwnVX",
        "colab_type": "code",
        "outputId": "80710807-67c2-4c32-cca3-4879cab46b84",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "# Cleaning operations\n",
        "import pandasvault as pv\n",
        "df_feat2 = df_feat.copy()\n",
        "df_feat = df_feat.dropna(thresh = len(df_feat)*0.50, axis = \"columns\")\n",
        "df_feat_cons = pv.constant_feature_detect(data=df_feat,threshold=0.9)\n",
        "df_feat = df_feat.drop(df_feat_cons, axis=1)\n",
        "df_feat = df_feat.ffill()\n",
        "df_feat = pd.merge(df_feat,df[[\"Close_1\"]],left_index=True,right_index=True,how=\"left\")\n",
        "print(df_feat.shape)\n",
        "model(df_feat) #noisy"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7  variables are found to be almost constant\n",
            "(208, 48)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2064.7813982935995"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vESUkUrzF5yr",
        "colab_type": "code",
        "outputId": "d14f5c8a-8285-4336-c10f-1c762c5d7fd1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "from tsfresh import select_features\n",
        "from tsfresh.utilities.dataframe_functions import impute\n",
        "\n",
        "impute(df_feat)\n",
        "df_feat_2 = select_features(df_feat.drop([\"Close_1\"],axis=1),df_feat[\"Close_1\"],fdr_level=0.05)\n",
        "df_feat_2[\"Close_1\"] = df_feat[\"Close_1\"]\n",
        "model(df_feat_2) #improvement (b/ not an augmentation method)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1577.5273071299482"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ir07IITAYmhS",
        "colab_type": "text"
      },
      "source": [
        "**(3) (6) (i) Feature Agglomoration; &nbsp; (1)(2)(i) Standard Scaler.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mb-IVKwtc12S",
        "colab_type": "text"
      },
      "source": [
        "Like in this step, after (1), (2), (3), (4) and (5), you can often circle back to the initial steps to normalise the data and dimensionally reduce the data for the final model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z9mG2Ln5yeGi",
        "colab_type": "code",
        "outputId": "6470f144-664c-4e76-db8f-bc5d02e9ecf6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "import numpy as np\n",
        "from sklearn import datasets, cluster\n",
        "\n",
        "def feature_agg(df, drop, components):\n",
        "  components = min(df.shape[1]-1,components)\n",
        "  agglo = cluster.FeatureAgglomeration(n_clusters=components,)\n",
        "  df = df.drop(drop,axis=1)\n",
        "  agglo.fit(df)\n",
        "  df = pd.DataFrame(agglo.transform(df))\n",
        "  df = df.add_prefix('fe_agg_')\n",
        "\n",
        "  return df\n",
        "\n",
        "df_final = transform.standard_scaler(df_feat_2, drop=[\"Close_1\"])\n",
        "df_final = mapper.feature_agg(df_final,[\"Close_1\"],4)\n",
        "df_final.index = df_feat.index\n",
        "df_final[\"Close_1\"] = df_feat[\"Close_1\"]\n",
        "model(df_final) #noisy"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1949.89085894338"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h65TcvKvY2XF",
        "colab_type": "text"
      },
      "source": [
        "**Final Model** After Applying 13 Arbitary Augmentation Techniques"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7aNKVT-FAluU",
        "colab_type": "code",
        "outputId": "1781d985-6439-4c76-d404-52c9f366ebb3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "model(df_final) #improvement"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1949.89085894338"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 117
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kF50vRO8Y8Rs",
        "colab_type": "text"
      },
      "source": [
        "**Original Model** Before Augmentation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WrU7hfqO95rp",
        "colab_type": "code",
        "outputId": "2e7b7562-000d-4d38-b44e-9c8eace39d8e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df_org = df.iloc[:,:7][df.index.isin(df_final.index)]\n",
        "model(df_org)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "389.783990984133"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R4l80vKYZFe8",
        "colab_type": "text"
      },
      "source": [
        "**Best Model** After Developing 8 Augmenting Features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "glPEl-gQZUdY",
        "colab_type": "code",
        "outputId": "a9f338ec-edc5-4d3e-d848-9da2a8c4b641",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "df_best = df_best.replace([np.inf, -np.inf], np.nan).ffill().fillna(0)\n",
        "model(df_best)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "267.1826850968307"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vKPJ20UrcScj",
        "colab_type": "text"
      },
      "source": [
        "**Commentary**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U0ZPaKNhcUyD",
        "colab_type": "text"
      },
      "source": [
        "There are countless ways in which the current model can be improved, this can take on an automated process where all techniques are tested against a hold out set, for example, we can perform the operation below, and even though it improves the score here, there is a need for more robust tests. The skeleton example above is not meant to highglight the performance of the package. It simply serves as an example of how one can go about applying augmentation methods. \n",
        "\n",
        "Quite naturally this example suffers from dimensionality issues with array shapes reaching ```(208, 48)```, furthermore you would need a sample that is at least 50-100 times larger before machine learning methods start to make sense.\n",
        "\n",
        "Nonetheless, in this example, *Transformation, Interactions* and *Mappings* (applied to exraction output) performed fairly well. *Extraction* augmentation was overkill, but created a reasonable model when dimensionally reduced. A better selection of one of the 50+ augmentation methods and the order of augmentation could further help improve the outcome if robustly tested against development sets."
      ]
    }
  ]
}