18_convolutional_neural_nets/05_cnn_for_trading_feature_engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNN for Trading - Part 1: Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To exploit the grid-like structure of time-series data, we can use CNN architectures for univariate and multivariate time series. In the latter case, we consider different time series as channels, similar to the different color signals.\n",
    "\n",
    "An alternative approach converts a time series of alpha factors into a two-dimensional format to leverage the ability of CNNs to detect local patterns. [Sezer and Ozbayoglu (2018)](https://www.researchgate.net/publication/324802031_Algorithmic_Financial_Trading_with_Deep_Convolutional_Neural_Networks_Time_Series_to_Image_Conversion_Approach) propose CNN-TA, which computes 15 technical indicators for different intervals and uses hierarchical clustering (see Chapter 13, Data-Driven Risk Factors and Asset Allocation with Unsupervised Learning) to locate indicators that behave similarly close to each other in a two-dimensional grid.\n",
    "\n",
    "The authors train a CNN similar to the CIFAR-10 example we used earlier to predict whether to buy, hold, or sell an asset on a given day. They compare the CNN performance to \"buy-and-hold\" and other models and find that it outperforms all alternatives using daily price series for Dow 30 stocks and the nine most-traded ETFs over the 2007-2017 time period.\n",
    "\n",
    "The section on *CNN for Trading* consists of three notebooks that experiment with this approach using daily US equity price data. They demonstrate \n",
    "1. How to compute relevant financial features\n",
    "2. How to convert a similar set of indicators into image format and cluster them by similarity\n",
    "3. How to train a CNN to predict daily returns and evaluate a simple long-short strategy based on the resulting signals."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating technical indicators at different intervals\n",
    "We first select a universe of the 500 most-traded US stocks from the Quandl Wiki dataset by dollar volume for rolling five-year periods for 2007-2017. \n",
    "\n",
    "- Our features consist of 15 technical indicators and risk factors that we compute for 15 different intervals and then arrange them in a 15x15 grid.\n",
    "- For each indicator, we vary the time period from 6 to 20 to obtain 15 distinct measurements. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports & Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To install `talib` with Python 3.7 follow [these](https://medium.com/@joelzhang/install-ta-lib-in-python-3-7-51219acacafb) instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:48.739909Z",
     "start_time": "2021-02-23T16:34:48.738054Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.433046Z",
     "start_time": "2021-02-23T16:34:48.744360Z"
    }
   },
   "outputs": [],
   "source": [
    "from talib import (RSI, BBANDS, MACD,\n",
    "                   NATR, WILLR, WMA,\n",
    "                   EMA, SMA, CCI, CMO,\n",
    "                   MACD, PPO, ROC,\n",
    "                   ADOSC, ADX, MOM)\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from statsmodels.regression.rolling import RollingOLS\n",
    "import statsmodels.api as sm\n",
    "import pandas_datareader.data as web\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.435390Z",
     "start_time": "2021-02-23T16:34:49.433913Z"
    }
   },
   "outputs": [],
   "source": [
    "DATA_STORE = '../data/assets.h5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.442890Z",
     "start_time": "2021-02-23T16:34:49.436319Z"
    }
   },
   "outputs": [],
   "source": [
    "MONTH = 21\n",
    "YEAR = 12 * MONTH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.450476Z",
     "start_time": "2021-02-23T16:34:49.443903Z"
    }
   },
   "outputs": [],
   "source": [
    "START = '2000-01-01'\n",
    "END = '2017-12-31'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.463187Z",
     "start_time": "2021-02-23T16:34:49.451269Z"
    }
   },
   "outputs": [],
   "source": [
    "sns.set_style('whitegrid')\n",
    "idx = pd.IndexSlice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.470587Z",
     "start_time": "2021-02-23T16:34:49.463928Z"
    }
   },
   "outputs": [],
   "source": [
    "T = [1, 5, 10, 21, 42, 63]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.478180Z",
     "start_time": "2021-02-23T16:34:49.471318Z"
    }
   },
   "outputs": [],
   "source": [
    "results_path = Path('results', 'cnn_for_trading')\n",
    "if not results_path.exists():\n",
    "    results_path.mkdir(parents=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Quandl Wiki Stock Prices & Meta Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:49.485541Z",
     "start_time": "2021-02-23T16:34:49.479542Z"
    }
   },
   "outputs": [],
   "source": [
    "adj_ohlcv = ['adj_open', 'adj_close', 'adj_low', 'adj_high', 'adj_volume']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:57.775897Z",
     "start_time": "2021-02-23T16:34:49.486697Z"
    }
   },
   "outputs": [],
   "source": [
    "with pd.HDFStore(DATA_STORE) as store:\n",
    "    prices = (store['quandl/wiki/prices']\n",
    "              .loc[idx[START:END, :], adj_ohlcv]\n",
    "              .rename(columns=lambda x: x.replace('adj_', ''))\n",
    "              .swaplevel()\n",
    "              .sort_index()\n",
    "             .dropna())\n",
    "    metadata = (store['us_equities/stocks'].loc[:, ['marketcap', 'sector']])\n",
    "ohlcv = prices.columns.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:34:57.847372Z",
     "start_time": "2021-02-23T16:34:57.776838Z"
    }
   },
   "outputs": [],
   "source": [
    "prices.volume /= 1e3\n",
    "prices.index.names = ['symbol', 'date']\n",
    "metadata.index.name = 'symbol'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rolling universe: pick 500 most-traded stocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:35:00.884579Z",
     "start_time": "2021-02-23T16:34:57.848254Z"
    }
   },
   "outputs": [],
   "source": [
    "dollar_vol = prices.close.mul(prices.volume).unstack('symbol').sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:35:01.061217Z",
     "start_time": "2021-02-23T16:35:00.885466Z"
    }
   },
   "outputs": [],
   "source": [
    "years = sorted(np.unique([d.year for d in prices.index.get_level_values('date').unique()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:35:01.063974Z",
     "start_time": "2021-02-23T16:35:01.062285Z"
    }
   },
   "outputs": [],
   "source": [
    "train_window = 5 # years\n",
    "universe_size = 500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:41:52.520056Z",
     "start_time": "2021-02-23T16:35:01.064917Z"
    }
   },
   "outputs": [],
   "source": [
    "universe = []\n",
    "for i, year in enumerate(years[5:], 5):\n",
    "    start = str(years[i-5])\n",
    "    end = str(years[i])\n",
    "    most_traded = (dollar_vol.loc[start:end, :]\n",
    "                   .dropna(thresh=1000, axis=1)\n",
    "                   .median()\n",
    "                   .nlargest(universe_size)\n",
    "                   .index)\n",
    "    universe.append(prices.loc[idx[most_traded, start:end], :])\n",
    "universe = pd.concat(universe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:41:52.882364Z",
     "start_time": "2021-02-23T16:41:52.520947Z"
    }
   },
   "outputs": [],
   "source": [
    "universe = universe.loc[~universe.index.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:41:52.930410Z",
     "start_time": "2021-02-23T16:41:52.883262Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "MultiIndex: 2530228 entries, ('BRK_A', Timestamp('2000-01-03 00:00:00')) to ('BLL', Timestamp('2017-12-29 00:00:00'))\n",
      "Data columns (total 5 columns):\n",
      " #   Column  Non-Null Count    Dtype  \n",
      "---  ------  --------------    -----  \n",
      " 0   open    2530228 non-null  float64\n",
      " 1   close   2530228 non-null  float64\n",
      " 2   low     2530228 non-null  float64\n",
      " 3   high    2530228 non-null  float64\n",
      " 4   volume  2530228 non-null  float64\n",
      "dtypes: float64(5)\n",
      "memory usage: 106.4+ MB\n"
     ]
    }
   ],
   "source": [
    "universe.info(null_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:41:52.985780Z",
     "start_time": "2021-02-23T16:41:52.931259Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     735.000000\n",
       "mean     3442.487075\n",
       "std      1145.365643\n",
       "min      1043.000000\n",
       "25%      2368.000000\n",
       "50%      3792.000000\n",
       "75%      4527.000000\n",
       "max      4528.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "universe.groupby('symbol').size().describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:41:53.060861Z",
     "start_time": "2021-02-23T16:41:52.986614Z"
    }
   },
   "outputs": [],
   "source": [
    "universe.to_hdf('data.h5', 'universe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Technical Indicators Factors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:41:53.063455Z",
     "start_time": "2021-02-23T16:41:53.061684Z"
    }
   },
   "outputs": [],
   "source": [
    "T = list(range(6, 21))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relative Strength Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:44:03.645054Z",
     "start_time": "2021-02-23T16:41:53.064605Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_RSI'] = universe.groupby(level='symbol').close.apply(RSI, timeperiod=t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Williams %R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:44:51.356647Z",
     "start_time": "2021-02-23T16:44:03.645883Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_WILLR'] = (universe.groupby(level='symbol', group_keys=False)\n",
    "     .apply(lambda x: WILLR(x.high, x.low, x.close, timeperiod=t)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute Bollinger Bands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:44:51.359531Z",
     "start_time": "2021-02-23T16:44:51.357628Z"
    }
   },
   "outputs": [],
   "source": [
    "def compute_bb(close, timeperiod):\n",
    "    high, mid, low = BBANDS(close, timeperiod=timeperiod)\n",
    "    return pd.DataFrame({f'{timeperiod:02}_BBH': high, f'{timeperiod:02}_BBL': low}, index=close.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:47:38.075231Z",
     "start_time": "2021-02-23T16:44:51.361691Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    bbh, bbl = f'{t:02}_BBH', f'{t:02}_BBL'\n",
    "    universe = (universe.join(\n",
    "        universe.groupby(level='symbol').close.apply(compute_bb,\n",
    "                                                     timeperiod=t)))\n",
    "    universe[bbh] = universe[bbh].sub(universe.close).div(universe[bbh]).apply(np.log1p)\n",
    "    universe[bbl] = universe.close.sub(universe[bbl]).div(universe.close).apply(np.log1p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalized Average True Range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:48:36.081047Z",
     "start_time": "2021-02-23T16:47:38.076757Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_NATR'] = universe.groupby(level='symbol', \n",
    "                                group_keys=False).apply(lambda x: \n",
    "                                                        NATR(x.high, x.low, x.close, timeperiod=t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Percentage Price Oscillator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:51:07.164873Z",
     "start_time": "2021-02-23T16:48:36.081894Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_PPO'] = universe.groupby(level='symbol').close.apply(PPO, fastperiod=t, matype=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Moving Average Convergence/Divergence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:51:07.168458Z",
     "start_time": "2021-02-23T16:51:07.166104Z"
    }
   },
   "outputs": [],
   "source": [
    "def compute_macd(close, signalperiod):\n",
    "    macd = MACD(close, signalperiod=signalperiod)[0]\n",
    "    return (macd - np.mean(macd))/np.std(macd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:53:45.147391Z",
     "start_time": "2021-02-23T16:51:07.169518Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_MACD'] = (universe\n",
    "                  .groupby('symbol', group_keys=False)\n",
    "                  .close\n",
    "                  .apply(compute_macd, signalperiod=t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Momentum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:56:17.798498Z",
     "start_time": "2021-02-23T16:53:45.148305Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_MOM'] = universe.groupby(level='symbol').close.apply(MOM, timeperiod=t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weighted Moving Average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T16:58:49.984209Z",
     "start_time": "2021-02-23T16:56:17.799393Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_WMA'] = universe.groupby(level='symbol').close.apply(WMA, timeperiod=t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exponential Moving Average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:01:21.951204Z",
     "start_time": "2021-02-23T16:58:49.985306Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_EMA'] = universe.groupby(level='symbol').close.apply(EMA, timeperiod=t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Commodity Channel Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:02:43.373600Z",
     "start_time": "2021-02-23T17:01:21.952111Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:    \n",
    "    universe[f'{t:02}_CCI'] = (universe.groupby(level='symbol', group_keys=False)\n",
    "     .apply(lambda x: CCI(x.high, x.low, x.close, timeperiod=t)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chande Momentum Oscillator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:05:18.290515Z",
     "start_time": "2021-02-23T17:02:43.374726Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_CMO'] = universe.groupby(level='symbol').close.apply(CMO, timeperiod=t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-20T02:28:43.696896Z",
     "start_time": "2020-03-20T02:28:43.694227Z"
    }
   },
   "source": [
    "### Rate of Change"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rate of change is a technical indicator that illustrates the speed of price change over a period of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:07:35.840629Z",
     "start_time": "2021-02-23T17:05:18.291413Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_ROC'] = universe.groupby(level='symbol').close.apply(ROC, timeperiod=t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chaikin A/D Oscillator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:08:58.366782Z",
     "start_time": "2021-02-23T17:07:35.841517Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_ADOSC'] = (universe.groupby(level='symbol', group_keys=False)\n",
    "     .apply(lambda x: ADOSC(x.high, x.low, x.close, x.volume, fastperiod=t-3, slowperiod=4+t)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Average Directional Movement Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:10:24.337030Z",
     "start_time": "2021-02-23T17:08:58.367644Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in T:\n",
    "    universe[f'{t:02}_ADX'] = universe.groupby(level='symbol', \n",
    "                                group_keys=False).apply(lambda x: \n",
    "                                                        ADX(x.high, x.low, x.close, timeperiod=t))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:10:29.716720Z",
     "start_time": "2021-02-23T17:10:24.337907Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "universe.drop(ohlcv, axis=1).to_hdf('data.h5', 'features')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute Historical Returns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Historical Returns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:10:30.028914Z",
     "start_time": "2021-02-23T17:10:29.717589Z"
    }
   },
   "outputs": [],
   "source": [
    "by_sym = universe.groupby(level='symbol').close\n",
    "for t in [1,5]:\n",
    "    universe[f'r{t:02}'] = by_sym.pct_change(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:10:31.555073Z",
     "start_time": "2021-02-23T17:10:30.029775Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>r01</th>\n",
       "      <th>r05</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2.529493e+06</td>\n",
       "      <td>2.526553e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>6.710840e-04</td>\n",
       "      <td>3.293540e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.875355e-02</td>\n",
       "      <td>6.344951e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-9.718670e-01</td>\n",
       "      <td>-9.795396e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-1.034141e-02</td>\n",
       "      <td>-2.246575e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.236246e-04</td>\n",
       "      <td>2.921130e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.122661e-02</td>\n",
       "      <td>2.811951e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.216425e+01</td>\n",
       "      <td>1.252657e+01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                r01           r05\n",
       "count  2.529493e+06  2.526553e+06\n",
       "mean   6.710840e-04  3.293540e-03\n",
       "std    2.875355e-02  6.344951e-02\n",
       "min   -9.718670e-01 -9.795396e-01\n",
       "25%   -1.034141e-02 -2.246575e-02\n",
       "50%    3.236246e-04  2.921130e-03\n",
       "75%    1.122661e-02  2.811951e-02\n",
       "max    1.216425e+01  1.252657e+01"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "universe[[f'r{t:02}' for t in [1, 5]]].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:10:31.561487Z",
     "start_time": "2021-02-23T17:10:31.555915Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outliers = universe[universe.r01>1].index.get_level_values('symbol').unique()\n",
    "len(outliers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:10:39.622170Z",
     "start_time": "2021-02-23T17:10:31.562450Z"
    }
   },
   "outputs": [],
   "source": [
    "universe = universe.drop(outliers, level='symbol')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Historical return quantiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:11:04.012013Z",
     "start_time": "2021-02-23T17:10:39.623047Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in [1, 5]:\n",
    "    universe[f'r{t:02}dec'] = (universe[f'r{t:02}'].groupby(level='date')\n",
    "             .apply(lambda x: pd.qcut(x, q=10, labels=False, duplicates='drop')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rolling Factor Betas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also use five Fama-French risk factors (Fama and French, 2015; see Chapter 4, Financial Feature Engineering – How to Research Alpha Factors). They reflect the sensitivity of a stock's returns to factors consistently demonstrated to impact equity returns. \n",
    "\n",
    "We capture these factors by computing the coefficients of a rolling OLS regression of a stock's daily returns on the returns of portfolios designed to reflect the underlying drivers:\n",
    "- **Equity risk premium**: Value-weighted returns of US stocks minus the 1-month US\n",
    "- **Treasury bill rate**\n",
    "- **Size (SMB)**: Returns of stocks categorized as Small (by market cap) Minus those of Big equities\n",
    "- **Value (HML)**: Returns of stocks with High book-to-market value Minus those with a Low value\n",
    "- **Investment (CMA)**: Returns differences for companies with Conservative investment expenditures Minus those with Aggressive spending\n",
    "- **Profitability (RMW)**: Similarly, return differences for stocks with Robust profitability Minus that with a Weak metric.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:11:05.424170Z",
     "start_time": "2021-02-23T17:11:04.012880Z"
    }
   },
   "outputs": [],
   "source": [
    "factor_data = (web.DataReader('F-F_Research_Data_5_Factors_2x3_daily', 'famafrench', \n",
    "                              start=START)[0].rename(columns={'Mkt-RF': 'Market'}))\n",
    "factor_data.index.names = ['date']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:11:05.432840Z",
     "start_time": "2021-02-23T17:11:05.425132Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 5284 entries, 2000-01-03 to 2020-12-31\n",
      "Data columns (total 6 columns):\n",
      " #   Column  Non-Null Count  Dtype  \n",
      "---  ------  --------------  -----  \n",
      " 0   Market  5284 non-null   float64\n",
      " 1   SMB     5284 non-null   float64\n",
      " 2   HML     5284 non-null   float64\n",
      " 3   RMW     5284 non-null   float64\n",
      " 4   CMA     5284 non-null   float64\n",
      " 5   RF      5284 non-null   float64\n",
      "dtypes: float64(6)\n",
      "memory usage: 289.0 KB\n"
     ]
    }
   ],
   "source": [
    "factor_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:11:05.453552Z",
     "start_time": "2021-02-23T17:11:05.433756Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "windows = list(range(15, 90, 5))\n",
    "len(windows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we apply `statsmodels`' `RollingOLS()` to run regressions over windowed periods of different lengths, ranging from 15 to 90 days. We set the `params_only` parameter on the `.fit()` method to speed up computation and capture the coefficients using the `.params` attribute of the fitted factor_model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:29:30.598816Z",
     "start_time": "2021-02-23T17:11:05.456787Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15\n",
      "20\n",
      "25\n",
      "30\n",
      "35\n",
      "40\n",
      "45\n",
      "50\n",
      "55\n",
      "60\n",
      "65\n",
      "70\n",
      "75\n",
      "80\n",
      "85\n"
     ]
    }
   ],
   "source": [
    "t = 1\n",
    "ret = f'r{t:02}'\n",
    "factors = ['Market', 'SMB', 'HML', 'RMW', 'CMA']\n",
    "windows = list(range(15, 90, 5))\n",
    "for window in windows:\n",
    "    print(window)\n",
    "    betas = []\n",
    "    for symbol, data in universe.groupby(level='symbol'):\n",
    "        model_data = data[[ret]].merge(factor_data, on='date').dropna()\n",
    "        model_data[ret] -= model_data.RF\n",
    "\n",
    "        rolling_ols = RollingOLS(endog=model_data[ret], \n",
    "                                 exog=sm.add_constant(model_data[factors]), window=window)\n",
    "        factor_model = rolling_ols.fit(params_only=True).params.drop('const', axis=1)\n",
    "        result = factor_model.assign(symbol=symbol).set_index('symbol', append=True)\n",
    "        betas.append(result)\n",
    "    betas = pd.concat(betas).rename(columns=lambda x: f'{window:02}_{x}')\n",
    "    universe = universe.join(betas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute Forward Returns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:29:31.294116Z",
     "start_time": "2021-02-23T17:29:30.599863Z"
    }
   },
   "outputs": [],
   "source": [
    "for t in [1, 5]:\n",
    "    universe[f'r{t:02}_fwd'] = universe.groupby(level='symbol')[f'r{t:02}'].shift(-t)\n",
    "    universe[f'r{t:02}dec_fwd'] = universe.groupby(level='symbol')[f'r{t:02}dec'].shift(-t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Store Model Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:29:36.009200Z",
     "start_time": "2021-02-23T17:29:31.295092Z"
    }
   },
   "outputs": [],
   "source": [
    "universe = universe.drop(ohlcv, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:29:36.058817Z",
     "start_time": "2021-02-23T17:29:36.010258Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "MultiIndex: 2499265 entries, ('BRK_A', Timestamp('2000-01-03 00:00:00')) to ('BLL', Timestamp('2017-12-29 00:00:00'))\n",
      "Columns: 308 entries, 06_RSI to r05dec_fwd\n",
      "dtypes: float64(308)\n",
      "memory usage: 5.7+ GB\n"
     ]
    }
   ],
   "source": [
    "universe.info(null_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:29:36.062384Z",
     "start_time": "2021-02-23T17:29:36.060100Z"
    }
   },
   "outputs": [],
   "source": [
    "drop_cols = ['r01', 'r01dec', 'r05',  'r05dec']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:29:36.088119Z",
     "start_time": "2021-02-23T17:29:36.063595Z"
    }
   },
   "outputs": [],
   "source": [
    "outcomes = universe.filter(like='_fwd').columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-23T17:29:54.934584Z",
     "start_time": "2021-02-23T17:29:36.089191Z"
    }
   },
   "outputs": [],
   "source": [
    "universe = universe.sort_index()\n",
    "with pd.HDFStore('data.h5') as store:\n",
    "    store.put('features', universe.drop(drop_cols, axis=1).drop(outcomes, axis=1).loc[idx[:, '2001':], :])\n",
    "    store.put('targets', universe.loc[idx[:, '2001':], outcomes])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "341.391px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}