06_Performance_Tips.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SimFin Tutorial 06 - Performance Tips\n",
    "\n",
    "[Original repository on GitHub](https://github.com/simfin/simfin-tutorials)\n",
    "\n",
    "This tutorial was originally written by [Hvass Labs](https://github.com/Hvass-Labs)\n",
    "\n",
    "----\n",
    "\n",
    "\"Are you employed, Sir? You don't go out looking for a job dressed like that on a week-day, do you? Is this a ... what day is this?\" &ndash; [The Big Lebowski](https://www.youtube.com/watch?v=xJjCnWm5cvE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This is a collection of tips on how to improve performance when using the simfin package. It is assumed you are already familiar with the previous tutorials on the basics of simfin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "\n",
    "# Import the main functionality from the SimFin Python API.\n",
    "import simfin as sf\n",
    "\n",
    "# Import names used for easy access to SimFin's data-columns.\n",
    "from simfin.names import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.3.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# SimFin Python API version.\n",
    "sf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.25.1'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pandas version.\n",
    "pd.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SimFin data-directory.\n",
    "sf.set_data_dir('~/simfin_data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SimFin load API key or use free data.\n",
    "sf.load_api_key(path='~/simfin_api_key.txt', default_key='free')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Datasets\n",
    "\n",
    "In these examples, we will use the following datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset \"us-shareprices-daily\" on disk (7 days old).\n",
      "- Loading from disk ... Done!\n",
      "CPU times: user 14.9 s, sys: 1.2 s, total: 16.1 s\n",
      "Wall time: 14.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Data for USA.\n",
    "market = 'us'\n",
    "\n",
    "# Daily Share-Prices.\n",
    "df_prices = sf.load_shareprices(variant='daily', market=market)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>SimFinId</th>\n",
       "      <th>Open</th>\n",
       "      <th>Low</th>\n",
       "      <th>High</th>\n",
       "      <th>Close</th>\n",
       "      <th>Adj. Close</th>\n",
       "      <th>Dividend</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ticker</th>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td rowspan=\"5\" valign=\"top\">A</td>\n",
       "      <td>2007-01-03</td>\n",
       "      <td>45846</td>\n",
       "      <td>34.99</td>\n",
       "      <td>34.05</td>\n",
       "      <td>35.48</td>\n",
       "      <td>34.30</td>\n",
       "      <td>22.95</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2574600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2007-01-04</td>\n",
       "      <td>45846</td>\n",
       "      <td>34.30</td>\n",
       "      <td>33.46</td>\n",
       "      <td>34.60</td>\n",
       "      <td>34.41</td>\n",
       "      <td>23.03</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2073700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2007-01-05</td>\n",
       "      <td>45846</td>\n",
       "      <td>34.30</td>\n",
       "      <td>34.00</td>\n",
       "      <td>34.40</td>\n",
       "      <td>34.09</td>\n",
       "      <td>22.81</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2676600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2007-01-08</td>\n",
       "      <td>45846</td>\n",
       "      <td>33.98</td>\n",
       "      <td>33.68</td>\n",
       "      <td>34.08</td>\n",
       "      <td>33.97</td>\n",
       "      <td>22.73</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1557200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2007-01-09</td>\n",
       "      <td>45846</td>\n",
       "      <td>34.08</td>\n",
       "      <td>33.63</td>\n",
       "      <td>34.32</td>\n",
       "      <td>34.01</td>\n",
       "      <td>22.76</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1386200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   SimFinId   Open    Low   High  Close  Adj. Close  Dividend  \\\n",
       "Ticker Date                                                                     \n",
       "A      2007-01-03     45846  34.99  34.05  35.48  34.30       22.95       NaN   \n",
       "       2007-01-04     45846  34.30  33.46  34.60  34.41       23.03       NaN   \n",
       "       2007-01-05     45846  34.30  34.00  34.40  34.09       22.81       NaN   \n",
       "       2007-01-08     45846  33.98  33.68  34.08  33.97       22.73       NaN   \n",
       "       2007-01-09     45846  34.08  33.63  34.32  34.01       22.76       NaN   \n",
       "\n",
       "                    Volume  \n",
       "Ticker Date                 \n",
       "A      2007-01-03  2574600  \n",
       "       2007-01-04  2073700  \n",
       "       2007-01-05  2676600  \n",
       "       2007-01-08  1557200  \n",
       "       2007-01-09  1386200  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_prices.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Columns To DataFrame\n",
    "\n",
    "For some reason, it is extremely slow to add new columns to an empty Pandas DataFrame that does not have an index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Names of the new columns.\n",
    "FOO = 'Foo'\n",
    "BAR = 'Bar'\n",
    "QUX = 'Qux'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an empty DataFrame without setting its index. This is extremely slow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 22.4 s, sys: 711 ms, total: 23.1 s\n",
      "Wall time: 19.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df2 = pd.DataFrame()\n",
    "df2[FOO] = df_prices[CLOSE] / df_prices[ADJ_CLOSE]\n",
    "df2[BAR] = df_prices[CLOSE] * df_prices[ADJ_CLOSE]\n",
    "df2[QUX] = df_prices[CLOSE] * df_prices[VOLUME]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we know what the index should be and we set it when creating the new DataFrame, it is much faster to add new columns to the DataFrame. This is the preferred method because it is fast and elegant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61 ms ± 122 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "df3 = pd.DataFrame(index=df_prices.index)\n",
    "df3[FOO] = df_prices[CLOSE] / df_prices[ADJ_CLOSE]\n",
    "df3[BAR] = df_prices[CLOSE] * df_prices[ADJ_CLOSE]\n",
    "df3[QUX] = df_prices[CLOSE] * df_prices[VOLUME]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `df.values` to do the computation directly using numpy arrays, but it is not faster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/magnus/anaconda3/envs/simfin/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: divide by zero encountered in true_divide\n",
      "  \n",
      "/home/magnus/anaconda3/envs/simfin/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in true_divide\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60.7 ms ± 770 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "df4 = pd.DataFrame(index=df_prices.index)\n",
    "df4[FOO] = df_prices[CLOSE].values / df_prices[ADJ_CLOSE].values\n",
    "df4[BAR] = df_prices[CLOSE].values * df_prices[ADJ_CLOSE].values\n",
    "df4[QUX] = df_prices[CLOSE].values * df_prices[VOLUME].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also construct the new DataFrame from several Pandas Series, which is usually about the same speed as the `df3` solution above, but not as elegant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61.3 ms ± 2.65 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "df_foo = df_prices[CLOSE] / df_prices[ADJ_CLOSE]\n",
    "df_bar = df_prices[CLOSE] * df_prices[ADJ_CLOSE]\n",
    "df_qux = df_prices[CLOSE] * df_prices[VOLUME]\n",
    "data = {FOO: df_foo, BAR: df_bar, QUX: df_qux}\n",
    "df5 = pd.DataFrame(data=data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disk Cache\n",
    "\n",
    "Some functions take a long time to process data, such as the signal-functions in the simfin package. If you want to rerun a Notebook with such functions, then you would have to rerun all these slow functions again, even though the results would be exactly the same, if the data has not changed.\n",
    "\n",
    "A simple solution is to cache the results of slow functions, by writing the results to a cache-file on disk. The next time the function is called, it automatically checks if a recent cache-file exists on disk and then loads it, otherwise the slow function will be computed and the results saved in the cache-file for future use.\n",
    "\n",
    "This is implemented by using a so-called decorator or wrapper-function ` @sf.cache` on the slow function. This is used in simfin's signal-functions, and you can also use this wrapper on your own functions (see below).\n",
    "\n",
    "A few things should be noted:\n",
    "\n",
    "1. The wrapper adds three more arguments to the original function: `cache_name` which allows you to distinguish cache-files from each other. `cache_refresh` which determines if the slow function should be called and the results saved to the cache-file on disk. `cache_format` which is the format of the cache-file. See below for details.\n",
    "\n",
    "2. Because of these new arguments, you **MUST** use keyword arguments when calling the wrapped function, otherwise the arguments will get passed to the cache-wrapper instead of the original function. This will raise a strange exception."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cache Refresh Conditions\n",
    "\n",
    "There are several ways of specifying the conditions for when the slow function must be called again and the results saved to the cache-file on disk. These conditions are specified by passing different `cache_refresh` arguments to the wrapped function:\n",
    "\n",
    "- If `cache_refresh=None` then the cache-file is never used and the wrapped function is always called as normal.\n",
    "\n",
    "- If `cache_refresh=True` then the wrapped function is called and the results are saved to the cache-file on disk.\n",
    "\n",
    "- If `cache_refresh=False` then the cache-file is always used, unless it does not exist, in which case the wrapped function is called and the cache-file saved.\n",
    "\n",
    "- If `cache_refresh` is an integer which is lower than the cache-file's age in days, then the wrapped function is called and the results are saved to the cache-file. The cache is also refreshed if the integer is 0 (zero).\n",
    "\n",
    "- If `cache_refresh` is a string or list of strings, these are considered to be file-paths e.g. for dataset-files. If the cache-file is older than any one of those files, then the wrapped function is called and the results are saved to the cache-file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cache Format\n",
    "\n",
    "By default `cache_format='pickle'` so the cache-files are saved as an uncompressed pickle-file, which is very fast to save and load, but also takes a lot of disk-space. The default `'pickle'` file-format should support all Pandas DataFrames and Series and properly save all meta-data such as which columns are used as indices, etc.\n",
    "\n",
    "You may compress the pickle-files using `cache_format='pickle.gz'` which can compress DataFrames with much repetitive data (e.g. forward-filled daily signals) by a factor of 100 or more, but this requires a little more computation time.\n",
    "\n",
    "Other file-formats such as `'parquet'` and `'feather'` are also supported, but these have some restrictions on the DataFrames they can save. The Parquet file-format only supports Pandas DataFrames (not Series). The column-names must also start with a letter. There may be other requirements imposed by the Parquet file-format, and you will get an exception if the DataFrame violates the requirements. The Feather file-format is even more basic and cannot save DataFrames with MultiIndex. So it is generally best to use the default pickle-format or the compressed pickle-format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Caching a SimFin Function\n",
    "\n",
    "Here is an example of a function from the simfin package for calculating share-price signals. This takes about 30 seconds to compute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 19.2 s, sys: 705 ms, total: 19.9 s\n",
      "Wall time: 16.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_price_signals = sf.price_signals(df_prices=df_prices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `sf.price_signals` is actually wrapped with ` @sf.cache` so the caching-feature is automatically enabled if we pass an argument `cache_refresh` different from `None`. For example, this is how we would instruct the cache-file to get updated once per day:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Name for the cache e.g. 'us-all'\n",
    "cache_name = market + '-all'\n",
    "\n",
    "# Refresh the cache once a day.\n",
    "cache_refresh_days = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cache-file 'price_signals-us-all.pickle' not on disk.\n",
      "- Running function price_signals() ... Done!\n",
      "- Saving cache-file to disk ... Done!\n",
      "CPU times: user 19.3 s, sys: 775 ms, total: 20 s\n",
      "Wall time: 17 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_price_signals2 = \\\n",
    "    sf.price_signals(df_prices=df_prices,\n",
    "                     cache_name=cache_name,\n",
    "                     cache_refresh=cache_refresh_days)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first time the function is called, it will compute the signals and save the resulting DataFrame to a cache-file on disk. When the function is called again, the cached DataFrame will be loaded instead. When the cache-file is too old, the function is called again and a new cache-file is saved to disk.\n",
    "\n",
    "Note that the cache-file is named `price_signals-us-all.pickle` which is constructed from the function's name `price_signals`, the cache-name we have supplied `us-all`, and the file-extension `.pickle`. This keeps the cache-files neatly organized on disk, while still allowing us to designate different cache-names for different calls of the same function, for example if we want to process different markets or stocks.\n",
    "\n",
    "If you want to pass the same cache-arguments to several functions, then it is more convenient to create a dict with the arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache_args = {'cache_name': cache_name,\n",
    "              'cache_refresh' : cache_refresh_days}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cache-file 'price_signals-us-all.pickle' on disk (0 days old).\n",
      "- Loading from disk ... Done!\n",
      "CPU times: user 73.1 ms, sys: 140 ms, total: 213 ms\n",
      "Wall time: 212 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_price_signals3 = \\\n",
    "    sf.price_signals(df_prices=df_prices, **cache_args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that the results are all identical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_price_signals.equals(df_price_signals2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_price_signals.equals(df_price_signals3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Caching Your Own Functions\n",
    "\n",
    "You can also use the caching-feature on your own functions simply by adding the decorator ` @sf.cache` to your function declaration. Here is an example of a function that calculates the sum of each row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "@sf.cache\n",
    "def my_function(df):\n",
    "    return df.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that you **MUST** call `my_function()` with named arguments! Otherwise you will get a strange exception. The reason is that the decorator has actually created a new function which takes the arguments: `cache_name`, `cache_refresh`, `cache_format` and `**kwargs`, as we can see from this slightly cryptic specification of the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FullArgSpec(args=['cache_name', 'cache_refresh', 'cache_format'], varargs=None, varkw='kwargs', defaults=(None, None, 'pickle'), kwonlyargs=[], kwonlydefaults=None, annotations={})"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import inspect\n",
    "inspect.getfullargspec(my_function)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So if you call `my_function()` with unnamed arguments, it expects the first arguments to be `cache_name`, `cache_refresh` and `cache_format`, while any remaining keyword arguments are passed to the original function. This raises a strange exception:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my_function() missing 1 required positional argument: 'df'\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    df_result = my_function(df_prices)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you **MUST** use keyword-arguments when calling a function wrapped with ` @sf.cache`. But we can still call the function without passing a `cache_refresh` argument, in which case it will disable the caching and just call the wrapped function as normal, but again you **MUST** use named arguments such as `df=df_prices` instead of just `df_prices`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 994 ms, sys: 236 ms, total: 1.23 s\n",
      "Wall time: 518 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_result = my_function(df=df_prices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ticker  Date      \n",
       "A       2007-01-03    2620607.77\n",
       "        2007-01-04    2119705.80\n",
       "        2007-01-05    2722605.60\n",
       "        2007-01-08    1603204.44\n",
       "        2007-01-09    1432204.80\n",
       "dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_result.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we pass the cache-arguments then the caching is automatically enabled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cache arguments.\n",
    "cache_name = market + '-all'\n",
    "cache_refresh = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cache-file 'my_function-us-all.pickle' not on disk.\n",
      "- Running function my_function() ... Done!\n",
      "- Saving cache-file to disk ... Done!\n",
      "CPU times: user 1.08 s, sys: 285 ms, total: 1.37 s\n",
      "Wall time: 647 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_result2 = my_function(df=df_prices,\n",
    "                         cache_name=cache_name,\n",
    "                         cache_refresh=cache_refresh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may also create a dict with the cache-arguments, which is convenient if we want to use the same arguments in several functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache_args = {'cache_name': cache_name,\n",
    "              'cache_refresh' : cache_refresh}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cache-file 'my_function-us-all.pickle' on disk (0 days old).\n",
      "- Loading from disk ... Done!\n",
      "CPU times: user 38.2 ms, sys: 32.4 ms, total: 70.6 ms\n",
      "Wall time: 69.6 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_result3 = my_function(df=df_prices, **cache_args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even for such a fairly quick function, the caching still saved a lot of time when using the raw pickle-format. But normally you would only use the caching-feature on functions that are very slow to compute.\n",
    "\n",
    "We can check that the results are all identical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_result.equals(df_result2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_result.equals(df_result3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "This is published under the\n",
    "[MIT License](https://github.com/simfin/simfin-tutorials/blob/master/LICENSE.txt)\n",
    "which allows very broad use for both academic and commercial purposes.\n",
    "\n",
    "You are very welcome to modify and use this source-code in your own project. Please keep a link to the [original repository](https://github.com/simfin/simfin-tutorials).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}