sktime_worksheet_forecasting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# sktime worksheet - forecasting\n",
    "\n",
    "This is an exercise style worksheet introducing the [sktime](https://github.com/alan-turing-institute/sktime) forecasting module, for self-study and user testing.\n",
    "\n",
    "This worksheet contains exercises for you to attempt after:\n",
    "* going through the basic `sktime` demo, live or [on youtube](https://www.youtube.com/watch?v=wqQKFu41FIw)\n",
    "* having had a look at the tutorial notebooks yourself.\n",
    "\n",
    "For this sheet, you are actively encouraged to seek out the sktime documentation and tutorial notebooks to solve the exercises.\n",
    "\n",
    "We would very much appreciate if you could leave feedback, especially critical feedback, in the markdown fields meant for this. You can also raise issues on any bugs or improvement suggestions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1: basic forecasting\n",
    "\n",
    "(a) use an ARMA(10,10) forecaster to forecast the series `y` below for the next 100 indices ahead.\n",
    "\n",
    "(b) plot the series and the forecast\n",
    "\n",
    "(c) obtain 5%/95% prediction intervals, and add them to the plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = np.sin(np.linspace(0, 30, 1000)) + 0.1*np.random.randn(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "source": [
    "Questions:\n",
    "* was the forecaster easy/difficult to find? was the documentation helpful?\n",
    "* was the forecasting interface easy/difficult to understand?\n",
    "* anything else you would like to tell us?"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "Remarks:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### Exercise 2: evaluating forecasters\n",
    "\n",
    "(a) evaluate the forecaster above, backtesting on the last 20% of the series `y`, evaluating by mean absolute error of prediction\n",
    "\n",
    "(b) plot the actual series against the prediction, with 5%/95% prediction intervals\n",
    "\n",
    "(c) evaluate the forecaster above, taking double average of absolute error over a rolling 10-steps-ahead prediction, starting at the 200th index, with a rolling step size of 10\n",
    "\n",
    "(d) obtain prediction intervals, and add them to the plot"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "source": [
    "Questions:\n",
    "* was it easy/difficult to find out how to do backtesting evaluation?\n",
    "* was it easy/difficult to find out how to do rolling evaluation?\n",
    "* was the absolute error metric easy/difficult to find?\n",
    "* was it easy/difficult to find out how to obtain prediction intervals?"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### Exercise 3: composite forecasters\n",
    "\n",
    "(a) build the following forecaster: reduction to supervised learning using the direct recursive strategy with ordinary least squares regression, on a rolling window of length 10.\n",
    "\n",
    "(b) use the forecaster in (a) to forecast the series `y` below for the next 100 indices ahead.\n",
    "\n",
    "(c) to the forecaster in (a), add deseasonalization as a pre-processing step. Tune the de-seasonalization parameter using grid parameter search with a suitable sliding window splitter. Try to avoid overly high runtimes.\n",
    "\n",
    "(d) fit the (deseasonalized) forecaster in (c) to `y`, and check whether the tuning procedure results in the \"obvious\" seasonality."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "source": [
    "Questions:\n",
    "* was it easy/difficult to find out how to construct the reduction?\n",
    "* was it easy/difficult to find out how to deseasonalize the forecaster?\n",
    "* was it easy/difficult to find out how to tune the forecaster?"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4: Model selection\n",
    "Construct a forecaster which, using model selection (=autoML) based on sliding window evaluation via mean absolute error, at each time step selects one of: (i) deseasonalized moving average, (ii) autoARIMA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "source": [
    "Questions:\n",
    "* was it easy/difficult to find out how to construct the model selection composite?\n",
    "* was it easy/difficult to find out how to use autoARIMA?"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General feedback:\n",
    "* What was the most difficult part to understand?\n",
    "* Did you get stuck anywhere or had to skip? \n",
    "* How can we improve the documentation? \n",
    "* What additional features would you like to see?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.7.10 64-bit ('sktime': conda)",
   "metadata": {
    "interpreter": {
     "hash": "1d08445f682e67118652b541d82c54c8d542bbf9d181dd3acc4827a948ac2085"
    }
   }
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}