triscale_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TriScale tutorial\n",
    "\n",
    "|18-05-2021|90 min version|CPS-IoTBench 2021|\n",
    "|---|---|---|\n",
    "\n",
    "This notebook contains \n",
    "1. A short demonstration of _TriScale_ API, how the different functions are meant to be used, and the visualizations they produce. \n",
    "2. A few simple tasks that invite you to use _TriScale_ to \n",
    "    1. design an experiment, and \n",
    "    2. analysis (synthetic) data to assess the replicability of the results.\n",
    "\n",
    "For more details about _TriScale,_ you may refer to [the paper](https://doi.org/10.5281/zenodo.3464273).\n",
    "\n",
    "\n",
    "## Following with tutorial\n",
    "To follow the tutorial, some basic knowledge of Python is required. \n",
    "You are invited to play around with the cells from this Jupyter notebook, which interleaves code snippets and text explanations. \n",
    "\n",
    "**Not familiar with Jupyter notebooks?** No worries!  \n",
    "You need very little knowledge to successfully follow the tutorial, most of which you can find from these doc pages\n",
    "- [Notebook user interface and Document structure](https://jupyter-notebook.readthedocs.io/en/stable/notebook.html#notebook-user-interface)\n",
    "- [Basic wokflow](https://jupyter-notebook.readthedocs.io/en/stable/notebook.html#basic-workflow)\n",
    "- [User interface components](https://jupyter-notebook.readthedocs.io/en/stable/ui_components.html) \n",
    "\n",
    "\n",
    "## List of imports\n",
    "To get started, let's import the Python libraries we need in this tutorial. All the _TriScale_-specific functions are part of one module called `triscale`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import triscale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 0. Performance evaluation of Banana\n",
    "\n",
    "Throughout the tutoriel presentation, we used the Banana communication protocol as an example. Let's see how easily we can improve the experiment design and analysis with _TriScale._\n",
    "\n",
    "> We want to measure the overall energy consumption achieved by the protocol.  \n",
    "For this purpose, we can use a simple metric: the sum of the energy consumed by all nodes in the network. The lower, the better.  \n",
    "> \n",
    "> Note that we could pick _any metric_; the choice of the metric is independent of TriScale's methodology.\n",
    "\n",
    "TriScale defines performance objectives as percentiles of the metric distributions. The experiments aim to estimate these percentiles with a given confidence level: that's what we call the KPIs, or Key Performance Indicators. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Definition of Banana's KPI\n",
    "percentile = 50 # the median\n",
    "confidence = 95 # the confidence level, in %"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two values are sufficient to define the minimal number of runs required to compute this KPI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "triscale.experiment_sizing(\n",
    "    percentile, \n",
    "    confidence,\n",
    "    verbose=True); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a **minimum of 5 runs.**\n",
    "\n",
    "We do the same thing to estimate the long-term variability with the variability score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Definition of Banana's variability score\n",
    "percentile = 25 # the median\n",
    "confidence = 95 # the confidence level, in %"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "triscale.experiment_sizing(\n",
    "    percentile, \n",
    "    confidence,\n",
    "    verbose=True); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a **minimum of 11 series.**\n",
    "\n",
    "Hence, with only these four parameters, we can connect the total number of runs one needs to run (a minimum of 11 series of 5 runs) with the corresponding performance claims that one can make:\n",
    "- **KPI**: In a series of runs, the median value of the runs metric values is lower or equal to the KPI with a confidence of 95%.\n",
    "- **Variability score**: The range of KPI values of the middle 50% of series is less or equal to the variability score, with a confidence of 95%.\n",
    "\n",
    "--- \n",
    "\n",
    "The rest of this notebook provides more details about how TriScale computes these minimal numbers of runs and series (Part 1.). It then introduces an actual experiment dataset as support to present TriScale's analysis functions (Part 2.). \n",
    "\n",
    "Have fun!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1. Experiment design\n",
    "\n",
    "During the design phase of an experiment, one important question to answer is \"how many time should the experiments be performed?\" This question directly relates to the definition of _TriScale_ KPIs and variability scores. \n",
    "\n",
    "_TriScale_ implements a statistical method that allows to estimate, based on a data sample, any percentile of the underlying distribution with any level of confidence. Importantly, the estimation does not rely on any assumption on the nature of the underlying distribution (eg normal, or Poisson). The estimate is valid as long as the sample is independent and identically distributed (or _iid_ ).\n",
    "\n",
    "Intuitively, it is \"easier\" to estimate the median (50th percentile) than the 99th percentile; the more extreme the percentile, the more samples are required to provide an estimate for a given level of confidence. More precisely, the minimal number of sample $N$ required to estimate a percentile $0<p<1$ with confidence $0<C<1$ is given by:\n",
    "\n",
    "$$N \\;\\geq\\; \\frac{log(1-C)}{log(1-p)}$$\n",
    "\n",
    "_TriScale_ `experiment_sizing()` function implements this computation and retuens the minimal number of samples $N$, as illustrated below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Select the percentile we want to estimate \n",
    "percentile = 10\n",
    "\n",
    "# Select the desired level of confidence for the estimation\n",
    "confidence = 99 # in %\n",
    "\n",
    "# Compute the minimal number of samples N required\n",
    "triscale.experiment_sizing(\n",
    "    percentile, \n",
    "    confidence,\n",
    "    verbose=True); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us consider the samples $x$ are ordered such that $x_1 \\leq x_2 \\ldots \\leq x_N$. \n",
    "The previous result indicates that for $N = 44$ samples and above,  $x_1$ is a lower bound for the 10th percentile with probibility larger than 99%. \n",
    "\n",
    "Observe that the probabilities are symetric; it takes the same number of samples to compute a lower bound for the 10th percentile as to compute an upper bound for the 90th percentile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "percentile = 90\n",
    "confidence = 99 # in %\n",
    "\n",
    "triscale.experiment_sizing(\n",
    "    percentile, \n",
    "    confidence,\n",
    "    verbose=True); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the `experiment_sizing` functions automatically returns an upper bound for percentiles $P>50$ and a lower bound for percentiles $P<50$.\n",
    "\n",
    "To get a better feeling of how this minimal number of samples evolves this increasing confidence and more extreme percentiles, let us compute a range of minimal number of samples and display the results in a table (where the columns are the percentiles to estimate)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sets of percentiles and confidence levels to try\n",
    "percentiles = [0.1, 1, 5, 10, 25, 50, 75, 90, 95, 99, 99.9]\n",
    "confidences = [75, 90, 95, 99, 99.9, 99.99]\n",
    "min_number_samples = []\n",
    "\n",
    "# Computing the minimum number of runs for each (perc., conf.) pair\n",
    "for c in confidences:\n",
    "    tmp = []\n",
    "    for p in percentiles:\n",
    "        N = triscale.experiment_sizing(p,c)\n",
    "        tmp.append(N[0])\n",
    "    min_number_samples.append(tmp)\n",
    "    \n",
    "# Put the results in a DataFrame for a convenient display of the results\n",
    "df = pd.DataFrame(columns=percentiles, data=min_number_samples)\n",
    "df['Confidence level'] = confidences\n",
    "df.set_index('Confidence level', inplace=True)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, one can compute the minimal number $N$ such that any sample $x_m$ is an estimate (instead of $x_1$). This can be obtained from the `experiment_sizing()` function using the option `robustness` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "percentile = 10\n",
    "confidence = 99\n",
    "triscale.experiment_sizing(\n",
    "    percentile, \n",
    "    confidence,\n",
    "    robustness=3,\n",
    "    verbose=True); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous result indicates that for $N = 97$ samples and above,  $x_4$ is a lower bound for the 10th percentile with probibility larger than 99%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1. Your turn! \n",
    "\n",
    "Based on the explanations above, use _TriScale_'s `experiment_sizing` function to answer the following questions:\n",
    "- What is the minimal number of runs required to estimate the **90th** percentile with **90%** confidence?\n",
    "- What is the minimal number of runs required to estimate the **90th** percentile with **95%** confidence?\n",
    "- What is the minimal number of runs required to estimate the **95th** percentile with **90%** confidence?\n",
    "- Based on the answers to the previous questions, is it harder (i.e., does it require more runs) to increase the confidence level, or to estimate a more extreme percentile? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \n",
    "# \n",
    "# \n",
    "# You can either write your own code here, or modify and re-execute some of the cells above. \n",
    "# \n",
    "# \n",
    "# Have fun! \n",
    "# \n",
    "# \n",
    "# \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answering these questions should be fairly easy. Want a harder one? Then try this:  \n",
    "- For $N = 50$ samples, what is the index $m$ of the best possible (i.e., the largest) lower bound for the 25th percentile, estimated with a 95% confidence level? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2. Data analysis\n",
    "\n",
    "### Runs and Metrics\n",
    "\n",
    "Metrics in _TriScale_ evaluate a performance dimension across a run. The computation of metrics is implmented in the `analysis_metric()` functions, which takes two compulsory arguments:\n",
    "- the raw data,\n",
    "- the metric definition,\n",
    "\n",
    "The raw data can be passed as a file name (ie, a string) or as a Pandas dataframe. \n",
    "- If a string is passed, the function tries to read the file name as a csv file (comma separated) with `x` data in the first column and `y` data in the second column. \n",
    "- If a pandas DataFrame is passed, `data` must contain columns named `x` and `y`.\n",
    "\n",
    "The metric definition is provided as a dictionary, with only the `measure` key being compulsory. This defines \"what is the computation to be performed\" on the data. The measure can be any percentile ($0<P<100$) or `mean`, `minimum`, `maximum`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input data file (one-way delay of a full-throttled flow using TPC BBR)\n",
    "data = 'ExampleData/raw_data.csv'\n",
    "\n",
    "# Definition of a TriScale metric\n",
    "metric = {  \n",
    "    'measure': 50,   # Integer: interpreted as a percentile\n",
    "    'unit'   : 'ms', # For display only\n",
    "         }\n",
    "\n",
    "has_converged, metric_measure, plot = triscale.analysis_metric( \n",
    "    data,\n",
    "    metric)\n",
    "\n",
    "print('Run metric: %0.2f %s' % (metric_measure, metric['unit']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Passing the optional argument `plot=True` generates a plot of the raw data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "has_converged, metric_measure, plot = triscale.analysis_metric( \n",
    "    data,\n",
    "    metric,\n",
    "    plot=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As presented here, this function is not very interesting: it \"only\" returns some percentile of an array... \n",
    "The main value of the `analysis_metric()` function is when the metric attempts to estimate the long-term performance; that is, the value one would obtain shall the run last longer/more data points be collected. \n",
    "When this is the case, _TriScale_ performs a convergence test on the data, which can be triggered in `analysis_metric()` function by passing the optional `convergence` parameter. \n",
    "The study is convergence goes beyond the scope of this tutorial; refer to the [paper](https://doi.org/10.5281/zenodo.3464273) for more details. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series and KPIs\n",
    "\n",
    "_TriScale_ ’s key performance indicators (KPIs) evaluate performance dimensions across a series of runs. Performing multiple runs allows to mitigate the inherent variability of the experimental conditions. KPIs capture this variability by estimating percentiles of the (unknown) metric distributions. Concretely, a _TriScale_ KPI is a one-sided confidence interval of a percentile, e.g., a lower bound for the 25th percentile of a throughput metric, estimated with a 95% confidence level.\n",
    "\n",
    "The computation of KPIs is implmented in the `analysis_kpi()` function, which takes two compulsory arguments:\n",
    "- the metric data\n",
    "- the KPI definition\n",
    "\n",
    "The metric data can be passed as a list or an NumPy array. \n",
    "The KPI definition is provided as a dictionary with three compulsory keys: `percentile` ($0<P<100$), `confidence` ($0<C<100$), and `bounds`. The KPI bounds are the expected extremal values of the metric; this is necessary to perform the independance test (see below).\n",
    "\n",
    "> If the metrics bounds are unknown, simply pass the minimum and maximum metric values as bounds.\n",
    "\n",
    "The `analysis_kpi()` function performs two computations:\n",
    "1. It performs an empirical independence test; that is, the function tests whether the metric data appears to be iid.\n",
    "2. It computes the KPI value for the metric data.  \n",
    "\n",
    "The metric data must be iid for the KPI to be a valid estimate of the underlying metric distribution. In general, independence is a property of the data collection process. However, in many preactical cases for networking experiment, independence cannot be guaranteed (for example, because there are correlations between the interference conditions between sucessive experiments). \n",
    "In such a case, one can perform an _empirical_ test for independence; essentially, this assesses whether the level of correlation in the data appears sufficiently low such that the data can be assumed iid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Load sample metrics data (failure recovery time, in seconds)\n",
    "data = 'ExampleData/metric_data.csv'\n",
    "df = pd.read_csv(data, header=0, names=['metric'])\n",
    "\n",
    "# Minimal KPI definition\n",
    "KPI = {\n",
    "    'percentile': 75,\n",
    "    'confidence': 95,\n",
    "    'bounds': [0,10],\n",
    "    'unit': 's'\n",
    "}\n",
    "\n",
    "# Computes the KPI\n",
    "indep_test_passed, KPI_value = triscale.analysis_kpi(\n",
    "    df.metric.values,\n",
    "    KPI,\n",
    ")\n",
    "\n",
    "# Output\n",
    "if indep_test_passed:\n",
    "    print('The metric data appears iid.')\n",
    "    print('KPI value: %0.2f %s' % (KPI_value, KPI['unit']))\n",
    "else:\n",
    "    print('The metric data does not appear iid.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the metric data appears to be iid, we can interpret the KPI value as follows:  \n",
    "> __With a confidence level of 95%, the 75th percentile on the metric is smaller or equal to 1.92s.__   \n",
    "In other words, with a probability of 95%, the performance metric is smaller or equale to 1.92s in three quarters of the runs. \n",
    "\n",
    "If the independence test fails, the KPI value is computed and returned nonetheless. However, the user must be aware that the resulting KPI is then not a trustworthy estimate of the corresponding percentile. Refer to the [paper](https://doi.org/10.5281/zenodo.3464273) for more details about the empirical independence test implemented in _TriScale._\n",
    "\n",
    "Optionally, the `analysis_kpi()` functions plots \n",
    "- the metric data series (`series`)\n",
    "- the autocorrelation plot (`autocorr`)\n",
    "- the metric data and the corresponding KPI value (`horizontal`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Computes the KPI and plot\n",
    "indep_test_passed, KPI_value = triscale.analysis_kpi(\n",
    "    df.metric.values,\n",
    "    KPI,\n",
    "    # to_plot=['series','autocorr','horizontal']\n",
    "    to_plot=['horizontal']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequels and Variability Scores\n",
    "\n",
    "Sequels are repetitions of series of runs. _TriScale_ ’s variability score evaluates the variations of KPI values across sequels. Sequels enable _TriScale_ to detect long-term variations of KPIs and ultimately quantify the reproducibility of an experiment. \n",
    "Concretely, a variability score is a two-sided confidence interval, i.e., an estimation of a symmetric pair of percentiles. For example, a 75% confidence interval for the 25-75th percentiles.\n",
    "The underlying computations are the same as for the [KPIs values](#Series-and-KPIs).\n",
    "\n",
    "The computation of variability scores is implmented in the `analysis_variability()` function, which takes two compulsory arguments:\n",
    "- the KPI data\n",
    "- the variability score definition\n",
    "\n",
    "The KPI data can be passed as a list or an NumPy array. \n",
    "The variability score definition is provided as a dictionary with three compulsory keys: `percentile` ($0<P<100$), `confidence` ($0<C<100$), and `bounds`. The bounds are the expected extremal values of the KPI; this is necessary to perform the independance test (see below).\n",
    "\n",
    "> If the KPI bounds are unknown, simply pass the minimum and maximum KPI values as bounds.\n",
    "\n",
    "Like in `analysis_kpi()`, `analysis_variability()` performs both the empirical independence test and the computation of the variability score. The same plotting options are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load sample KPI data (failure recovery time, in seconds)\n",
    "data = 'ExampleData/kpi_data.csv'\n",
    "df = pd.read_csv(data, header=0, names=['kpi'])\n",
    "\n",
    "# Minimal KPI definition\n",
    "score = {\n",
    "    'percentile': 25, # the 25th-75th  percentiles range\n",
    "    'confidence': 95,\n",
    "    'bounds': [0,10],\n",
    "    'unit': 's'\n",
    "}\n",
    "\n",
    "# Computes the KPI\n",
    "(indep_test_passed, \n",
    " upper_bound, \n",
    " lower_bound, \n",
    " var_score, \n",
    " rel_score) = triscale.analysis_variability(\n",
    "    df.kpi.values,\n",
    "    score,\n",
    "    to_plot=['series','horizontal']\n",
    ")\n",
    "\n",
    "# Output\n",
    "if indep_test_passed:\n",
    "    print('The KPI data appears iid.')\n",
    "    print('Variability score: %0.2f %s' % (var_score, score['unit']))\n",
    "else:\n",
    "    print('The KPI data does not appear iid.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the KPI data appears to be iid, we can interpret the variability score as follows:  \n",
    "> __With a confidence level of 95%, the inter-quartile (25th-75th perc) range on the KPIs is smaller or equal to 0.4s.__   \n",
    "In other words, with a probability of 95%, across all series, the middle 50% of KPI values differ by 0.4s or less.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2. Your turn! \n",
    "We have collected data for a comparative evaluation of congestion-control schemes using the [Pantheon platform](https://pantheon.stanford.edu/). Details about the experiment setup can be found in the [TriScale paper](https://doi.org/10.5281/zenodo.3464273).\n",
    "\n",
    "For the purpose of this tutorial, we provide a dataset containing the metric values computed based on the raw data of each run:\n",
    "- the mean **throughput** \n",
    "- the 95th percentile of the **one-way delay**\n",
    "\n",
    "We performed five series of ten runs each. You task consist in analysing this data using _TriScale_ in order to compute and interpret variability scores for different congestion-control schemes. \n",
    "\n",
    "Let us first load and visualise the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the entire dataset\n",
    "df = pd.read_csv(Path('ExampleData/metrics_wo_convergence.csv'))\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell contains a simple function to filter this dataset to extract metric values per scheme and per series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the list of congestion control schemes and experiment dates\n",
    "cc_scheme = df.cc.unique()\n",
    "dates = df.datetime.unique()\n",
    "\n",
    "# Uncomment to print out the list of schemes\n",
    "# print(cc_scheme)\n",
    "\n",
    "# Function parsing the dataset to extract metric values for one scheme and one series of runs\n",
    "def filter_results(df, cc, date, metric):\n",
    "\n",
    "    # Setup the data filter\n",
    "    filter = (\n",
    "        (df.cc == cc) &\n",
    "        (df.datetime == date) \n",
    "    )\n",
    "    # Filter\n",
    "    df_filtered = df.where(filter).dropna()\n",
    "\n",
    "    # Return the desired metric data\n",
    "    return df_filtered[(metric+'_value')].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this function to easily extract all metrics values for one scheme and one metric (e.g., the mean `throughput` of `copa`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let us extract the metric values for the 5 series of runs\n",
    "cc = 'copa'\n",
    "metric = 'throughput' # valid options: 'throughput' and 'delay'\n",
    "results = []\n",
    "\n",
    "for date in dates:\n",
    "    values = filter_results(df, cc, date, metric)\n",
    "    results.append(list(values))\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now analyse these data! The KPI and variability score definitions for both the `throughput` and `delay` are provided below. \n",
    "\n",
    "> Note that we aim to estimate the 25th percentile for the `throughput`, where higher is better; whereas we estimate the 75th percentile of the `delay`, where lower is better.  \n",
    "Thus, for both metrics, the KPI provides the least performance expected in at least 75% of the runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# KPIs\n",
    "KPI_tput  = {'percentile': 25,\n",
    "             'confidence': 75,\n",
    "             'name': 'KPI Throughput',\n",
    "             'unit': 'Mbit/s',\n",
    "             'bounds':[0,120],    # expected value range\n",
    "            }\n",
    "KPI_delay = {'percentile': 75,\n",
    "             'confidence': 75,\n",
    "             'name': 'KPI One-way delay',\n",
    "             'unit': 'ms',\n",
    "             'bounds':[0,100],    # expected value range\n",
    "            }\n",
    "\n",
    "# Variability scores\n",
    "score_tput  = {'percentile': 75,\n",
    "             'confidence': 75,\n",
    "             'name': 'Throughput',\n",
    "             'unit': 'Mbit/s',\n",
    "             'bounds':[0,120],    # expected value range\n",
    "            }\n",
    "score_delay = {'percentile': 75,\n",
    "             'confidence': 75,\n",
    "             'name': 'One-way delay',\n",
    "             'unit': 'ms',\n",
    "             'bounds':[0,100],    # expected value range\n",
    "            }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are now all set to analyse these data! \n",
    "1. Use _TriScale_'s `analysis_kpi` and `analysis_variability` functions to compute the variability score for one scheme and one performance metric (whichever ones).\n",
    "2. Modify the definition of the variability score to estimate the median (`'percentile': 50`) instead of the 25-75th percentile range. How does this change the variability score obtained? Does this make sense to you?\n",
    "3. Compute the scores for a few different schemes. Do they vary a lot? Does this variability appear \"big\" with respect to the range of KPI values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \n",
    "# \n",
    "# \n",
    "# You can either write your own code here, or modify and re-execute some of the cells above. \n",
    "# \n",
    "# \n",
    "# Have fun! \n",
    "# \n",
    "# \n",
    "# \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "If you got down all this way, congratulations! We hope you found this tutorial useful, and that you will remember some these techniques for the design and analysis of your next experiment! \n",
    "\n",
    "Of course, many complexities and aspects of _TriScale_ were not discussed in this tutorial. For more details, you can explore the [(longer) live demo](https://mybinder.org/v2/gh/TriScale-Anon/triscale/master?filepath=triscale_demo.ipynb) and the [TriScale paper](https://doi.org/10.5281/zenodo.3464273)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.1 64-bit ('triscale': conda)",
   "name": "python391jvsc74a57bd0684f90775fb1f43db0d8eed0780ba829f42a97c2f4b9a1bd592c18e47e5c272e"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  },
  "metadata": {
   "interpreter": {
    "hash": "684f90775fb1f43db0d8eed0780ba829f42a97c2f4b9a1bd592c18e47e5c272e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}