Inference Time.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference Time\n",
    "This notebook contains details on the inference time measurements reported in table V of the paper.\n",
    "\n",
    "### *Optional Config and Installation*\n",
    "\n",
    "Simply jump over the steps you already did set up.\n",
    "\n",
    "**1. Configuration**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xview\n",
    "from os import path\n",
    "\n",
    "# PLEASE EDIT THE FOLLOWING PATHS FOR YOUR LOCAL SETTINGS\n",
    "\n",
    "# path where the image data will be/is stored\n",
    "DATA_BASEPATH = '/tmp/test'\n",
    "\n",
    "# path where experiment configs will be stored\n",
    "EXPERIMENT_STORAGE_FOLDER = '/tmp/exp'\n",
    "\n",
    "# only change in case you want to use tensorboard for model training, otherwise not relevant\n",
    "EXP_OUT = '/tmp'\n",
    "\n",
    "print('writing settings to %s' % path.join(path.dirname(xview.__file__), 'settings.py'))\n",
    "with open(path.join(path.dirname(xview.__file__), 'settings.py'), 'w') as settings_file:\n",
    "    settings_file.write(\"DATA_BASEPATH = '%s'\\n\" % DATA_BASEPATH)\n",
    "    settings_file.write(\"EXPERIMENT_STORAGE_FOLDER = '%s'\\n\" % EXPERIMENT_STORAGE_FOLDER)\n",
    "    settings_file.write(\"EXP_OUT = '%s'\\n\" % EXP_OUT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Downloading Experimental Data**  \n",
    "All training and measurement experiments are bundeled into an archive that is downloaded and installed. This gives you access to pre-trained models and all experimental configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget http://robotics.ethz.ch/~asl-datasets/2018_modular_semantic_segmentation/experimental_data.tar.gz -O /tmp/experimental_data.tar.gz\n",
    "import tarfile\n",
    "from os import path\n",
    "from xview.settings import EXPERIMENT_STORAGE_FOLDER\n",
    "tar = tarfile.open('/tmp/experimental_data.tar.gz')\n",
    "tar.extractall(path=EXPERIMENT_STORAGE_FOLDER)\n",
    "tar.close()\n",
    "! rm /tmp/experimental_data.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference Time\n",
    "The following cells load the results from our timing experiments.\n",
    "\n",
    "The inference time was measured using on a single GPU that did not have enough memory to make use of any parallellisation possibilities. In addition, the inference time measurements do not include time required to load data into the gpu as this heavily depends on code optimization and used hardware specs. We evaluated the inference time on a defined constant of equal size as the actual RGB and Depth images used. For more details, please have a look at [the implementation of the experiment](https://github.com/ethz-asl/modular_semantic_segmentation/blob/publish/experiments/timing.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from experiments.utils import ExperimentData\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Load the relevant experiments. If this fails, your config from above is not setup properly.\n",
    "rgb = ExperimentData(1062)\n",
    "depth = ExperimentData(1063)\n",
    "bayes = ExperimentData(1071)\n",
    "dirichlet = ExperimentData(1067)\n",
    "variance = ExperimentData(1065)\n",
    "fusionfcn = ExperimentData(1059)\n",
    "average = ExperimentData(1064)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fusion_Fcn</th>\n",
       "      <th>Dirichlet</th>\n",
       "      <th>Bayes</th>\n",
       "      <th>Average</th>\n",
       "      <th>Variance</th>\n",
       "      <th>RGB</th>\n",
       "      <th>Depth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.0720</td>\n",
       "      <td>0.0517</td>\n",
       "      <td>0.0461</td>\n",
       "      <td>0.0432</td>\n",
       "      <td>0.3064</td>\n",
       "      <td>0.0219</td>\n",
       "      <td>0.0218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.0221</td>\n",
       "      <td>0.0238</td>\n",
       "      <td>0.0156</td>\n",
       "      <td>0.0113</td>\n",
       "      <td>0.0183</td>\n",
       "      <td>0.0114</td>\n",
       "      <td>0.0121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fusion_Fcn  Dirichlet   Bayes  Average  Variance     RGB   Depth\n",
       "mean      0.0720     0.0517  0.0461   0.0432    0.3064  0.0219  0.0218\n",
       "std       0.0221     0.0238  0.0156   0.0113    0.0183  0.0114  0.0121"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def print_table(**experiments):\n",
    "    table_data = {}\n",
    "    for key, experiment_id in experiments.items():\n",
    "        text = ExperimentData(experiment_id).get_record()['captured_out'].split('\\n')\n",
    "        line = next(line for line in text if line.startswith('Mean Time'))\n",
    "        mean = float(line[10:17])\n",
    "        std = float(line[24:31])\n",
    "        table_data[key] = [mean, std]\n",
    "\n",
    "    classes = pd.DataFrame(table_data, index=['mean', 'std'])\n",
    "    \n",
    "    # now order the columns\n",
    "    ordered_columns = []\n",
    "    for key in ['Fusion_Fcn', 'Dirichlet', 'Bayes', 'Average', 'Variance', 'RGB', 'Depth']:\n",
    "        if key in classes.columns:\n",
    "            ordered_columns.append(key)\n",
    "    return classes[ordered_columns]\n",
    "\n",
    "pd.set_option('display.precision', 4)\n",
    "\n",
    "print_table(RGB=1062, Depth=1063, Bayes=1071, Dirichlet=1067, Variance=1065, Fusion_Fcn=1059, Average=1064)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to reproduce timing measurements\n",
    "To reproduce the evaluation experiments listed above, simply invoke the following command with the correct `experiment_id`:  \n",
    "More info on the experiment management can be found [here](http://sacred.readthedocs.io/en/latest/).\n",
    "\n",
    "**The timing results in the paper were measured on an NVIDIA GTX 1080 Ti major. Any other GPU or CPU, aswell as other hardware setups can have influence on the timing analysis. It is therefore very hard to exactly reproduce the above results.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m experiments.rerun with experiment_id=1071 -u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}