Experimental Details.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experimental Details\n",
    "\n",
    "This notebook contains short examples on how to access all data from our published experiments. For further information, we refer the reader to [the documentation of the sacred framework](https://sacred.readthedocs.io/en/latest/) and [the util functions in our repository](https://github.com/ethz-asl/modular_semantic_segmentation/blob/6f7ff1839ac2f9b22fc3213f57a28d67807c2c50/experiments/utils.py#L61).\n",
    "\n",
    "### *Optional Config and Installation*\n",
    "\n",
    "Simply jump over the steps you already did set up.\n",
    "\n",
    "**1. Configuration**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xview\n",
    "from os import path\n",
    "\n",
    "# PLEASE EDIT THE FOLLOWING PATHS FOR YOUR LOCAL SETTINGS\n",
    "\n",
    "# path where the image data will be/is stored\n",
    "DATA_BASEPATH = '/tmp/test'\n",
    "\n",
    "# path where experiment configs will be stored\n",
    "EXPERIMENT_STORAGE_FOLDER = '/tmp/exp'\n",
    "\n",
    "# only change in case you want to use tensorboard for model training, otherwise not relevant\n",
    "EXP_OUT = '/tmp'\n",
    "\n",
    "print('writing settings to %s' % path.join(path.dirname(xview.__file__), 'settings.py'))\n",
    "with open(path.join(path.dirname(xview.__file__), 'settings.py'), 'w') as settings_file:\n",
    "    settings_file.write(\"DATA_BASEPATH = '%s'\\n\" % DATA_BASEPATH)\n",
    "    settings_file.write(\"EXPERIMENT_STORAGE_FOLDER = '%s'\\n\" % EXPERIMENT_STORAGE_FOLDER)\n",
    "    settings_file.write(\"EXP_OUT = '%s'\\n\" % EXP_OUT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Downloading Experimental Data**  \n",
    "All training and measurement experiments are bundeled into an archive that is downloaded and installed. This gives you access to pre-trained models and all experimental configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-07-23 16:04:06--  http://robotics.ethz.ch/~asl-datasets/2018_modular_semantic_segmentation/experimental_data.tar.gz\n",
      "Resolving robotics.ethz.ch (robotics.ethz.ch)... 129.132.38.186\n",
      "Connecting to robotics.ethz.ch (robotics.ethz.ch)|129.132.38.186|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1588955377 (1.5G) [application/x-gzip]\n",
      "Saving to: ‘/tmp/experimental_data.tar.gz’\n",
      "\n",
      "/tmp/experimental_d 100%[===================>]   1.48G  87.5MB/s    in 31s     \n",
      "\n",
      "2018-07-23 16:04:36 (49.7 MB/s) - ‘/tmp/experimental_data.tar.gz’ saved [1588955377/1588955377]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget http://robotics.ethz.ch/~asl-datasets/2018_modular_semantic_segmentation/experimental_data.tar.gz -O /tmp/experimental_data.tar.gz\n",
    "import tarfile\n",
    "from os import path\n",
    "from xview.settings import EXPERIMENT_STORAGE_FOLDER\n",
    "tar = tarfile.open('/tmp/experimental_data.tar.gz')\n",
    "tar.extractall(path=EXPERIMENT_STORAGE_FOLDER)\n",
    "tar.close()\n",
    "! rm /tmp/experimental_data.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing Details of an Experiment\n",
    "All experiments referenced in other notebooks were recorded using [sacred](http://sacred.readthedocs.io/en/latest/). You can access details in the following ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from experiments.utils import ExperimentData\n",
    "exp = ExperimentData(868)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Configuration**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'evaluation_data': {'batchsize': 4,\n",
       "  'dataset': 'cityscapes_c',\n",
       "  'in_memory': True,\n",
       "  'resize': True},\n",
       " 'net_config': {'expert_model': 'adapnet',\n",
       "  'learning_rate': 0,\n",
       "  'num_channels': {'depth': 1, 'rgb': 3},\n",
       "  'num_classes': 12,\n",
       "  'num_units': 20,\n",
       "  'prefixes': {'depth': 'depth', 'rgb': 'rgb'},\n",
       "  'trainer': 'rmsprop'},\n",
       " 'seed': 453280996,\n",
       " 'starting_weights': {'depth': 861, 'rgb': 860}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp.get_record()['config']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Captured Output**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'INFO - cityC bayes fusion - Running command \\'fit_and_evaluate\\'\\nINFO - cityC bayes fusion - Started run with ID \"868\"\\nINFO loading dataset into memory\\n2018-03-01 11:00:50.202875: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn\\'t compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\\n2018-03-01 11:00:50.202949: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn\\'t compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\\n2018-03-01 11:00:50.202956: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn\\'t compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\\n2018-03-01 11:00:50.202961: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn\\'t compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\\n2018-03-01 11:00:50.202965: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn\\'t compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\\n2018-03-01 11:00:50.701879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: \\nname: GeForce GTX 1080 Ti\\nmajor: 6 minor: 1 memoryClockRate (GHz) 1.582\\npciBusID 0000:04:00.0\\nTotal memory: 10.91GiB\\nFree memory: 10.76GiB\\n2018-03-01 11:00:50.701917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 \\n2018-03-01 11:00:50.701923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y \\n2018-03-01 11:00:50.701932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:04:00.0)\\n2018-03-01 11:00:50.703757: E tensorflow/stream_executor/cuda/cuda_driver.cc:924] failed to allocate 10.91G (11715084288 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY\\nEvaluated network depth on cityscapes_c train data:\\nINFO now getting test results\\ntotal accuracy 0.873 IoU 0.541\\n2018-03-01 11:04:22.671354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:04:00.0)\\nEvaluated network rgb on cityscapes_c train data:\\nINFO now getting test results\\ntotal accuracy 0.924 IoU 0.692\\n2018-03-01 11:07:38.013681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:04:00.0)\\nWARNING: rgb/block_0_1/kernel not found in saved weights\\nWARNING: rgb/block_0_1/bias not found in saved weights\\nWARNING: rgb/block_0_1/beta not found in saved weights\\nWARNING: rgb/block_0_1/gamma not found in saved weights\\nWARNING: rgb/block_0_1/moving_mean not found in saved weights\\nWARNING: rgb/block_0_1/moving_variance not found in saved weights\\nWARNING: rgb/block_0_2/kernel not found in saved weights\\nWARNING: rgb/block_0_2/bias not found in saved weights\\nWARNING: rgb/block_0_2/beta not found in saved weights\\nWARNING: rgb/block_0_2/gamma not found in saved weights\\nWARNING: rgb/block_0_2/moving_mean not found in saved weights\\nWARNING: rgb/block_0_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_1/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_1/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_1/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_1/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_1/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_1/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_1/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_1/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_1/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_1/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_1/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_1/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_1/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_1/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_1/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_1/shortcut/kernel not found in saved weights\\nWARNING: rgb/block_layer_1/shortcut/beta not found in saved weights\\nWARNING: rgb/block_layer_1/shortcut/gamma not found in saved weights\\nWARNING: rgb/block_layer_1/shortcut/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_1/shortcut/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_2/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_2/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_2/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_2/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_2/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_2/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_2/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_2/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_2/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_2/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_2/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_2/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_2/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_2/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_2/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_3/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_3/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_3/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_3/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_3/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_3/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_3/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_3/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_3/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_3/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_3/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_3/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_3/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_3/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_3/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_4/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_4/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_4/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_4/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_4/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_4/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_4/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_4/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_4/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_4/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_4/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_4/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_4/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_4/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_4/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_4/shortcut/kernel not found in saved weights\\nWARNING: rgb/block_layer_4/shortcut/beta not found in saved weights\\nWARNING: rgb/block_layer_4/shortcut/gamma not found in saved weights\\nWARNING: rgb/block_layer_4/shortcut/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_4/shortcut/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_5/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_5/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_5/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_5/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_5/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_5/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_5/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_5/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_5/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_5/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_5/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_5/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_5/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_5/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_5/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_6/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_6/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_6/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_6/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_6/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_6/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_6/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_6/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_6/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_6/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_6/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_6/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_6/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_6/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_6/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_7/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_7/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_7/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_7/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_7/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_7/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_7/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_7/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_7/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_7/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_7/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/shortcut/kernel not found in saved weights\\nWARNING: rgb/shortcut/bias not found in saved weights\\nWARNING: rgb/shortcut/beta not found in saved weights\\nWARNING: rgb/shortcut/gamma not found in saved weights\\nWARNING: rgb/shortcut/moving_mean not found in saved weights\\nWARNING: rgb/shortcut/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_8/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_8/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_8/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_8/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_8/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_8/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_8/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_8/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_8/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_8/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_8/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_8/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_8/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_8/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_8/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_8/shortcut/kernel not found in saved weights\\nWARNING: rgb/block_layer_8/shortcut/beta not found in saved weights\\nWARNING: rgb/block_layer_8/shortcut/gamma not found in saved weights\\nWARNING: rgb/block_layer_8/shortcut/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_8/shortcut/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_9/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_9/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_9/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_9/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_9/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_9/stage_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_9/stage_2/beta not found in saved weights\\nWARNING: rgb/block_layer_9/stage_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_9/stage_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_9/stage_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_9/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_9/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_9/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_9/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_9/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_10/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_10/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_10/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_10/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_10/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_10/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_10/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_10/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_10/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_10/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_10/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_11/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_11/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_11/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_11/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_11/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_11/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_11/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_11/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_11/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_11/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_11/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_12/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_12/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_12/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_12/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_12/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_12/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_12/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_12/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_12/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_12/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_12/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_13/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_13/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_13/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_13/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_13/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_13/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_13/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_13/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_13/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_13/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_13/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_14/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_14/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_14/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_14/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_14/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_14/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_14/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_14/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_14/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_14/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_14/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_14/shortcut/kernel not found in saved weights\\nWARNING: rgb/block_layer_14/shortcut/beta not found in saved weights\\nWARNING: rgb/block_layer_14/shortcut/gamma not found in saved weights\\nWARNING: rgb/block_layer_14/shortcut/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_14/shortcut/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_15/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_15/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_15/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_15/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_15/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_15/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_15/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_15/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_15/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_15/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_15/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_16/stage_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_16/stage_1/beta not found in saved weights\\nWARNING: rgb/block_layer_16/stage_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_16/stage_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_16/stage_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_1/kernel not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_1/beta not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_1/gamma not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_1/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_1/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_2/kernel not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_2/beta not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_2/gamma not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_2/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_16/stage_2_2/moving_variance not found in saved weights\\nWARNING: rgb/block_layer_16/stage_3/kernel not found in saved weights\\nWARNING: rgb/block_layer_16/stage_3/beta not found in saved weights\\nWARNING: rgb/block_layer_16/stage_3/gamma not found in saved weights\\nWARNING: rgb/block_layer_16/stage_3/moving_mean not found in saved weights\\nWARNING: rgb/block_layer_16/stage_3/moving_variance not found in saved weights\\nWARNING: rgb/first_deconvolution_conv/kernel not found in saved weights\\nWARNING: rgb/first_deconvolution_conv/bias not found in saved weights\\nWARNING: rgb/first_deconvolution_conv/beta not found in saved weights\\nWARNING: rgb/first_deconvolution_conv/gamma not found in saved weights\\nWARNING: rgb/first_deconvolution_conv/moving_mean not found in saved weights\\nWARNING: rgb/first_deconvolution_conv/moving_variance not found in saved weights\\nWARNING: rgb/first_deconvolution_upconv/kernel not found in saved weights\\nWARNING: rgb/first_deconvolution_upconv/beta not found in saved weights\\nWARNING: rgb/first_deconvolution_upconv/gamma not found in saved weights\\nWARNING: rgb/first_deconvolution_upconv/moving_mean not found in saved weights\\nWARNING: rgb/first_deconvolution_upconv/moving_variance not found in saved weights\\nWARNING: rgb/second_deconvolution_upconv/kernel not found in saved weights\\nWARNING: rgb/second_deconvolution_upconv/beta not found in saved weights\\nWARNING: rgb/second_deconvolution_upconv/gamma not found in saved weights\\nWARNING: rgb/second_deconvolution_upconv/moving_mean not found in saved weights\\nWARNING: rgb/second_deconvolution_upconv/moving_variance not found in saved weights\\nWARNING: depth/block_0_1/kernel not found in saved weights\\nWARNING: depth/block_0_1/bias not found in saved weights\\nWARNING: depth/block_0_1/beta not found in saved weights\\nWARNING: depth/block_0_1/gamma not found in saved weights\\nWARNING: depth/block_0_1/moving_mean not found in saved weights\\nWARNING: depth/block_0_1/moving_variance not found in saved weights\\nWARNING: depth/block_0_2/kernel not found in saved weights\\nWARNING: depth/block_0_2/bias not found in saved weights\\nWARNING: depth/block_0_2/beta not found in saved weights\\nWARNING: depth/block_0_2/gamma not found in saved weights\\nWARNING: depth/block_0_2/moving_mean not found in saved weights\\nWARNING: depth/block_0_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_1/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_1/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_1/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_1/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_1/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_1/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_1/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_1/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_1/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_1/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_1/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_1/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_1/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_1/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_1/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_1/shortcut/kernel not found in saved weights\\nWARNING: depth/block_layer_1/shortcut/beta not found in saved weights\\nWARNING: depth/block_layer_1/shortcut/gamma not found in saved weights\\nWARNING: depth/block_layer_1/shortcut/moving_mean not found in saved weights\\nWARNING: depth/block_layer_1/shortcut/moving_variance not found in saved weights\\nWARNING: depth/block_layer_2/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_2/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_2/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_2/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_2/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_2/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_2/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_2/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_2/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_2/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_2/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_2/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_2/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_2/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_2/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_3/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_3/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_3/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_3/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_3/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_3/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_3/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_3/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_3/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_3/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_3/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_3/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_3/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_3/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_3/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_4/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_4/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_4/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_4/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_4/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_4/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_4/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_4/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_4/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_4/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_4/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_4/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_4/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_4/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_4/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_4/shortcut/kernel not found in saved weights\\nWARNING: depth/block_layer_4/shortcut/beta not found in saved weights\\nWARNING: depth/block_layer_4/shortcut/gamma not found in saved weights\\nWARNING: depth/block_layer_4/shortcut/moving_mean not found in saved weights\\nWARNING: depth/block_layer_4/shortcut/moving_variance not found in saved weights\\nWARNING: depth/block_layer_5/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_5/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_5/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_5/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_5/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_5/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_5/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_5/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_5/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_5/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_5/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_5/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_5/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_5/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_5/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_6/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_6/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_6/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_6/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_6/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_6/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_6/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_6/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_6/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_6/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_6/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_6/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_6/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_6/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_6/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_7/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_7/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_7/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_7/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_7/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_7/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_7/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_7/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_7/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_7/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_7/stage_3/moving_variance not found in saved weights\\nWARNING: depth/shortcut/kernel not found in saved weights\\nWARNING: depth/shortcut/bias not found in saved weights\\nWARNING: depth/shortcut/beta not found in saved weights\\nWARNING: depth/shortcut/gamma not found in saved weights\\nWARNING: depth/shortcut/moving_mean not found in saved weights\\nWARNING: depth/shortcut/moving_variance not found in saved weights\\nWARNING: depth/block_layer_8/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_8/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_8/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_8/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_8/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_8/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_8/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_8/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_8/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_8/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_8/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_8/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_8/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_8/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_8/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_8/shortcut/kernel not found in saved weights\\nWARNING: depth/block_layer_8/shortcut/beta not found in saved weights\\nWARNING: depth/block_layer_8/shortcut/gamma not found in saved weights\\nWARNING: depth/block_layer_8/shortcut/moving_mean not found in saved weights\\nWARNING: depth/block_layer_8/shortcut/moving_variance not found in saved weights\\nWARNING: depth/block_layer_9/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_9/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_9/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_9/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_9/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_9/stage_2/kernel not found in saved weights\\nWARNING: depth/block_layer_9/stage_2/beta not found in saved weights\\nWARNING: depth/block_layer_9/stage_2/gamma not found in saved weights\\nWARNING: depth/block_layer_9/stage_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_9/stage_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_9/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_9/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_9/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_9/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_9/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_10/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_10/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_10/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_10/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_10/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_10/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_10/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_10/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_10/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_10/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_10/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_11/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_11/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_11/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_11/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_11/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_11/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_11/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_11/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_11/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_11/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_11/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_12/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_12/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_12/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_12/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_12/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_12/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_12/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_12/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_12/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_12/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_12/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_13/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_13/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_13/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_13/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_13/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_13/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_13/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_13/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_13/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_13/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_13/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_14/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_14/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_14/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_14/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_14/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_14/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_14/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_14/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_14/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_14/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_14/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_14/shortcut/kernel not found in saved weights\\nWARNING: depth/block_layer_14/shortcut/beta not found in saved weights\\nWARNING: depth/block_layer_14/shortcut/gamma not found in saved weights\\nWARNING: depth/block_layer_14/shortcut/moving_mean not found in saved weights\\nWARNING: depth/block_layer_14/shortcut/moving_variance not found in saved weights\\nWARNING: depth/block_layer_15/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_15/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_15/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_15/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_15/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_15/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_15/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_15/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_15/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_15/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_15/stage_3/moving_variance not found in saved weights\\nWARNING: depth/block_layer_16/stage_1/kernel not found in saved weights\\nWARNING: depth/block_layer_16/stage_1/beta not found in saved weights\\nWARNING: depth/block_layer_16/stage_1/gamma not found in saved weights\\nWARNING: depth/block_layer_16/stage_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_16/stage_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_1/kernel not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_1/beta not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_1/gamma not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_1/moving_mean not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_1/moving_variance not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_2/kernel not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_2/beta not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_2/gamma not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_2/moving_mean not found in saved weights\\nWARNING: depth/block_layer_16/stage_2_2/moving_variance not found in saved weights\\nWARNING: depth/block_layer_16/stage_3/kernel not found in saved weights\\nWARNING: depth/block_layer_16/stage_3/beta not found in saved weights\\nWARNING: depth/block_layer_16/stage_3/gamma not found in saved weights\\nWARNING: depth/block_layer_16/stage_3/moving_mean not found in saved weights\\nWARNING: depth/block_layer_16/stage_3/moving_variance not found in saved weights\\nWARNING: depth/first_deconvolution_conv/kernel not found in saved weights\\nWARNING: depth/first_deconvolution_conv/bias not found in saved weights\\nWARNING: depth/first_deconvolution_conv/beta not found in saved weights\\nWARNING: depth/first_deconvolution_conv/gamma not found in saved weights\\nWARNING: depth/first_deconvolution_conv/moving_mean not found in saved weights\\nWARNING: depth/first_deconvolution_conv/moving_variance not found in saved weights\\nWARNING: depth/first_deconvolution_upconv/kernel not found in saved weights\\nWARNING: depth/first_deconvolution_upconv/beta not found in saved weights\\nWARNING: depth/first_deconvolution_upconv/gamma not found in saved weights\\nWARNING: depth/first_deconvolution_upconv/moving_mean not found in saved weights\\nWARNING: depth/first_deconvolution_upconv/moving_variance not found in saved weights\\nWARNING: depth/second_deconvolution_upconv/kernel not found in saved weights\\nWARNING: depth/second_deconvolution_upconv/beta not found in saved weights\\nWARNING: depth/second_deconvolution_upconv/gamma not found in saved weights\\nWARNING: depth/second_deconvolution_upconv/moving_mean not found in saved weights\\nWARNING: depth/second_deconvolution_upconv/moving_variance not found in saved weights\\nEvaluated Bayes Fusion on cityscapes_c data:\\ntotal accuracy 0.920 IoU 0.688\\nINFO - cityC bayes fusion - Completed after 0:17:53\\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp.get_record()['captured_out']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Results**  \n",
    "Results are stored in a machine-readable form. For more documentation of which metric is stored where, please have a look at the corresponding [experimental script](https://github.com/ethz-asl/modular_semantic_segmentation/tree/publish/experiments). Which file and command were used can be accessed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file: experiments/bayes_fusion.py, function: fit_and_evaluate\n"
     ]
    }
   ],
   "source": [
    "print('file: experiments/{}, function: {}'.format(\n",
    "    exp.get_record()['experiment']['mainfile'], exp.get_record()['meta']['command']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['confusion_matrices', 'confusion_matrix', 'measurements'])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'confusion_matrices': {'depth': array([[3.5048800e+06, 1.7594000e+04, 2.3711300e+05, 4.4469600e+05,\n",
       "          1.2071100e+05, 1.7732000e+04, 2.8964900e+05, 3.1910000e+04,\n",
       "          5.5801000e+04, 2.9157000e+04, 2.6291000e+04, 1.2909000e+04],\n",
       "         [3.0920000e+03, 1.4245360e+06, 6.6781000e+04, 0.0000000e+00,\n",
       "          0.0000000e+00, 6.6000000e+01, 9.0863000e+04, 5.3550000e+03,\n",
       "          2.1200000e+02, 3.0100000e+02, 1.1000000e+01, 0.0000000e+00],\n",
       "         [1.0437800e+05, 6.9574000e+04, 7.3386870e+06, 4.8310000e+03,\n",
       "          3.8904000e+04, 4.9772000e+04, 1.0295860e+06, 3.5637000e+04,\n",
       "          5.2496000e+04, 1.1867000e+04, 3.4657000e+04, 4.0990000e+03],\n",
       "         [1.1955900e+05, 1.6000000e+01, 5.9970000e+03, 1.3999703e+07,\n",
       "          1.2126400e+05, 1.4010000e+03, 3.6060000e+04, 1.9580000e+03,\n",
       "          6.2925000e+04, 7.2000000e+02, 1.1658000e+04, 1.5760000e+03],\n",
       "         [9.7751000e+04, 0.0000000e+00, 1.7849000e+04, 3.0471700e+05,\n",
       "          1.9612140e+06, 6.2880000e+03, 8.2734000e+04, 8.2530000e+03,\n",
       "          9.2890000e+03, 4.6900000e+02, 1.1756000e+04, 3.6820000e+03],\n",
       "         [1.7488000e+04, 3.5000000e+01, 5.2790000e+04, 2.1500000e+03,\n",
       "          1.3752000e+04, 1.6585000e+05, 1.1904700e+05, 4.7570000e+03,\n",
       "          1.8078000e+04, 3.0800000e+02, 8.0940000e+03, 2.4850000e+03],\n",
       "         [8.5145000e+04, 4.6415000e+04, 5.1791000e+05, 4.3733000e+04,\n",
       "          1.2617800e+05, 2.6928000e+04, 6.3768510e+06, 4.1267000e+04,\n",
       "          5.6902000e+04, 8.8780000e+03, 1.1593000e+04, 3.0480000e+03],\n",
       "         [2.3693000e+04, 8.8190000e+03, 7.9714000e+04, 3.9760000e+03,\n",
       "          1.2875000e+04, 3.6170000e+03, 1.1165200e+05, 1.9575800e+05,\n",
       "          8.1420000e+03, 8.6730000e+03, 7.4770000e+03, 1.0900000e+03],\n",
       "         [1.5444000e+04, 1.3290000e+03, 5.5522000e+04, 5.6918000e+04,\n",
       "          1.0847000e+04, 5.9890000e+03, 1.1242900e+05, 4.7370000e+03,\n",
       "          2.3876440e+06, 7.2000000e+02, 3.7941000e+04, 9.7700000e+03],\n",
       "         [2.7220000e+04, 4.0500000e+02, 2.6082000e+04, 3.3430000e+03,\n",
       "          1.0600000e+03, 2.1300000e+02, 4.5954000e+04, 6.6940000e+03,\n",
       "          6.1870000e+03, 4.5684000e+04, 1.2519000e+04, 6.7000000e+02],\n",
       "         [2.4397000e+04, 6.0000000e+00, 4.0033000e+04, 1.3443000e+04,\n",
       "          1.6044000e+04, 5.1310000e+03, 4.8186000e+04, 5.5580000e+03,\n",
       "          3.7473000e+04, 2.2220000e+03, 4.2063900e+05, 7.4330000e+03],\n",
       "         [2.4769000e+04, 0.0000000e+00, 2.1527000e+04, 7.7690000e+03,\n",
       "          1.6474000e+04, 8.6860000e+03, 1.7293000e+04, 3.9160000e+03,\n",
       "          2.7734000e+04, 5.4000000e+01, 2.3293000e+04, 5.8332000e+04]]),\n",
       "  'rgb': array([[3.7824380e+06, 8.3440000e+03, 3.1672300e+05, 2.9147000e+05,\n",
       "          1.2518500e+05, 1.4142000e+04, 1.0825600e+05, 3.9004000e+04,\n",
       "          4.8277000e+04, 2.2050000e+04, 2.1755000e+04, 1.0799000e+04],\n",
       "         [2.5400000e+03, 1.5397690e+06, 1.6027000e+04, 0.0000000e+00,\n",
       "          0.0000000e+00, 1.6600000e+02, 2.8737000e+04, 3.2440000e+03,\n",
       "          5.6500000e+02, 1.0300000e+02, 6.6000000e+01, 0.0000000e+00],\n",
       "         [1.4783600e+05, 4.5554000e+04, 8.0478060e+06, 6.2250000e+03,\n",
       "          3.4989000e+04, 6.0497000e+04, 2.0184100e+05, 4.8140000e+04,\n",
       "          1.2833600e+05, 1.8056000e+04, 2.7784000e+04, 7.4240000e+03],\n",
       "         [1.3754300e+05, 0.0000000e+00, 6.3130000e+03, 1.4035546e+07,\n",
       "          1.2000300e+05, 1.0930000e+03, 1.1722000e+04, 1.8210000e+03,\n",
       "          3.7967000e+04, 6.9200000e+02, 9.0040000e+03, 1.1330000e+03],\n",
       "         [1.7900900e+05, 0.0000000e+00, 2.5386000e+04, 2.1606200e+05,\n",
       "          1.9990370e+06, 4.4010000e+03, 3.6454000e+04, 7.2170000e+03,\n",
       "          2.1520000e+04, 2.2100000e+02, 8.9500000e+03, 5.7450000e+03],\n",
       "         [2.2772000e+04, 1.4000000e+01, 8.7697000e+04, 6.6300000e+02,\n",
       "          8.2160000e+03, 2.4242100e+05, 2.6272000e+04, 4.4430000e+03,\n",
       "          5.1630000e+03, 9.5100000e+02, 3.1560000e+03, 3.0660000e+03],\n",
       "         [5.0720000e+04, 2.3791000e+04, 1.9366500e+05, 2.5504000e+04,\n",
       "          3.1481000e+04, 1.7188000e+04, 6.9348670e+06, 2.7573000e+04,\n",
       "          2.5737000e+04, 4.3540000e+03, 7.0240000e+03, 2.9440000e+03],\n",
       "         [2.3502000e+04, 6.8200000e+03, 9.5289000e+04, 1.9350000e+03,\n",
       "          8.1470000e+03, 5.2920000e+03, 4.7315000e+04, 2.5502100e+05,\n",
       "          7.6880000e+03, 4.6630000e+03, 6.4520000e+03, 3.3620000e+03],\n",
       "         [1.6546000e+04, 3.2500000e+02, 3.9782000e+04, 2.9349000e+04,\n",
       "          3.3840000e+03, 4.8330000e+03, 2.0614000e+04, 4.2120000e+03,\n",
       "          2.5554390e+06, 1.3450000e+03, 1.4782000e+04, 8.6790000e+03],\n",
       "         [8.6790000e+03, 6.4000000e+01, 2.6148000e+04, 1.1360000e+03,\n",
       "          1.6400000e+02, 5.8500000e+02, 8.1130000e+03, 3.3270000e+03,\n",
       "          4.8370000e+03, 1.2185600e+05, 1.1110000e+03, 1.1000000e+01],\n",
       "         [1.5197000e+04, 2.0000000e+01, 3.4517000e+04, 1.2075000e+04,\n",
       "          9.3650000e+03, 2.1970000e+03, 1.2511000e+04, 3.6270000e+03,\n",
       "          1.8633000e+04, 9.3400000e+02, 5.0311000e+05, 8.3790000e+03],\n",
       "         [7.6090000e+03, 0.0000000e+00, 1.1629000e+04, 2.4170000e+03,\n",
       "          6.6310000e+03, 1.1380000e+03, 4.4590000e+03, 1.8850000e+03,\n",
       "          7.7380000e+03, 8.1000000e+01, 1.6452000e+04, 1.4980800e+05]])},\n",
       " 'confusion_matrix': array([[1.3308637e+07, 7.3978000e+04, 9.9948800e+05, 1.3221280e+06,\n",
       "         7.6665500e+05, 9.3593000e+04, 3.5487200e+05, 1.1003000e+05,\n",
       "         1.6913600e+05, 7.0348000e+04, 9.4801000e+04, 7.7104000e+04],\n",
       "        [1.4515000e+04, 4.0472110e+06, 6.8578000e+04, 4.0000000e+00,\n",
       "         0.0000000e+00, 6.2000000e+01, 7.1474000e+04, 7.9600000e+03,\n",
       "         1.0830000e+03, 1.3930000e+03, 1.9000000e+01, 0.0000000e+00],\n",
       "        [7.5202100e+05, 1.0397300e+05, 2.6032379e+07, 1.1156000e+04,\n",
       "         1.1512600e+05, 6.1214000e+04, 7.4011300e+05, 2.6632200e+05,\n",
       "         7.0848000e+04, 4.8759000e+04, 1.0204400e+05, 3.7641000e+04],\n",
       "        [1.1500190e+06, 1.2300000e+03, 8.3120000e+03, 4.5505469e+07,\n",
       "         8.3306300e+05, 4.8940000e+03, 2.2687000e+04, 5.2430000e+03,\n",
       "         1.3113900e+05, 2.3730000e+03, 3.5163000e+04, 1.1246000e+04],\n",
       "        [3.4053400e+05, 4.0000000e+00, 8.9095000e+04, 3.9571800e+05,\n",
       "         5.7158730e+06, 2.0669000e+04, 6.7300000e+04, 3.1542000e+04,\n",
       "         2.4526000e+04, 9.6800000e+02, 3.2093000e+04, 2.8145000e+04],\n",
       "        [1.0581800e+05, 9.0000000e+00, 2.5487900e+05, 4.7430000e+03,\n",
       "         2.8495000e+04, 5.0000400e+05, 5.9192000e+04, 1.6426000e+04,\n",
       "         3.9833000e+04, 4.3120000e+03, 9.4840000e+03, 1.7896000e+04],\n",
       "        [1.6251700e+05, 6.4317000e+04, 5.7027800e+05, 4.0420000e+04,\n",
       "         1.5050200e+05, 3.9479000e+04, 2.1511475e+07, 9.9169000e+04,\n",
       "         4.9161000e+04, 2.2332000e+04, 2.3735000e+04, 1.8362000e+04],\n",
       "        [1.9284500e+05, 1.4998000e+04, 2.3502400e+05, 4.8950000e+03,\n",
       "         5.4749000e+04, 1.7954000e+04, 2.0329600e+05, 1.0279260e+06,\n",
       "         2.6752000e+04, 1.5144000e+04, 3.1326000e+04, 1.9247000e+04],\n",
       "        [1.4565400e+05, 1.7970000e+03, 3.0439300e+05, 9.3030000e+04,\n",
       "         4.7208000e+04, 1.0906000e+04, 6.4690000e+04, 1.8490000e+04,\n",
       "         8.5702100e+06, 7.6380000e+03, 4.0911000e+04, 3.0588000e+04],\n",
       "        [8.5660000e+04, 1.7160000e+03, 1.1079000e+05, 4.3770000e+03,\n",
       "         3.4590000e+03, 1.7811000e+04, 4.3809000e+04, 2.0171000e+04,\n",
       "         1.0761000e+04, 5.2151400e+05, 1.7266000e+04, 1.4320000e+03],\n",
       "        [5.8344000e+04, 1.8300000e+02, 1.2169100e+05, 2.3988000e+04,\n",
       "         2.6711000e+04, 8.0710000e+03, 3.7608000e+04, 1.4042000e+04,\n",
       "         5.5667000e+04, 2.7820000e+03, 1.4846210e+06, 5.5634000e+04],\n",
       "        [3.1767000e+04, 0.0000000e+00, 3.8174000e+04, 9.3840000e+03,\n",
       "         1.9418000e+04, 2.8090000e+03, 1.7197000e+04, 7.3560000e+03,\n",
       "         2.7283000e+04, 3.6700000e+02, 6.0431000e+04, 6.6554700e+05]]),\n",
       " 'measurements': {'depth': {'F1': array([0.79349162, 0.88020529, 0.84281192, 0.95720035, 0.73083095,\n",
       "          0.38209789, 0.79921726, 0.49938325, 0.87591671, 0.31719918,\n",
       "          0.68657824, 0.36064467]),\n",
       "   'IoU': array([0.65767601, 0.78604166, 0.72832751, 0.91791395, 0.57583421,\n",
       "          0.23616873, 0.66558024, 0.33278533, 0.77922759, 0.18849479,\n",
       "          0.52274011, 0.21999176]),\n",
       "   'confusion_matrix': array([[1.2386747e+07, 9.4195000e+04, 8.5439900e+05, 1.6906100e+06,\n",
       "           6.9409300e+05, 6.8570000e+04, 1.0384780e+06, 1.0120800e+05,\n",
       "           1.7661300e+05, 1.2214000e+05, 1.1089700e+05, 1.0282000e+05],\n",
       "          [9.0040000e+03, 3.8391320e+06, 1.5783300e+05, 1.3000000e+01,\n",
       "           2.0000000e+01, 1.8000000e+02, 1.9154900e+05, 1.1208000e+04,\n",
       "           8.4500000e+02, 2.4460000e+03, 6.9000000e+01, 0.0000000e+00],\n",
       "          [3.1533900e+05, 3.9219200e+05, 2.3798033e+07, 1.8348000e+04,\n",
       "           1.2356200e+05, 1.6781600e+05, 2.9939100e+06, 1.4126600e+05,\n",
       "           1.9992200e+05, 4.7704000e+04, 1.1697200e+05, 2.6532000e+04],\n",
       "          [1.4171300e+05, 7.1200000e+02, 1.0682000e+04, 4.6620440e+07,\n",
       "           6.1256200e+05, 8.0030000e+03, 7.9091000e+04, 9.9540000e+03,\n",
       "           1.6189200e+05, 8.0500000e+02, 5.5517000e+04, 9.4670000e+03],\n",
       "          [1.7305800e+05, 0.0000000e+00, 6.1928000e+04, 9.5153400e+05,\n",
       "           5.0685520e+06, 2.1284000e+04, 3.1151900e+05, 3.8744000e+04,\n",
       "           5.3213000e+04, 3.9500000e+02, 4.6135000e+04, 2.0105000e+04],\n",
       "          [4.5217000e+04, 6.0000000e+00, 1.5181800e+05, 1.1143000e+04,\n",
       "           5.3801000e+04, 3.5915100e+05, 2.3597400e+05, 1.7455000e+04,\n",
       "           1.0126800e+05, 2.7570000e+03, 2.5913000e+04, 3.6588000e+04],\n",
       "          [2.5192300e+05, 1.4323600e+05, 2.1788590e+06, 1.0535600e+05,\n",
       "           3.5252600e+05, 9.7758000e+04, 1.9123112e+07, 1.1970200e+05,\n",
       "           2.4153000e+05, 3.4968000e+04, 5.8198000e+04, 4.4579000e+04],\n",
       "          [1.1489200e+05, 2.9934000e+04, 2.9688100e+05, 1.8564000e+04,\n",
       "           7.6329000e+04, 2.2817000e+04, 3.7293400e+05, 7.9390700e+05,\n",
       "           3.5038000e+04, 3.3347000e+04, 3.6490000e+04, 1.3023000e+04],\n",
       "          [8.7410000e+04, 1.2720000e+03, 2.3630000e+05, 1.9508600e+05,\n",
       "           4.3480000e+04, 4.2105000e+04, 3.7111100e+05, 2.4155000e+04,\n",
       "           8.2126310e+06, 4.0910000e+03, 7.0685000e+04, 4.7189000e+04],\n",
       "          [1.4005900e+05, 1.0210000e+04, 1.2878700e+05, 1.3146000e+04,\n",
       "           7.5790000e+03, 1.0467000e+04, 2.0207700e+05, 4.3938000e+04,\n",
       "           2.3863000e+04, 2.0611000e+05, 4.8541000e+04, 3.9890000e+03],\n",
       "          [5.7839000e+04, 7.7000000e+01, 1.4222900e+05, 3.9582000e+04,\n",
       "           4.3755000e+04, 1.6749000e+04, 9.4086000e+04, 1.7464000e+04,\n",
       "           9.5468000e+04, 5.1230000e+03, 1.3448540e+06, 3.2116000e+04],\n",
       "          [5.6893000e+04, 0.0000000e+00, 1.1359400e+05, 3.5334000e+04,\n",
       "           4.7929000e+04, 2.3899000e+04, 8.9014000e+04, 1.6393000e+04,\n",
       "           1.1428400e+05, 9.1000000e+02, 1.1394200e+05, 2.6754100e+05]]),\n",
       "   'mean_F1': 0.6771314431391106,\n",
       "   'mean_IoU': 0.5411914429519398,\n",
       "   'precision': array([0.89888697, 0.85106649, 0.8459615 , 0.93805295, 0.7114568 ,\n",
       "          0.4281729 , 0.76179032, 0.59451143, 0.87214704, 0.44729121,\n",
       "          0.66307336, 0.44298608]),\n",
       "   'recall': array([0.7102179 , 0.91141013, 0.8396857 , 0.9771457 , 0.75128982,\n",
       "          0.34497561, 0.84051181, 0.43049883, 0.87971912, 0.24573004,\n",
       "          0.71181078, 0.30411614]),\n",
       "   'total_accuracy': 0.872936618745449},\n",
       "  'fusion': {'F1': array([0.78774733, 0.94985833, 0.91062619, 0.95673942, 0.78797642,\n",
       "          0.54989093, 0.93639176, 0.59266387, 0.9259129 , 0.67874713,\n",
       "          0.77703706, 0.72240967]),\n",
       "   'IoU': array([0.64982107, 0.90450495, 0.83591709, 0.9170666 , 0.65013291,\n",
       "          0.3792066 , 0.88039159, 0.42112461, 0.86204639, 0.51371478,\n",
       "          0.63537254, 0.56544704]),\n",
       "   'confusion_matrix': {'py/id': 2},\n",
       "   'mean_F1': 0.7980000845934097,\n",
       "   'mean_IoU': 0.6877204624612501,\n",
       "   'precision': array([0.81406701, 0.93915533, 0.90286498, 0.95972097, 0.73646209,\n",
       "          0.64312008, 0.92747009, 0.63269561, 0.93394043, 0.74722966,\n",
       "          0.76847953, 0.69123179]),\n",
       "   'recall': array([0.76307623, 0.9608081 , 0.91852198, 0.95377635, 0.84723945,\n",
       "          0.48026926, 0.94548674, 0.55739645, 0.9180222 , 0.6217634 ,\n",
       "          0.78578733, 0.75653295]),\n",
       "   'total_accuracy': 0.920302592013555},\n",
       "  'rgb': {'F1': array([0.7933971 , 0.95033181, 0.9136032 , 0.95576611, 0.79821948,\n",
       "          0.56981706, 0.93639198, 0.59095372, 0.9278162 , 0.68520577,\n",
       "          0.77702714, 0.72265355]),\n",
       "   'IoU': array([0.65754616, 0.90536401, 0.84094798, 0.91527973, 0.66419739,\n",
       "          0.3984225 , 0.88039198, 0.4193998 , 0.86535182, 0.52115058,\n",
       "          0.63535927, 0.56574592]),\n",
       "   'confusion_matrix': array([[1.3215247e+07, 7.6386000e+04, 1.1078640e+06, 1.5862030e+06,\n",
       "           3.7955800e+05, 1.0798300e+05, 3.5487200e+05, 1.0706500e+05,\n",
       "           2.6372700e+05, 6.9788000e+04, 9.4805000e+04, 7.7272000e+04],\n",
       "          [6.6500000e+03, 4.0552440e+06, 6.8980000e+04, 4.0000000e+00,\n",
       "           0.0000000e+00, 1.3700000e+02, 7.1457000e+04, 7.8890000e+03,\n",
       "           5.6200000e+02, 1.3540000e+03, 2.2000000e+01, 0.0000000e+00],\n",
       "          [5.0751400e+05, 1.0440900e+05, 2.6198584e+07, 2.2234000e+04,\n",
       "           1.3009600e+05, 1.0095200e+05, 7.4011300e+05, 1.7310100e+05,\n",
       "           1.7641700e+05, 4.8409000e+04, 1.0211000e+05, 3.7657000e+04],\n",
       "          [1.0708350e+06, 1.2440000e+03, 7.8080000e+03, 4.5998527e+07,\n",
       "           4.0587800e+05, 9.5390000e+03, 2.2687000e+04, 7.8260000e+03,\n",
       "           1.3764400e+05, 2.4340000e+03, 3.5163000e+04, 1.1253000e+04],\n",
       "          [4.8872700e+05, 7.4000000e+01, 6.2074000e+04, 7.2256900e+05,\n",
       "           5.2498660e+06, 2.2518000e+04, 6.7300000e+04, 3.3574000e+04,\n",
       "           3.8562000e+04, 9.6300000e+02, 3.2093000e+04, 2.8147000e+04],\n",
       "          [9.3483000e+04, 9.0000000e+00, 2.0354200e+05, 7.8400000e+03,\n",
       "           2.2601000e+04, 5.5266300e+05, 5.9192000e+04, 1.5120000e+04,\n",
       "           5.3843000e+04, 5.4170000e+03, 9.4840000e+03, 1.7897000e+04],\n",
       "          [1.5817100e+05, 6.4664000e+04, 5.7450500e+05, 5.5837000e+04,\n",
       "           1.1957100e+05, 4.6241000e+04, 2.1511466e+07, 9.4839000e+04,\n",
       "           6.2040000e+04, 2.2288000e+04, 2.3758000e+04, 1.8367000e+04],\n",
       "          [1.1615000e+05, 1.5418000e+04, 3.5155000e+05, 9.5260000e+03,\n",
       "           4.8116000e+04, 2.0080000e+04, 2.0329600e+05, 9.8162200e+05,\n",
       "           3.2573000e+04, 1.5192000e+04, 3.1326000e+04, 1.9307000e+04],\n",
       "          [7.4142000e+04, 2.5340000e+03, 1.4279200e+05, 9.7262000e+04,\n",
       "           1.1167000e+04, 1.3944000e+04, 6.4690000e+04, 1.7956000e+04,\n",
       "           8.8346320e+06, 4.8900000e+03, 4.0911000e+04, 3.0595000e+04],\n",
       "          [6.3519000e+04, 1.8910000e+03, 1.3231300e+05, 6.1120000e+03,\n",
       "           1.3030000e+03, 1.1910000e+04, 4.3805000e+04, 1.7564000e+04,\n",
       "           1.3983000e+04, 5.2766600e+05, 1.7266000e+04, 1.4340000e+03],\n",
       "          [5.0167000e+04, 2.0300000e+02, 1.2462400e+05, 2.6209000e+04,\n",
       "           2.2839000e+04, 8.6290000e+03, 3.7608000e+04, 1.3826000e+04,\n",
       "           6.2268000e+04, 2.6370000e+03, 1.4846510e+06, 5.5681000e+04],\n",
       "          [2.7696000e+04, 0.0000000e+00, 3.5984000e+04, 1.1616000e+04,\n",
       "           1.6479000e+04, 4.1040000e+03, 1.7197000e+04, 7.6240000e+03,\n",
       "           3.2161000e+04, 3.6400000e+02, 6.0431000e+04, 6.6607700e+05]]),\n",
       "   'mean_F1': 0.8017652599652513,\n",
       "   'mean_IoU': 0.6919646343907639,\n",
       "   'precision': array([0.83259806, 0.938263  , 0.90306874, 0.94756478, 0.81933473,\n",
       "          0.61495827, 0.9274709 , 0.66415292, 0.90999764, 0.75230182,\n",
       "          0.76844494, 0.69117566]),\n",
       "   'recall': array([0.75772153, 0.96271513, 0.92438633, 0.96411065, 0.77816522,\n",
       "          0.53084985, 0.94548634, 0.53228794, 0.94634651, 0.62909798,\n",
       "          0.78580321, 0.7571354 ]),\n",
       "   'total_accuracy': 0.9241147035767932}}}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(exp.get_record()['info'].keys())\n",
    "exp.get_record()['info']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}