tfvos/fstream_offline_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `FStream` Offline Training\n",
    "\n",
    "This notebook performs offline training of the **flow stream** on the **DAVIS 2016** dataset.\n",
    "\n",
    "MaskRNN's' binary segmentation net is a 2-stream convnet (`astream` and `fstream`):\n",
    "\n",
    "![](img/maskrnn.png)\n",
    "\n",
    "Section \"3.3 Binary Segmentation\" of the MaskRNN paper and \"Figure 2\" are inconsistent when it comes to describing the inputs of the two-stream network. In this implementation, we chose the input of the flow stream `fstream` to be the concatenation of the magnitude of the flow field from <sub>t-1</sub> to I<sub>t</sub> and I<sub>t</sub> to frame I<sub>t+1</sub> and the warped prediction of the previous frame's segmentation mask b<sub>t-1</sub>, denoted as φ<sub>t-1,t</sub>(b<sub>t-1</sub>). The warping function φ<sub>t-1,t</sub>(.) transforms the input based on the optical flow fields from frame I<sub>t-1</sub> to frame I<sub>t</sub>. The `FStream` network takes 3-channel inputs (||φ<sub>t-1,t</sub>||, ||φ<sub>t,t+1</sub>||, φ<sub>t-1,t</sub>(b<sub>t-1</sub>)):\n",
    "\n",
    "The offline training of the `AStream` is done using a **3-channel input** VGG16 network pre-trained on ImageNet:\n",
    "\n",
    "![](img/osvos_parent.png)\n",
    "\n",
    "\n",
    "To monitor training, run:\n",
    "```\n",
    "tensorboard --logdir E:\\repos\\tf-video-seg\\tfvos\\models\\fstream_parent\n",
    "http://localhost:6006\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "fstream_offline_training.ipynb\n",
    "\n",
    "FStream offline trainer\n",
    "\n",
    "Written by Phil Ferriere\n",
    "\n",
    "Licensed under the MIT License (see LICENSE for details)\n",
    "\n",
    "Based on:\n",
    "  - https://github.com/scaelles/OSVOS-TensorFlow/blob/master/osvos_parent_demo.py\n",
    "    Written by Sergi Caelles (scaelles@vision.ee.ethz.ch)\n",
    "    This file is part of the OSVOS paper presented in:\n",
    "      Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixe, Daniel Cremers, Luc Van Gool\n",
    "      One-Shot Video Object Segmentation\n",
    "      CVPR 2017\n",
    "    Unknown code license\n",
    "\"\"\"\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import os, sys\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "slim = tf.contrib.slim\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import model files\n",
    "import model\n",
    "import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Model paths\n",
    "imagenet_ckpt = 'models/vgg_16_3chan.ckpt' # copy of checkpoint in http://download.tensorflow.org/models/vgg_16_2016_08_28.tar.gz\n",
    "segnet_stream = 'fstream'\n",
    "ckpt_name= segnet_stream + '_parent'\n",
    "logs_path = os.path.join('models', ckpt_name)\n",
    "\n",
    "# Offline training parameters\n",
    "gpu_id = 0\n",
    "iter_mean_grad = 10\n",
    "max_training_iters_1 = 15000\n",
    "max_training_iters_2 = 30000\n",
    "max_training_iters_3 = 50000\n",
    "save_step = 5000\n",
    "test_image = None\n",
    "display_step = 100\n",
    "ini_learning_rate = 1e-8\n",
    "boundaries = [10000, 15000, 25000, 30000, 40000]\n",
    "values = [ini_learning_rate, ini_learning_rate * 0.1, ini_learning_rate, ini_learning_rate * 0.1, ini_learning_rate,\n",
    "          ini_learning_rate * 0.1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing dataset...\n",
      "E:/datasets/davis2016/ImageSets/480p/train.txt\n",
      "Cache files:\n",
      "   videos container: E:/datasets/davis2016/davis2016_videos_train.npy\n",
      "   video_frame_idx container: E:/datasets/davis2016/davis2016_video_frame_idx_train.npy\n",
      "   images_train container: E:/datasets/davis2016/davis2016_images_train.npy\n",
      "   images_train_path container: E:/datasets/davis2016/davis2016_images_train_path.npy\n",
      "   masks_train container: E:/datasets/davis2016/davis2016_masks_train.npy\n",
      "   masks_train_path container: E:/datasets/davis2016/davis2016_masks_train_path.npy\n",
      "   flow_norms_train container: E:/datasets/davis2016/davis2016_flow_norms_train.npy\n",
      "   warped_prev_masks_train container: E:/datasets/davis2016/davis2016_warped_prev_masks_train.npy\n",
      "   masks_bboxes_train container: E:/datasets/davis2016/davis2016_masks_bboxes_train.npy\n",
      "Loading ndarrays from cache...\n",
      " videos container... done.\n",
      " video_frame_idx container... done.\n",
      " images_train container... done.\n",
      " images_train_path container... done.\n",
      " masks_train container... done.\n",
      " masks_train_path container... done.\n",
      " flow_norms_train container... done.\n",
      " warped_prev_masks_train container... done.\n",
      " masks_bboxes_train container... done.\n",
      "...done loading from cache.\n",
      "...done initializing Dataset\n"
     ]
    }
   ],
   "source": [
    "# Load DAVIS 2016 dataset\n",
    "options = datasets._DEFAULT_DAVIS16_OPTIONS\n",
    "# Set the following to True if you have a lot of RAM\n",
    "options['data_aug'] = False\n",
    "# Set the following to wherever you have downloaded the DAVIS 2016 dataset\n",
    "dataset_root = 'E:/datasets/davis2016/' if sys.platform.startswith(\"win\") else '/media/EDrive/datasets/davis2016/'\n",
    "train_file = dataset_root + 'ImageSets/480p/train.txt'\n",
    "dataset = datasets.davis16(train_file, None, dataset_root, options)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Configuration:\n",
      "  in_memory            True\n",
      "  data_aug             False\n",
      "  use_cache            True\n",
      "  use_optical_flow     True\n",
      "  use_warped_masks     True\n",
      "  use_bboxes           True\n",
      "  optical_flow_mgr     pyflow\n"
     ]
    }
   ],
   "source": [
    "# Display dataset configuration\n",
    "dataset.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Offline Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Layers:\n",
      "   name = fstream/conv1/conv1_1/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv1/conv1_2/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/pool1/MaxPool:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv2/conv2_1/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv2/conv2_2/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/pool2/MaxPool:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv3/conv3_1/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_2/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_3/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/pool3/MaxPool:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv4/conv4_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/pool4/MaxPool:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv2_2_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv3_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv4_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv5_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-dsn_2/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_2-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_1:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_2:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_3:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-multi2-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_4:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi3-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_5:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi4-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_6:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi5-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_7:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/upscore-fuse/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "Network Parameters:\n",
      "   name = fstream/conv1/conv1_1/weights:0, shape = (3, 3, 3, 64)\n",
      "   name = fstream/conv1/conv1_1/biases:0, shape = (64,)\n",
      "   name = fstream/conv1/conv1_2/weights:0, shape = (3, 3, 64, 64)\n",
      "   name = fstream/conv1/conv1_2/biases:0, shape = (64,)\n",
      "   name = fstream/conv2/conv2_1/weights:0, shape = (3, 3, 64, 128)\n",
      "   name = fstream/conv2/conv2_1/biases:0, shape = (128,)\n",
      "   name = fstream/conv2/conv2_2/weights:0, shape = (3, 3, 128, 128)\n",
      "   name = fstream/conv2/conv2_2/biases:0, shape = (128,)\n",
      "   name = fstream/conv3/conv3_1/weights:0, shape = (3, 3, 128, 256)\n",
      "   name = fstream/conv3/conv3_1/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_2/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_2/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_3/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_3/biases:0, shape = (256,)\n",
      "   name = fstream/conv4/conv4_1/weights:0, shape = (3, 3, 256, 512)\n",
      "   name = fstream/conv4/conv4_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_1/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv2_2_16/weights:0, shape = (3, 3, 128, 16)\n",
      "   name = fstream/conv2_2_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv3_3_16/weights:0, shape = (3, 3, 256, 16)\n",
      "   name = fstream/conv3_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv4_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv4_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv5_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv5_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/score-dsn_2/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_2/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_3/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_3/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_4/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_4/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_5/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_5/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_2-up/weights:0, shape = (4, 4, 1, 1)\n",
      "   name = fstream/score-dsn_3-up/weights:0, shape = (8, 8, 1, 1)\n",
      "   name = fstream/score-dsn_4-up/weights:0, shape = (16, 16, 1, 1)\n",
      "   name = fstream/score-dsn_5-up/weights:0, shape = (32, 32, 1, 1)\n",
      "   name = fstream/score-multi2-up/weights:0, shape = (4, 4, 16, 16)\n",
      "   name = fstream/score-multi3-up/weights:0, shape = (8, 8, 16, 16)\n",
      "   name = fstream/score-multi4-up/weights:0, shape = (16, 16, 16, 16)\n",
      "   name = fstream/score-multi5-up/weights:0, shape = (32, 32, 16, 16)\n",
      "   name = fstream/upscore-fuse/weights:0, shape = (1, 1, 64, 1)\n",
      "   name = fstream/upscore-fuse/biases:0, shape = (1,)\n",
      "Init variable\n",
      "Initializing from pre-trained imagenet model...\n",
      "INFO:tensorflow:Restoring parameters from models/vgg_16_3chan.ckpt\n",
      "Weights initialized\n",
      "Start training\n",
      "2018-01-30 23:45:59.082977 Iter 100: Training Loss = 21670.6406\n",
      "2018-01-30 23:51:43.965694 Iter 200: Training Loss = 31604.1562\n",
      "2018-01-30 23:57:28.466026 Iter 300: Training Loss = 4770.3442\n",
      "2018-01-31 00:03:12.656227 Iter 400: Training Loss = 46678.4766\n",
      "2018-01-31 00:08:56.760468 Iter 500: Training Loss = 20255.7227\n",
      "2018-01-31 00:14:40.859251 Iter 600: Training Loss = 12546.0723\n",
      "2018-01-31 00:20:23.616584 Iter 700: Training Loss = 11494.8213\n",
      "2018-01-31 00:26:08.081618 Iter 800: Training Loss = 14285.2002\n",
      "2018-01-31 00:31:51.831223 Iter 900: Training Loss = 26010.9609\n",
      "2018-01-31 00:37:34.597122 Iter 1000: Training Loss = 17008.3711\n",
      "2018-01-31 00:43:16.577418 Iter 1100: Training Loss = 46729.0742\n",
      "2018-01-31 00:48:58.953378 Iter 1200: Training Loss = 255745.2500\n",
      "2018-01-31 00:54:40.356550 Iter 1300: Training Loss = 2812.5693\n",
      "2018-01-31 01:00:21.594282 Iter 1400: Training Loss = 39330.1758\n",
      "2018-01-31 01:06:02.960519 Iter 1500: Training Loss = 31579.3496\n",
      "2018-01-31 01:11:44.435744 Iter 1600: Training Loss = 9190.6680\n",
      "2018-01-31 01:17:25.656363 Iter 1700: Training Loss = 20035.0938\n",
      "2018-01-31 01:23:06.821637 Iter 1800: Training Loss = 9401.5371\n",
      "2018-01-31 01:28:48.036414 Iter 1900: Training Loss = 9600.9443\n",
      "2018-01-31 01:34:29.296627 Iter 2000: Training Loss = 26167.8164\n",
      "2018-01-31 01:40:10.658002 Iter 2100: Training Loss = 27976.8828\n",
      "2018-01-31 01:45:51.722393 Iter 2200: Training Loss = 34597.3711\n",
      "2018-01-31 01:51:33.277223 Iter 2300: Training Loss = 21541.8652\n",
      "2018-01-31 01:57:15.197451 Iter 2400: Training Loss = 37720.3086\n",
      "2018-01-31 02:02:55.974301 Iter 2500: Training Loss = 16745.3516\n",
      "2018-01-31 02:08:36.815423 Iter 2600: Training Loss = 11005.9600\n",
      "2018-01-31 02:14:17.349800 Iter 2700: Training Loss = 11313.5957\n",
      "2018-01-31 02:19:58.383237 Iter 2800: Training Loss = 8410.4326\n",
      "2018-01-31 02:25:39.345694 Iter 2900: Training Loss = 12330.4414\n",
      "2018-01-31 02:31:20.075223 Iter 3000: Training Loss = 20254.0098\n",
      "2018-01-31 02:37:01.000036 Iter 3100: Training Loss = 24543.7598\n",
      "2018-01-31 02:42:41.045753 Iter 3200: Training Loss = 21505.1348\n",
      "2018-01-31 02:48:22.259596 Iter 3300: Training Loss = 15200.6582\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-01-31 02:54:02.291664 Iter 3400: Training Loss = 11270.4795\n",
      "2018-01-31 02:59:42.345527 Iter 3500: Training Loss = 60530.7891\n",
      "2018-01-31 03:05:22.844060 Iter 3600: Training Loss = 5610.5703\n",
      "2018-01-31 03:11:03.057009 Iter 3700: Training Loss = 25830.4277\n",
      "2018-01-31 03:16:43.472930 Iter 3800: Training Loss = 27597.2734\n",
      "2018-01-31 03:22:24.413489 Iter 3900: Training Loss = 12331.9375\n",
      "2018-01-31 03:28:04.443860 Iter 4000: Training Loss = 21082.2598\n",
      "2018-01-31 03:33:44.497701 Iter 4100: Training Loss = 13443.0225\n",
      "2018-01-31 03:39:24.851565 Iter 4200: Training Loss = 15971.2969\n",
      "2018-01-31 03:45:05.194807 Iter 4300: Training Loss = 19679.0352\n",
      "2018-01-31 03:50:45.266596 Iter 4400: Training Loss = 159844.8906\n",
      "2018-01-31 03:56:25.225160 Iter 4500: Training Loss = 2067.9700\n",
      "2018-01-31 04:02:05.950620 Iter 4600: Training Loss = 35693.9414\n",
      "2018-01-31 04:07:45.238442 Iter 4700: Training Loss = 73718.4141\n",
      "2018-01-31 04:13:25.732018 Iter 4800: Training Loss = 48485.9219\n",
      "2018-01-31 04:19:06.132029 Iter 4900: Training Loss = 5152.8135\n",
      "2018-01-31 04:24:45.757937 Iter 5000: Training Loss = 4567.3691\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-5000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-5000\n",
      "2018-01-31 04:30:29.578556 Iter 5100: Training Loss = 29687.1289\n",
      "2018-01-31 04:36:09.955024 Iter 5200: Training Loss = 3200.8884\n",
      "2018-01-31 04:41:49.367716 Iter 5300: Training Loss = 21213.5488\n",
      "2018-01-31 04:47:29.300568 Iter 5400: Training Loss = 30421.9648\n",
      "2018-01-31 04:53:08.766890 Iter 5500: Training Loss = 6820.2227\n",
      "2018-01-31 04:58:48.492744 Iter 5600: Training Loss = 9454.5186\n",
      "2018-01-31 05:04:28.084301 Iter 5700: Training Loss = 7649.7339\n",
      "2018-01-31 05:10:07.553226 Iter 5800: Training Loss = 8578.9795\n",
      "2018-01-31 05:15:46.531095 Iter 5900: Training Loss = 18094.0430\n",
      "2018-01-31 05:21:26.044504 Iter 6000: Training Loss = 18463.7715\n",
      "2018-01-31 05:27:05.341853 Iter 6100: Training Loss = 10570.9111\n",
      "2018-01-31 05:32:44.882840 Iter 6200: Training Loss = 11141.6553\n",
      "2018-01-31 05:38:24.505235 Iter 6300: Training Loss = 17885.6855\n",
      "2018-01-31 05:44:02.950426 Iter 6400: Training Loss = 16537.9023\n",
      "2018-01-31 05:49:42.643994 Iter 6500: Training Loss = 54193.2734\n",
      "2018-01-31 05:55:21.172051 Iter 6600: Training Loss = 85179.1016\n",
      "2018-01-31 06:01:00.287124 Iter 6700: Training Loss = 16717.6113\n",
      "2018-01-31 06:06:39.139296 Iter 6800: Training Loss = 3711.8887\n",
      "2018-01-31 06:12:17.605536 Iter 6900: Training Loss = 10098.3467\n",
      "2018-01-31 06:17:57.341979 Iter 7000: Training Loss = 19998.2324\n",
      "2018-01-31 06:23:36.588377 Iter 7100: Training Loss = 22698.5020\n",
      "2018-01-31 06:29:15.092922 Iter 7200: Training Loss = 23610.2285\n",
      "2018-01-31 06:34:53.520350 Iter 7300: Training Loss = 4484.7466\n",
      "2018-01-31 06:40:31.826111 Iter 7400: Training Loss = 56230.0508\n",
      "2018-01-31 06:46:10.688635 Iter 7500: Training Loss = 20711.3184\n",
      "2018-01-31 06:51:49.300363 Iter 7600: Training Loss = 20964.7910\n",
      "2018-01-31 06:57:27.305506 Iter 7700: Training Loss = 45281.4922\n",
      "2018-01-31 07:03:05.485475 Iter 7800: Training Loss = 10705.0928\n",
      "2018-01-31 07:08:43.893423 Iter 7900: Training Loss = 39947.3125\n",
      "2018-01-31 07:14:22.196627 Iter 8000: Training Loss = 20265.2695\n",
      "2018-01-31 07:20:00.103222 Iter 8100: Training Loss = 9779.9775\n",
      "2018-01-31 07:25:37.787396 Iter 8200: Training Loss = 23274.8359\n",
      "2018-01-31 07:31:15.353435 Iter 8300: Training Loss = 19757.5234\n",
      "2018-01-31 07:36:53.181103 Iter 8400: Training Loss = 12795.8857\n",
      "2018-01-31 07:42:31.074700 Iter 8500: Training Loss = 19920.6348\n",
      "2018-01-31 07:48:08.636268 Iter 8600: Training Loss = 16027.0752\n",
      "2018-01-31 07:53:46.826116 Iter 8700: Training Loss = 12566.1367\n",
      "2018-01-31 07:59:24.139936 Iter 8800: Training Loss = 85294.4297\n",
      "2018-01-31 08:05:01.978496 Iter 8900: Training Loss = 29906.6719\n",
      "2018-01-31 08:10:39.929635 Iter 9000: Training Loss = 55784.7852\n",
      "2018-01-31 08:16:17.622254 Iter 9100: Training Loss = 12775.3320\n",
      "2018-01-31 08:21:56.257460 Iter 9200: Training Loss = 20320.3652\n",
      "2018-01-31 08:27:34.006194 Iter 9300: Training Loss = 11911.9893\n",
      "2018-01-31 08:33:11.525179 Iter 9400: Training Loss = 37287.7188\n",
      "2018-01-31 08:38:52.029780 Iter 9500: Training Loss = 21909.3809\n",
      "2018-01-31 08:44:29.921536 Iter 9600: Training Loss = 13449.4912\n",
      "2018-01-31 08:50:07.389087 Iter 9700: Training Loss = 17437.5195\n",
      "2018-01-31 08:55:45.136348 Iter 9800: Training Loss = 20867.7305\n",
      "2018-01-31 09:01:22.834190 Iter 9900: Training Loss = 8798.9219\n",
      "2018-01-31 09:07:00.459114 Iter 10000: Training Loss = 13394.2812\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-10000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-10000\n",
      "2018-01-31 09:12:41.487198 Iter 10100: Training Loss = 16946.2188\n",
      "2018-01-31 09:18:19.110830 Iter 10200: Training Loss = 13096.6797\n",
      "2018-01-31 09:23:57.863872 Iter 10300: Training Loss = 31992.6133\n",
      "2018-01-31 09:29:35.561219 Iter 10400: Training Loss = 36858.0859\n",
      "2018-01-31 09:35:12.024874 Iter 10500: Training Loss = 18079.6133\n",
      "2018-01-31 09:40:48.767930 Iter 10600: Training Loss = 7318.8628\n",
      "2018-01-31 09:46:24.992822 Iter 10700: Training Loss = 12882.1455\n",
      "2018-01-31 09:52:01.220563 Iter 10800: Training Loss = 1338.2773\n",
      "2018-01-31 09:57:37.410034 Iter 10900: Training Loss = 9975.7676\n",
      "2018-01-31 10:03:13.553970 Iter 11000: Training Loss = 17349.9629\n",
      "2018-01-31 10:08:49.631269 Iter 11100: Training Loss = 14498.2773\n",
      "2018-01-31 10:14:25.622920 Iter 11200: Training Loss = 2617.5945\n",
      "2018-01-31 10:20:01.849204 Iter 11300: Training Loss = 13247.2344\n",
      "2018-01-31 10:25:37.714900 Iter 11400: Training Loss = 4226.3359\n",
      "2018-01-31 10:31:13.354026 Iter 11500: Training Loss = 31437.2344\n",
      "2018-01-31 10:36:49.115574 Iter 11600: Training Loss = 8664.8691\n",
      "2018-01-31 10:42:24.983677 Iter 11700: Training Loss = 6289.5356\n",
      "2018-01-31 10:48:01.381914 Iter 11800: Training Loss = 27375.6191\n",
      "2018-01-31 10:53:37.279477 Iter 11900: Training Loss = 4641.3716\n",
      "2018-01-31 10:59:13.274897 Iter 12000: Training Loss = 1146.8312\n",
      "2018-01-31 11:04:49.050446 Iter 12100: Training Loss = 9099.6670\n",
      "2018-01-31 11:10:24.915755 Iter 12200: Training Loss = 8491.0566\n",
      "2018-01-31 11:16:00.616091 Iter 12300: Training Loss = 15944.8516\n",
      "2018-01-31 11:21:36.398670 Iter 12400: Training Loss = 2452.9556\n",
      "2018-01-31 11:27:12.161637 Iter 12500: Training Loss = 13183.6631\n",
      "2018-01-31 11:32:47.947639 Iter 12600: Training Loss = 8526.6904\n",
      "2018-01-31 11:38:23.838176 Iter 12700: Training Loss = 39337.6992\n",
      "2018-01-31 11:43:59.785664 Iter 12800: Training Loss = 19786.0156\n",
      "2018-01-31 11:49:35.383228 Iter 12900: Training Loss = 9235.0635\n",
      "2018-01-31 11:55:11.109127 Iter 13000: Training Loss = 3845.6125\n",
      "2018-01-31 12:00:46.828645 Iter 13100: Training Loss = 4239.6357\n",
      "2018-01-31 12:06:22.565702 Iter 13200: Training Loss = 4059.3093\n",
      "2018-01-31 12:11:58.074592 Iter 13300: Training Loss = 54424.3828\n",
      "2018-01-31 12:17:33.455735 Iter 13400: Training Loss = 16340.7061\n",
      "2018-01-31 12:23:09.107786 Iter 13500: Training Loss = 9696.9268\n",
      "2018-01-31 12:28:44.768210 Iter 13600: Training Loss = 29197.7793\n",
      "2018-01-31 12:34:20.643778 Iter 13700: Training Loss = 10606.6289\n",
      "2018-01-31 12:39:56.450746 Iter 13800: Training Loss = 1970.6229\n",
      "2018-01-31 12:45:32.183913 Iter 13900: Training Loss = 8080.3457\n",
      "2018-01-31 12:51:09.490782 Iter 14000: Training Loss = 2303.2515\n",
      "2018-01-31 12:56:48.054656 Iter 14100: Training Loss = 6322.1426\n",
      "2018-01-31 13:02:26.576110 Iter 14200: Training Loss = 19759.3789\n",
      "2018-01-31 13:08:04.415613 Iter 14300: Training Loss = 41832.4531\n",
      "2018-01-31 13:13:42.170433 Iter 14400: Training Loss = 6614.3906\n",
      "2018-01-31 13:19:20.281538 Iter 14500: Training Loss = 15541.6768\n",
      "2018-01-31 13:24:59.015190 Iter 14600: Training Loss = 23590.2598\n",
      "2018-01-31 13:30:37.538656 Iter 14700: Training Loss = 22678.8613\n",
      "2018-01-31 13:36:16.423355 Iter 14800: Training Loss = 13555.0615\n",
      "2018-01-31 13:41:54.188247 Iter 14900: Training Loss = 20164.2480\n",
      "2018-01-31 13:47:32.634977 Iter 15000: Training Loss = 59137.1484\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-15000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-15000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished training.\n"
     ]
    }
   ],
   "source": [
    "# Train the network with strong side outputs supervision\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        global_step = tf.Variable(0, name='global_step', trainable=False)\n",
    "        learning_rate = tf.train.piecewise_constant(global_step, boundaries, values)\n",
    "        model.train_parent(dataset, imagenet_ckpt, 1, learning_rate, logs_path, max_training_iters_1, save_step,\n",
    "                           display_step, global_step, segnet_stream, iter_mean_grad=iter_mean_grad, test_image_path=test_image,\n",
    "                           ckpt_name=ckpt_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Layers:\n",
      "   name = fstream/conv1/conv1_1/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv1/conv1_2/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/pool1/MaxPool:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv2/conv2_1/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv2/conv2_2/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/pool2/MaxPool:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv3/conv3_1/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_2/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_3/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/pool3/MaxPool:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv4/conv4_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/pool4/MaxPool:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv2_2_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv3_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv4_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv5_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-dsn_2/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_2-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_1:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_2:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_3:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-multi2-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_4:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi3-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_5:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi4-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_6:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi5-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_7:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/upscore-fuse/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "Network Parameters:\n",
      "   name = fstream/conv1/conv1_1/weights:0, shape = (3, 3, 3, 64)\n",
      "   name = fstream/conv1/conv1_1/biases:0, shape = (64,)\n",
      "   name = fstream/conv1/conv1_2/weights:0, shape = (3, 3, 64, 64)\n",
      "   name = fstream/conv1/conv1_2/biases:0, shape = (64,)\n",
      "   name = fstream/conv2/conv2_1/weights:0, shape = (3, 3, 64, 128)\n",
      "   name = fstream/conv2/conv2_1/biases:0, shape = (128,)\n",
      "   name = fstream/conv2/conv2_2/weights:0, shape = (3, 3, 128, 128)\n",
      "   name = fstream/conv2/conv2_2/biases:0, shape = (128,)\n",
      "   name = fstream/conv3/conv3_1/weights:0, shape = (3, 3, 128, 256)\n",
      "   name = fstream/conv3/conv3_1/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_2/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_2/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_3/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_3/biases:0, shape = (256,)\n",
      "   name = fstream/conv4/conv4_1/weights:0, shape = (3, 3, 256, 512)\n",
      "   name = fstream/conv4/conv4_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_1/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv2_2_16/weights:0, shape = (3, 3, 128, 16)\n",
      "   name = fstream/conv2_2_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv3_3_16/weights:0, shape = (3, 3, 256, 16)\n",
      "   name = fstream/conv3_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv4_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv4_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv5_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv5_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/score-dsn_2/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_2/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_3/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_3/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_4/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_4/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_5/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_5/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_2-up/weights:0, shape = (4, 4, 1, 1)\n",
      "   name = fstream/score-dsn_3-up/weights:0, shape = (8, 8, 1, 1)\n",
      "   name = fstream/score-dsn_4-up/weights:0, shape = (16, 16, 1, 1)\n",
      "   name = fstream/score-dsn_5-up/weights:0, shape = (32, 32, 1, 1)\n",
      "   name = fstream/score-multi2-up/weights:0, shape = (4, 4, 16, 16)\n",
      "   name = fstream/score-multi3-up/weights:0, shape = (8, 8, 16, 16)\n",
      "   name = fstream/score-multi4-up/weights:0, shape = (16, 16, 16, 16)\n",
      "   name = fstream/score-multi5-up/weights:0, shape = (32, 32, 16, 16)\n",
      "   name = fstream/upscore-fuse/weights:0, shape = (1, 1, 64, 1)\n",
      "   name = fstream/upscore-fuse/biases:0, shape = (1,)\n",
      "Init variable\n",
      "Initializing from previous checkpoint...\n",
      "INFO:tensorflow:Restoring parameters from models\\fstream_parent\\fstream_parent.ckpt-15000\n",
      "Weights initialized\n",
      "Start training\n",
      "2018-01-31 13:53:30.955952 Iter 15100: Training Loss = 15431.0742\n",
      "2018-01-31 13:59:09.119220 Iter 15200: Training Loss = 6597.1104\n",
      "2018-01-31 14:04:47.446319 Iter 15300: Training Loss = 11310.4219\n",
      "2018-01-31 14:10:27.397060 Iter 15400: Training Loss = 1234.6230\n",
      "2018-01-31 14:16:07.248379 Iter 15500: Training Loss = 15918.1426\n",
      "2018-01-31 14:21:47.034874 Iter 15600: Training Loss = 958.7816\n",
      "2018-01-31 14:27:26.818328 Iter 15700: Training Loss = 13990.2168\n",
      "2018-01-31 14:33:07.105479 Iter 15800: Training Loss = 6759.7231\n",
      "2018-01-31 14:38:49.143635 Iter 15900: Training Loss = 2436.2405\n",
      "2018-01-31 14:44:30.947285 Iter 16000: Training Loss = 10097.5234\n",
      "2018-01-31 14:50:12.524862 Iter 16100: Training Loss = 1186.8881\n",
      "2018-01-31 14:55:54.466289 Iter 16200: Training Loss = 3665.7603\n",
      "2018-01-31 15:01:36.976715 Iter 16300: Training Loss = 9137.4443\n",
      "2018-01-31 15:07:19.990133 Iter 16400: Training Loss = 13283.2979\n",
      "2018-01-31 15:13:02.116798 Iter 16500: Training Loss = 9579.1641\n",
      "2018-01-31 15:18:42.209376 Iter 16600: Training Loss = 912.9023\n",
      "2018-01-31 15:24:23.298855 Iter 16700: Training Loss = 4390.7397\n",
      "2018-01-31 15:30:02.485584 Iter 16800: Training Loss = 5300.5947\n",
      "2018-01-31 15:35:41.452069 Iter 16900: Training Loss = 3703.9678\n",
      "2018-01-31 15:41:20.264832 Iter 17000: Training Loss = 21684.7676\n",
      "2018-01-31 15:46:59.501617 Iter 17100: Training Loss = 7097.8105\n",
      "2018-01-31 15:52:38.299876 Iter 17200: Training Loss = 4911.6372\n",
      "2018-01-31 15:58:16.989975 Iter 17300: Training Loss = 0.4082\n",
      "2018-01-31 16:03:55.473163 Iter 17400: Training Loss = 9050.5840\n",
      "2018-01-31 16:09:34.477475 Iter 17500: Training Loss = 7887.2432\n",
      "2018-01-31 16:15:12.890463 Iter 17600: Training Loss = 3910.7222\n",
      "2018-01-31 16:20:54.044038 Iter 17700: Training Loss = 15780.2656\n",
      "2018-01-31 16:26:34.094361 Iter 17800: Training Loss = 44427.6523\n",
      "2018-01-31 16:32:14.148300 Iter 17900: Training Loss = 2471.6216\n",
      "2018-01-31 16:37:54.327232 Iter 18000: Training Loss = 14121.1211\n",
      "2018-01-31 16:43:34.397674 Iter 18100: Training Loss = 7418.1411\n",
      "2018-01-31 16:49:14.117799 Iter 18200: Training Loss = 12510.0264\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-01-31 16:54:53.902193 Iter 18300: Training Loss = 953.1269\n",
      "2018-01-31 17:00:33.313879 Iter 18400: Training Loss = 8677.8760\n",
      "2018-01-31 17:06:13.068090 Iter 18500: Training Loss = 13860.7119\n",
      "2018-01-31 17:11:53.670203 Iter 18600: Training Loss = 14735.6367\n",
      "2018-01-31 17:17:32.427196 Iter 18700: Training Loss = 15957.6230\n",
      "2018-01-31 17:23:09.629562 Iter 18800: Training Loss = 11599.0400\n",
      "2018-01-31 17:28:44.997200 Iter 18900: Training Loss = 3127.4773\n",
      "2018-01-31 17:34:20.234634 Iter 19000: Training Loss = 12191.3691\n",
      "2018-01-31 17:39:55.924644 Iter 19100: Training Loss = 1164.7054\n",
      "2018-01-31 17:45:31.250823 Iter 19200: Training Loss = 16669.8105\n",
      "2018-01-31 17:51:06.991163 Iter 19300: Training Loss = 7274.4878\n",
      "2018-01-31 17:56:42.060803 Iter 19400: Training Loss = 4742.8896\n",
      "2018-01-31 18:02:17.165247 Iter 19500: Training Loss = 4301.2656\n",
      "2018-01-31 18:07:52.455863 Iter 19600: Training Loss = 3335.3042\n",
      "2018-01-31 18:13:27.826877 Iter 19700: Training Loss = 39116.0508\n",
      "2018-01-31 18:19:02.862392 Iter 19800: Training Loss = 3625.6650\n",
      "2018-01-31 18:24:37.822046 Iter 19900: Training Loss = 7443.8906\n",
      "2018-01-31 18:30:13.007134 Iter 20000: Training Loss = 7692.5332\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-20000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-20000\n",
      "2018-01-31 18:35:51.440197 Iter 20100: Training Loss = 5011.0054\n",
      "2018-01-31 18:41:26.655659 Iter 20200: Training Loss = 13486.9863\n",
      "2018-01-31 18:47:01.848479 Iter 20300: Training Loss = 7272.0967\n",
      "2018-01-31 18:52:36.669943 Iter 20400: Training Loss = 9240.9277\n",
      "2018-01-31 18:58:11.668116 Iter 20500: Training Loss = 34649.3398\n",
      "2018-01-31 19:03:46.532205 Iter 20600: Training Loss = 5024.4531\n",
      "2018-01-31 19:09:21.196247 Iter 20700: Training Loss = 8717.9961\n",
      "2018-01-31 19:14:56.111223 Iter 20800: Training Loss = 16799.0312\n",
      "2018-01-31 19:20:32.420697 Iter 20900: Training Loss = 6756.6440\n",
      "2018-01-31 19:26:07.717421 Iter 21000: Training Loss = 1942.8683\n",
      "2018-01-31 19:31:42.847991 Iter 21100: Training Loss = 9367.2588\n",
      "2018-01-31 19:37:17.850483 Iter 21200: Training Loss = 8018.0439\n",
      "2018-01-31 19:42:52.582116 Iter 21300: Training Loss = 9121.9229\n",
      "2018-01-31 19:48:27.875401 Iter 21400: Training Loss = 6001.3628\n",
      "2018-01-31 19:54:02.969640 Iter 21500: Training Loss = 55039.9961\n",
      "2018-01-31 19:59:38.151838 Iter 21600: Training Loss = 22188.3438\n",
      "2018-01-31 20:05:12.806639 Iter 21700: Training Loss = 9178.8711\n",
      "2018-01-31 20:10:47.538981 Iter 21800: Training Loss = 4478.2515\n",
      "2018-01-31 20:16:22.349764 Iter 21900: Training Loss = 44152.9453\n",
      "2018-01-31 20:21:57.255278 Iter 22000: Training Loss = 45342.6641\n",
      "2018-01-31 20:27:32.022137 Iter 22100: Training Loss = 5726.3301\n",
      "2018-01-31 20:33:06.851695 Iter 22200: Training Loss = 10287.2227\n",
      "2018-01-31 20:38:41.697021 Iter 22300: Training Loss = 6549.6084\n",
      "2018-01-31 20:44:16.828347 Iter 22400: Training Loss = 0.4084\n",
      "2018-01-31 20:49:51.588414 Iter 22500: Training Loss = 1763.3690\n",
      "2018-01-31 20:55:26.538310 Iter 22600: Training Loss = 4634.7544\n",
      "2018-01-31 21:01:01.814388 Iter 22700: Training Loss = 3600.0388\n",
      "2018-01-31 21:06:37.504244 Iter 22800: Training Loss = 7504.5566\n",
      "2018-01-31 21:12:12.192670 Iter 22900: Training Loss = 2427.8801\n",
      "2018-01-31 21:17:47.158128 Iter 23000: Training Loss = 819.3314\n",
      "2018-01-31 21:23:22.046602 Iter 23100: Training Loss = 4721.0928\n",
      "2018-01-31 21:28:56.803522 Iter 23200: Training Loss = 2101.1716\n",
      "2018-01-31 21:34:31.372568 Iter 23300: Training Loss = 3622.8538\n",
      "2018-01-31 21:40:06.178274 Iter 23400: Training Loss = 4708.5435\n",
      "2018-01-31 21:45:40.994037 Iter 23500: Training Loss = 9196.4414\n",
      "2018-01-31 21:51:15.738171 Iter 23600: Training Loss = 15374.9805\n",
      "2018-01-31 21:56:50.443009 Iter 23700: Training Loss = 1917.7927\n",
      "2018-01-31 22:02:25.096042 Iter 23800: Training Loss = 8170.7759\n",
      "2018-01-31 22:07:59.983963 Iter 23900: Training Loss = 27457.5039\n",
      "2018-01-31 22:13:34.882535 Iter 24000: Training Loss = 2331.6611\n",
      "2018-01-31 22:19:09.247505 Iter 24100: Training Loss = 86202.8906\n",
      "2018-01-31 22:24:43.933441 Iter 24200: Training Loss = 11405.8770\n",
      "2018-01-31 22:30:18.553143 Iter 24300: Training Loss = 20430.2559\n",
      "2018-01-31 22:35:52.833979 Iter 24400: Training Loss = 8356.1211\n",
      "2018-01-31 22:41:27.476013 Iter 24500: Training Loss = 6665.7236\n",
      "2018-01-31 22:47:02.042380 Iter 24600: Training Loss = 19011.3945\n",
      "2018-01-31 22:52:36.711635 Iter 24700: Training Loss = 4639.7568\n",
      "2018-01-31 22:58:11.676681 Iter 24800: Training Loss = 4555.5669\n",
      "2018-01-31 23:03:46.642714 Iter 24900: Training Loss = 1217.0258\n",
      "2018-01-31 23:09:21.221173 Iter 25000: Training Loss = 14918.8486\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-25000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-25000\n",
      "2018-01-31 23:14:58.877991 Iter 25100: Training Loss = 25192.7559\n",
      "2018-01-31 23:20:33.298025 Iter 25200: Training Loss = 7700.8672\n",
      "2018-01-31 23:26:07.726415 Iter 25300: Training Loss = 11611.7588\n",
      "2018-01-31 23:31:41.958245 Iter 25400: Training Loss = 14193.5098\n",
      "2018-01-31 23:37:16.375631 Iter 25500: Training Loss = 4248.2559\n",
      "2018-01-31 23:42:50.802223 Iter 25600: Training Loss = 636.0203\n",
      "2018-01-31 23:48:25.201930 Iter 25700: Training Loss = 7214.0776\n",
      "2018-01-31 23:53:59.733717 Iter 25800: Training Loss = 15119.1855\n",
      "2018-01-31 23:59:34.425775 Iter 25900: Training Loss = 7409.4517\n",
      "2018-02-01 00:05:09.146305 Iter 26000: Training Loss = 13193.4121\n",
      "2018-02-01 00:10:43.796020 Iter 26100: Training Loss = 2369.3105\n",
      "2018-02-01 00:16:18.347103 Iter 26200: Training Loss = 3461.9719\n",
      "2018-02-01 00:21:52.708876 Iter 26300: Training Loss = 5180.1255\n",
      "2018-02-01 00:27:27.039869 Iter 26400: Training Loss = 25902.1777\n",
      "2018-02-01 00:33:01.329125 Iter 26500: Training Loss = 5238.7808\n",
      "2018-02-01 00:38:35.824539 Iter 26600: Training Loss = 8021.6479\n",
      "2018-02-01 00:44:10.243360 Iter 26700: Training Loss = 9220.1816\n",
      "2018-02-01 00:49:44.841605 Iter 26800: Training Loss = 4929.3906\n",
      "2018-02-01 00:55:19.134777 Iter 26900: Training Loss = 8119.1929\n",
      "2018-02-01 01:00:53.041907 Iter 27000: Training Loss = 604.2535\n",
      "2018-02-01 01:06:27.908921 Iter 27100: Training Loss = 6017.8682\n",
      "2018-02-01 01:12:02.174734 Iter 27200: Training Loss = 11906.5752\n",
      "2018-02-01 01:17:36.351042 Iter 27300: Training Loss = 3587.4683\n",
      "2018-02-01 01:23:10.458981 Iter 27400: Training Loss = 2233.3633\n",
      "2018-02-01 01:28:44.763861 Iter 27500: Training Loss = 24126.9883\n",
      "2018-02-01 01:34:19.019193 Iter 27600: Training Loss = 5557.8218\n",
      "2018-02-01 01:39:53.360702 Iter 27700: Training Loss = 8336.5801\n",
      "2018-02-01 01:45:27.554118 Iter 27800: Training Loss = 15114.3828\n",
      "2018-02-01 01:51:01.730669 Iter 27900: Training Loss = 11467.8398\n",
      "2018-02-01 01:56:36.060058 Iter 28000: Training Loss = 8899.9688\n",
      "2018-02-01 02:02:10.301640 Iter 28100: Training Loss = 6001.1416\n",
      "2018-02-01 02:07:44.597285 Iter 28200: Training Loss = 7414.1724\n",
      "2018-02-01 02:13:18.911064 Iter 28300: Training Loss = 22435.5801\n",
      "2018-02-01 02:18:53.080366 Iter 28400: Training Loss = 4798.7803\n",
      "2018-02-01 02:24:27.252859 Iter 28500: Training Loss = 9069.6270\n",
      "2018-02-01 02:30:01.325921 Iter 28600: Training Loss = 6849.0371\n",
      "2018-02-01 02:35:35.853404 Iter 28700: Training Loss = 3247.5718\n",
      "2018-02-01 02:41:10.175419 Iter 28800: Training Loss = 6483.4751\n",
      "2018-02-01 02:46:44.223687 Iter 28900: Training Loss = 11656.8535\n",
      "2018-02-01 02:52:18.695609 Iter 29000: Training Loss = 9469.4766\n",
      "2018-02-01 02:57:52.922265 Iter 29100: Training Loss = 7223.0625\n",
      "2018-02-01 03:03:27.189279 Iter 29200: Training Loss = 5931.9614\n",
      "2018-02-01 03:09:01.485313 Iter 29300: Training Loss = 8240.8213\n",
      "2018-02-01 03:14:35.925902 Iter 29400: Training Loss = 3114.8179\n",
      "2018-02-01 03:20:10.395969 Iter 29500: Training Loss = 7387.5840\n",
      "2018-02-01 03:25:44.909684 Iter 29600: Training Loss = 7674.1743\n",
      "2018-02-01 03:31:19.050620 Iter 29700: Training Loss = 9632.8545\n",
      "2018-02-01 03:36:53.066943 Iter 29800: Training Loss = 2225.5620\n",
      "2018-02-01 03:42:27.331539 Iter 29900: Training Loss = 46279.3438\n",
      "2018-02-01 03:48:01.581024 Iter 30000: Training Loss = 20610.1387\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-30000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-30000\n",
      "Finished training.\n"
     ]
    }
   ],
   "source": [
    "# Train the network with weak side outputs supervision\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        global_step = tf.Variable(max_training_iters_1, name='global_step', trainable=False)\n",
    "        learning_rate = tf.train.piecewise_constant(global_step, boundaries, values)\n",
    "        model.train_parent(dataset, imagenet_ckpt, 2, learning_rate, logs_path, max_training_iters_2, save_step,\n",
    "                           display_step, global_step, segnet_stream, iter_mean_grad=iter_mean_grad, resume_training=True,\n",
    "                           test_image_path=test_image, ckpt_name=ckpt_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Layers:\n",
      "   name = fstream/conv1/conv1_1/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv1/conv1_2/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/pool1/MaxPool:0, shape = (1, ?, ?, 64)\n",
      "   name = fstream/conv2/conv2_1/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv2/conv2_2/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/pool2/MaxPool:0, shape = (1, ?, ?, 128)\n",
      "   name = fstream/conv3/conv3_1/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_2/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv3/conv3_3/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/pool3/MaxPool:0, shape = (1, ?, ?, 256)\n",
      "   name = fstream/conv4/conv4_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv4/conv4_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/pool4/MaxPool:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv5/conv5_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = fstream/conv2_2_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv3_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv4_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/conv5_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-dsn_2/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_2-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_3-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_1:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_4-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_2:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-dsn_5-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/Reshape_3:0, shape = (1, ?, ?, 1)\n",
      "   name = fstream/score-multi2-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_4:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi3-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_5:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi4-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_6:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/score-multi5-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/Reshape_7:0, shape = (1, ?, ?, 16)\n",
      "   name = fstream/upscore-fuse/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "Network Parameters:\n",
      "   name = fstream/conv1/conv1_1/weights:0, shape = (3, 3, 3, 64)\n",
      "   name = fstream/conv1/conv1_1/biases:0, shape = (64,)\n",
      "   name = fstream/conv1/conv1_2/weights:0, shape = (3, 3, 64, 64)\n",
      "   name = fstream/conv1/conv1_2/biases:0, shape = (64,)\n",
      "   name = fstream/conv2/conv2_1/weights:0, shape = (3, 3, 64, 128)\n",
      "   name = fstream/conv2/conv2_1/biases:0, shape = (128,)\n",
      "   name = fstream/conv2/conv2_2/weights:0, shape = (3, 3, 128, 128)\n",
      "   name = fstream/conv2/conv2_2/biases:0, shape = (128,)\n",
      "   name = fstream/conv3/conv3_1/weights:0, shape = (3, 3, 128, 256)\n",
      "   name = fstream/conv3/conv3_1/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_2/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_2/biases:0, shape = (256,)\n",
      "   name = fstream/conv3/conv3_3/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = fstream/conv3/conv3_3/biases:0, shape = (256,)\n",
      "   name = fstream/conv4/conv4_1/weights:0, shape = (3, 3, 256, 512)\n",
      "   name = fstream/conv4/conv4_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv4/conv4_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv4/conv4_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_1/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_1/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_2/biases:0, shape = (512,)\n",
      "   name = fstream/conv5/conv5_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = fstream/conv5/conv5_3/biases:0, shape = (512,)\n",
      "   name = fstream/conv2_2_16/weights:0, shape = (3, 3, 128, 16)\n",
      "   name = fstream/conv2_2_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv3_3_16/weights:0, shape = (3, 3, 256, 16)\n",
      "   name = fstream/conv3_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv4_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv4_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/conv5_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = fstream/conv5_3_16/biases:0, shape = (16,)\n",
      "   name = fstream/score-dsn_2/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_2/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_3/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_3/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_4/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_4/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_5/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = fstream/score-dsn_5/biases:0, shape = (1,)\n",
      "   name = fstream/score-dsn_2-up/weights:0, shape = (4, 4, 1, 1)\n",
      "   name = fstream/score-dsn_3-up/weights:0, shape = (8, 8, 1, 1)\n",
      "   name = fstream/score-dsn_4-up/weights:0, shape = (16, 16, 1, 1)\n",
      "   name = fstream/score-dsn_5-up/weights:0, shape = (32, 32, 1, 1)\n",
      "   name = fstream/score-multi2-up/weights:0, shape = (4, 4, 16, 16)\n",
      "   name = fstream/score-multi3-up/weights:0, shape = (8, 8, 16, 16)\n",
      "   name = fstream/score-multi4-up/weights:0, shape = (16, 16, 16, 16)\n",
      "   name = fstream/score-multi5-up/weights:0, shape = (32, 32, 16, 16)\n",
      "   name = fstream/upscore-fuse/weights:0, shape = (1, 1, 64, 1)\n",
      "   name = fstream/upscore-fuse/biases:0, shape = (1,)\n",
      "Init variable\n",
      "Initializing from previous checkpoint...\n",
      "INFO:tensorflow:Restoring parameters from models\\fstream_parent\\fstream_parent.ckpt-30000\n",
      "Weights initialized\n",
      "Start training\n",
      "2018-02-01 03:53:22.659234 Iter 30100: Training Loss = 2700.2056\n",
      "2018-02-01 03:58:29.068189 Iter 30200: Training Loss = 5143.9966\n",
      "2018-02-01 04:03:35.925540 Iter 30300: Training Loss = 2465.3069\n",
      "2018-02-01 04:08:42.480883 Iter 30400: Training Loss = 3253.0298\n",
      "2018-02-01 04:13:48.957736 Iter 30500: Training Loss = 2752.2295\n",
      "2018-02-01 04:18:55.757193 Iter 30600: Training Loss = 254.1394\n",
      "2018-02-01 04:24:02.472689 Iter 30700: Training Loss = 1412.7574\n",
      "2018-02-01 04:29:09.158338 Iter 30800: Training Loss = 2779.4854\n",
      "2018-02-01 04:34:15.944166 Iter 30900: Training Loss = 1239.5007\n",
      "2018-02-01 04:39:22.833616 Iter 31000: Training Loss = 5239.0220\n",
      "2018-02-01 04:44:29.383735 Iter 31100: Training Loss = 4501.0571\n",
      "2018-02-01 04:49:36.766170 Iter 31200: Training Loss = 2942.9084\n",
      "2018-02-01 04:54:44.108249 Iter 31300: Training Loss = 1184.5924\n",
      "2018-02-01 04:59:51.748314 Iter 31400: Training Loss = 1277.4968\n",
      "2018-02-01 05:04:59.154284 Iter 31500: Training Loss = 525.6161\n",
      "2018-02-01 05:10:06.612863 Iter 31600: Training Loss = 568.0780\n",
      "2018-02-01 05:15:14.097243 Iter 31700: Training Loss = 1337.8909\n",
      "2018-02-01 05:20:21.579360 Iter 31800: Training Loss = 3360.7490\n",
      "2018-02-01 05:25:29.227065 Iter 31900: Training Loss = 3935.5923\n",
      "2018-02-01 05:30:36.610988 Iter 32000: Training Loss = 2116.6509\n",
      "2018-02-01 05:35:44.162576 Iter 32100: Training Loss = 1257.2236\n",
      "2018-02-01 05:40:51.941017 Iter 32200: Training Loss = 7106.9321\n",
      "2018-02-01 05:45:59.426775 Iter 32300: Training Loss = 973.3500\n",
      "2018-02-01 05:51:06.160869 Iter 32400: Training Loss = 3044.3245\n",
      "2018-02-01 05:56:12.982423 Iter 32500: Training Loss = 1597.3276\n",
      "2018-02-01 06:01:19.957236 Iter 32600: Training Loss = 2317.1558\n",
      "2018-02-01 06:06:26.673838 Iter 32700: Training Loss = 1615.8861\n",
      "2018-02-01 06:11:33.466546 Iter 32800: Training Loss = 1040.5656\n",
      "2018-02-01 06:16:40.311409 Iter 32900: Training Loss = 2864.6465\n",
      "2018-02-01 06:21:47.022237 Iter 33000: Training Loss = 4440.3633\n",
      "2018-02-01 06:26:53.939087 Iter 33100: Training Loss = 280.4832\n",
      "2018-02-01 06:32:00.945941 Iter 33200: Training Loss = 598.6611\n",
      "2018-02-01 06:37:07.922956 Iter 33300: Training Loss = 1011.4592\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-02-01 06:42:14.914684 Iter 33400: Training Loss = 8093.8384\n",
      "2018-02-01 06:47:21.759702 Iter 33500: Training Loss = 2032.3414\n",
      "2018-02-01 06:52:29.734673 Iter 33600: Training Loss = 1796.6763\n",
      "2018-02-01 06:57:37.762587 Iter 33700: Training Loss = 69.6234\n",
      "2018-02-01 07:02:45.600817 Iter 33800: Training Loss = 844.9960\n",
      "2018-02-01 07:07:53.608262 Iter 33900: Training Loss = 11779.7393\n",
      "2018-02-01 07:13:01.353649 Iter 34000: Training Loss = 1849.7102\n",
      "2018-02-01 07:18:09.389271 Iter 34100: Training Loss = 1278.1311\n",
      "2018-02-01 07:23:17.500768 Iter 34200: Training Loss = 835.7748\n",
      "2018-02-01 07:28:25.099870 Iter 34300: Training Loss = 71.5322\n",
      "2018-02-01 07:33:32.876707 Iter 34400: Training Loss = 1316.3887\n",
      "2018-02-01 07:38:40.502646 Iter 34500: Training Loss = 385.2216\n",
      "2018-02-01 07:43:48.023997 Iter 34600: Training Loss = 1876.2085\n",
      "2018-02-01 07:48:55.219889 Iter 34700: Training Loss = 1489.0850\n",
      "2018-02-01 07:54:01.868462 Iter 34800: Training Loss = 141.1523\n",
      "2018-02-01 07:59:08.836424 Iter 34900: Training Loss = 1784.8315\n",
      "2018-02-01 08:04:15.687380 Iter 35000: Training Loss = 2843.7061\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-35000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-35000\n",
      "2018-02-01 08:09:25.523524 Iter 35100: Training Loss = 385.8978\n",
      "2018-02-01 08:14:32.387166 Iter 35200: Training Loss = 857.5366\n",
      "2018-02-01 08:19:38.814203 Iter 35300: Training Loss = 1758.1235\n",
      "2018-02-01 08:24:45.368643 Iter 35400: Training Loss = 7010.6870\n",
      "2018-02-01 08:29:52.042075 Iter 35500: Training Loss = 1959.6260\n",
      "2018-02-01 08:34:59.070951 Iter 35600: Training Loss = 396.3411\n",
      "2018-02-01 08:40:05.772018 Iter 35700: Training Loss = 1254.4596\n",
      "2018-02-01 08:45:12.474502 Iter 35800: Training Loss = 1246.2988\n",
      "2018-02-01 08:50:19.811540 Iter 35900: Training Loss = 1755.2744\n",
      "2018-02-01 08:55:27.378119 Iter 36000: Training Loss = 1312.3473\n",
      "2018-02-01 09:00:35.238109 Iter 36100: Training Loss = 3805.5938\n",
      "2018-02-01 09:05:43.196021 Iter 36200: Training Loss = 1002.5414\n",
      "2018-02-01 09:10:50.879955 Iter 36300: Training Loss = 3074.8740\n",
      "2018-02-01 09:15:59.932153 Iter 36400: Training Loss = 498.1718\n",
      "2018-02-01 09:21:10.501259 Iter 36500: Training Loss = 109.6332\n",
      "2018-02-01 09:26:21.309064 Iter 36600: Training Loss = 1572.2368\n",
      "2018-02-01 09:31:32.171558 Iter 36700: Training Loss = 3249.7515\n",
      "2018-02-01 09:36:41.388442 Iter 36800: Training Loss = 2838.1714\n",
      "2018-02-01 09:41:51.162805 Iter 36900: Training Loss = 3042.5171\n",
      "2018-02-01 09:47:00.374166 Iter 37000: Training Loss = 1345.2029\n",
      "2018-02-01 09:52:08.998130 Iter 37100: Training Loss = 1817.6531\n",
      "2018-02-01 09:57:18.407566 Iter 37200: Training Loss = 1090.2870\n",
      "2018-02-01 10:02:27.453338 Iter 37300: Training Loss = 4903.3506\n",
      "2018-02-01 10:07:36.903026 Iter 37400: Training Loss = 1532.4037\n",
      "2018-02-01 10:12:45.738517 Iter 37500: Training Loss = 3759.6328\n",
      "2018-02-01 10:17:54.992274 Iter 37600: Training Loss = 1184.1426\n",
      "2018-02-01 10:23:04.691819 Iter 37700: Training Loss = 2633.9580\n",
      "2018-02-01 10:28:14.152388 Iter 37800: Training Loss = 2624.3091\n",
      "2018-02-01 10:33:23.628147 Iter 37900: Training Loss = 3274.7878\n",
      "2018-02-01 10:38:33.013100 Iter 38000: Training Loss = 3594.2290\n",
      "2018-02-01 10:43:42.247257 Iter 38100: Training Loss = 1265.2119\n",
      "2018-02-01 10:48:53.408265 Iter 38200: Training Loss = 4311.5342\n",
      "2018-02-01 10:54:04.892317 Iter 38300: Training Loss = 75.3486\n",
      "2018-02-01 10:59:18.058611 Iter 38400: Training Loss = 1922.2854\n",
      "2018-02-01 11:04:29.601605 Iter 38500: Training Loss = 989.1518\n",
      "2018-02-01 11:09:43.614416 Iter 38600: Training Loss = 2351.8303\n",
      "2018-02-01 11:14:57.935385 Iter 38700: Training Loss = 332.7806\n",
      "2018-02-01 11:20:08.962407 Iter 38800: Training Loss = 845.9764\n",
      "2018-02-01 11:25:20.727327 Iter 38900: Training Loss = 846.5536\n",
      "2018-02-01 11:30:32.347801 Iter 39000: Training Loss = 827.6400\n",
      "2018-02-01 11:35:43.988798 Iter 39100: Training Loss = 146.4621\n",
      "2018-02-01 11:40:59.002182 Iter 39200: Training Loss = 1121.4253\n",
      "2018-02-01 11:46:09.398803 Iter 39300: Training Loss = 1280.6951\n",
      "2018-02-01 11:51:18.296957 Iter 39400: Training Loss = 852.2782\n",
      "2018-02-01 11:56:27.403995 Iter 39500: Training Loss = 579.1174\n",
      "2018-02-01 12:01:36.206115 Iter 39600: Training Loss = 505.8630\n",
      "2018-02-01 12:06:45.814576 Iter 39700: Training Loss = 1642.7704\n",
      "2018-02-01 12:11:55.229467 Iter 39800: Training Loss = 470.1259\n",
      "2018-02-01 12:17:04.753974 Iter 39900: Training Loss = 3775.8008\n",
      "2018-02-01 12:22:15.822972 Iter 40000: Training Loss = 486.2006\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-40000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-40000\n",
      "2018-02-01 12:27:29.904893 Iter 40100: Training Loss = 878.9332\n",
      "2018-02-01 12:32:38.413265 Iter 40200: Training Loss = 880.9199\n",
      "2018-02-01 12:37:47.481700 Iter 40300: Training Loss = 224.0037\n",
      "2018-02-01 12:42:56.235683 Iter 40400: Training Loss = 1156.0458\n",
      "2018-02-01 12:48:05.019513 Iter 40500: Training Loss = 1007.7582\n",
      "2018-02-01 12:53:13.664366 Iter 40600: Training Loss = 2833.0762\n",
      "2018-02-01 12:58:22.140475 Iter 40700: Training Loss = 2418.3123\n",
      "2018-02-01 13:03:31.303031 Iter 40800: Training Loss = 1039.4186\n",
      "2018-02-01 13:08:41.349817 Iter 40900: Training Loss = 984.2020\n",
      "2018-02-01 13:13:50.355944 Iter 41000: Training Loss = 11.6565\n",
      "2018-02-01 13:18:58.305631 Iter 41100: Training Loss = 224.3706\n",
      "2018-02-01 13:24:07.220263 Iter 41200: Training Loss = 124.7253\n",
      "2018-02-01 13:29:15.887596 Iter 41300: Training Loss = 397.8228\n",
      "2018-02-01 13:34:24.699696 Iter 41400: Training Loss = 2073.4365\n",
      "2018-02-01 13:39:34.142401 Iter 41500: Training Loss = 2695.6680\n",
      "2018-02-01 13:44:41.968014 Iter 41600: Training Loss = 4181.4683\n",
      "2018-02-01 13:49:51.013082 Iter 41700: Training Loss = 4343.2373\n",
      "2018-02-01 13:55:02.480450 Iter 41800: Training Loss = 1214.1328\n",
      "2018-02-01 14:00:13.445247 Iter 41900: Training Loss = 459.9630\n",
      "2018-02-01 14:05:24.513282 Iter 42000: Training Loss = 4013.1162\n",
      "2018-02-01 14:10:35.418674 Iter 42100: Training Loss = 1903.4730\n",
      "2018-02-01 14:15:46.166732 Iter 42200: Training Loss = 1434.8354\n",
      "2018-02-01 14:20:56.855834 Iter 42300: Training Loss = 6560.1812\n",
      "2018-02-01 14:26:07.478182 Iter 42400: Training Loss = 2214.7686\n",
      "2018-02-01 14:31:18.173412 Iter 42500: Training Loss = 2105.3196\n",
      "2018-02-01 14:36:28.900990 Iter 42600: Training Loss = 1248.2416\n",
      "2018-02-01 14:41:39.509002 Iter 42700: Training Loss = 1713.9091\n",
      "2018-02-01 14:46:49.814958 Iter 42800: Training Loss = 1781.8402\n",
      "2018-02-01 14:52:00.052370 Iter 42900: Training Loss = 139.2783\n",
      "2018-02-01 14:57:10.138778 Iter 43000: Training Loss = 1153.2369\n",
      "2018-02-01 15:02:19.814110 Iter 43100: Training Loss = 991.6226\n",
      "2018-02-01 15:07:29.456760 Iter 43200: Training Loss = 525.7216\n",
      "2018-02-01 15:12:39.152843 Iter 43300: Training Loss = 351.5569\n",
      "2018-02-01 15:17:48.867495 Iter 43400: Training Loss = 299.8214\n",
      "2018-02-01 15:22:58.614893 Iter 43500: Training Loss = 2580.8438\n",
      "2018-02-01 15:28:08.459611 Iter 43600: Training Loss = 1167.7209\n",
      "2018-02-01 15:33:18.193949 Iter 43700: Training Loss = 180.0735\n",
      "2018-02-01 15:38:27.907643 Iter 43800: Training Loss = 1374.4872\n",
      "2018-02-01 15:43:37.679655 Iter 43900: Training Loss = 1312.7997\n",
      "2018-02-01 15:48:47.655370 Iter 44000: Training Loss = 1762.7064\n",
      "2018-02-01 15:53:57.666476 Iter 44100: Training Loss = 877.2939\n",
      "2018-02-01 15:59:07.867489 Iter 44200: Training Loss = 1230.7665\n",
      "2018-02-01 16:04:18.042823 Iter 44300: Training Loss = 527.5059\n",
      "2018-02-01 16:09:28.130859 Iter 44400: Training Loss = 4800.1509\n",
      "2018-02-01 16:14:38.020299 Iter 44500: Training Loss = 2845.0159\n",
      "2018-02-01 16:19:48.189041 Iter 44600: Training Loss = 257.4416\n",
      "2018-02-01 16:24:58.440230 Iter 44700: Training Loss = 2191.6370\n",
      "2018-02-01 16:30:08.765360 Iter 44800: Training Loss = 621.9987\n",
      "2018-02-01 16:35:19.243296 Iter 44900: Training Loss = 1899.4232\n",
      "2018-02-01 16:40:29.800026 Iter 45000: Training Loss = 1930.3331\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-45000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-45000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-02-01 16:45:44.104479 Iter 45100: Training Loss = 0.4088\n",
      "2018-02-01 16:50:55.779447 Iter 45200: Training Loss = 750.8162\n",
      "2018-02-01 16:56:07.414582 Iter 45300: Training Loss = 2302.9490\n",
      "2018-02-01 17:01:19.265632 Iter 45400: Training Loss = 3132.6301\n",
      "2018-02-01 17:06:30.903599 Iter 45500: Training Loss = 1459.1300\n",
      "2018-02-01 17:11:42.837958 Iter 45600: Training Loss = 2140.6804\n",
      "2018-02-01 17:16:54.583203 Iter 45700: Training Loss = 2253.5552\n",
      "2018-02-01 17:22:05.287269 Iter 45800: Training Loss = 1032.7069\n",
      "2018-02-01 17:27:15.671825 Iter 45900: Training Loss = 954.1885\n",
      "2018-02-01 17:32:26.101411 Iter 46000: Training Loss = 1567.7511\n",
      "2018-02-01 17:37:35.995277 Iter 46100: Training Loss = 1361.3478\n",
      "2018-02-01 17:42:45.046858 Iter 46200: Training Loss = 3688.9390\n",
      "2018-02-01 17:47:54.415659 Iter 46300: Training Loss = 910.2719\n",
      "2018-02-01 17:53:03.714443 Iter 46400: Training Loss = 576.1975\n",
      "2018-02-01 17:58:12.646903 Iter 46500: Training Loss = 337.3216\n",
      "2018-02-01 18:03:21.651449 Iter 46600: Training Loss = 907.4417\n",
      "2018-02-01 18:08:30.791089 Iter 46700: Training Loss = 2066.4534\n",
      "2018-02-01 18:13:39.970758 Iter 46800: Training Loss = 938.5623\n",
      "2018-02-01 18:18:49.384593 Iter 46900: Training Loss = 161.3423\n",
      "2018-02-01 18:23:58.539246 Iter 47000: Training Loss = 1947.3262\n",
      "2018-02-01 18:29:07.786962 Iter 47100: Training Loss = 2054.3594\n",
      "2018-02-01 18:34:17.508014 Iter 47200: Training Loss = 1138.2236\n",
      "2018-02-01 18:39:26.503326 Iter 47300: Training Loss = 722.3050\n",
      "2018-02-01 18:44:34.176202 Iter 47400: Training Loss = 1355.1034\n",
      "2018-02-01 18:49:42.218335 Iter 47500: Training Loss = 334.9878\n",
      "2018-02-01 18:54:50.856618 Iter 47600: Training Loss = 341.0543\n",
      "2018-02-01 18:59:59.696045 Iter 47700: Training Loss = 494.1223\n",
      "2018-02-01 19:05:08.432400 Iter 47800: Training Loss = 182.2528\n",
      "2018-02-01 19:10:17.189769 Iter 47900: Training Loss = 1193.8585\n",
      "2018-02-01 19:15:26.005179 Iter 48000: Training Loss = 1807.1185\n",
      "2018-02-01 19:20:34.854616 Iter 48100: Training Loss = 1447.5284\n",
      "2018-02-01 19:25:43.680031 Iter 48200: Training Loss = 252.9626\n",
      "2018-02-01 19:30:52.578505 Iter 48300: Training Loss = 485.8509\n",
      "2018-02-01 19:36:01.266821 Iter 48400: Training Loss = 224.8193\n",
      "2018-02-01 19:41:09.691954 Iter 48500: Training Loss = 664.4562\n",
      "2018-02-01 19:46:18.311945 Iter 48600: Training Loss = 1950.5671\n",
      "2018-02-01 19:51:26.251935 Iter 48700: Training Loss = 1953.9207\n",
      "2018-02-01 19:56:33.888546 Iter 48800: Training Loss = 924.1735\n",
      "2018-02-01 20:01:41.262875 Iter 48900: Training Loss = 1755.1198\n",
      "2018-02-01 20:06:48.878152 Iter 49000: Training Loss = 764.7027\n",
      "2018-02-01 20:11:56.878624 Iter 49100: Training Loss = 2072.4609\n",
      "2018-02-01 20:17:04.507687 Iter 49200: Training Loss = 3122.6301\n",
      "2018-02-01 20:22:12.194470 Iter 49300: Training Loss = 245.1895\n",
      "2018-02-01 20:27:19.707572 Iter 49400: Training Loss = 92.3267\n",
      "2018-02-01 20:32:27.214887 Iter 49500: Training Loss = 2886.7917\n",
      "2018-02-01 20:37:35.086314 Iter 49600: Training Loss = 610.7500\n",
      "2018-02-01 20:42:42.710146 Iter 49700: Training Loss = 2605.5012\n",
      "2018-02-01 20:47:50.735567 Iter 49800: Training Loss = 789.3651\n",
      "2018-02-01 20:52:59.808203 Iter 49900: Training Loss = 2357.7737\n",
      "2018-02-01 20:58:09.079894 Iter 50000: Training Loss = 889.5798\n",
      "INFO:tensorflow:models\\fstream_parent\\fstream_parent.ckpt-50000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models\\fstream_parent\\fstream_parent.ckpt-50000\n",
      "Finished training.\n"
     ]
    }
   ],
   "source": [
    "# Train the network without side outputs supervision\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        global_step = tf.Variable(max_training_iters_2, name='global_step', trainable=False)\n",
    "        learning_rate = tf.train.piecewise_constant(global_step, boundaries, values)\n",
    "        model.train_parent(dataset, imagenet_ckpt, 3, learning_rate, logs_path, max_training_iters_3, save_step,\n",
    "                           display_step, global_step, segnet_stream, iter_mean_grad=iter_mean_grad, resume_training=True,\n",
    "                           test_image_path=test_image, ckpt_name=ckpt_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training losses & learning rate\n",
    "You should see training curves similar to the following:\n",
    "![](img/fstream_parent_dsn_2_loss.png)\n",
    "![](img/fstream_parent_dsn_3_loss.png)\n",
    "![](img/fstream_parent_dsn_4_loss.png)\n",
    "![](img/fstream_parent_dsn_5_loss.png)\n",
    "![](img/fstream_parent_main_loss.png)\n",
    "![](img/fstream_parent_total_loss.png)\n",
    "![](img/fstream_parent_learning_rate.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}