tfoptflow/pwcnet_train_sm-6-2-multisteps-chairsthingsmix.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PWC-Net-small model training (with multisteps learning rate schedule)\n",
    "=========================================================\n",
    "\n",
    "In this notebook, we:\n",
    "- Use a PWC-Net-small model (no dense or residual connections), 6 level pyramid, uspample level 2 by 4 as the final flow prediction\n",
    "- Train the model on a mix of the `FlyingChairs` and `FlyingThings3DHalfRes` dataset using a S<sub>long</sub> schedule described in [[2018a]](#2018a)\n",
    "- The S<sub>long</sub> schedule is borrowed from [[2016a]](#2016a) and looks as follows:\n",
    "\n",
    "![](img/lr_multisteps.png)\n",
    "\n",
    "Below, look for `TODO` references and customize this notebook based on your own machine setup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\n",
    "- [2018a]<a name=\"2018a\"></a> Sun et al. 2018. PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume. [[arXiv]](https://arxiv.org/abs/1709.02371) [[web]](http://research.nvidia.com/publication/2018-02_PWC-Net%3A-CNNs-for) [[PyTorch (Official)]](https://github.com/NVlabs/PWC-Net/tree/master/PyTorch) [[Caffe (Official)]](https://github.com/NVlabs/PWC-Net/tree/master/Caffe)\n",
    "- [2016a]<a name=\"2016a\"></a> Ilg et al. 2016. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. [[arXiv]](https://arxiv.org/abs/1612.01925) [[PyTorch (Official)]](https://github.com/NVIDIA/flownet2-pytorch) [[TensorFlow]](https://github.com/sampepose/flownet2-tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "pwcnet_train.ipynb\n",
    "\n",
    "PWC-Net model training.\n",
    "\n",
    "Written by Phil Ferriere\n",
    "\n",
    "Licensed under the MIT License (see LICENSE for details)\n",
    "\n",
    "Tensorboard:\n",
    "    [win] tensorboard --logdir=E:\\\\repos\\\\tf-optflow\\\\tfoptflow\\\\pwcnet-sm-6-2-multisteps-chairsthingsmix\n",
    "    [ubu] tensorboard --logdir=/media/EDrive/repos/tf-optflow/tfoptflow/pwcnet-sm-6-2-multisteps-chairsthingsmix\n",
    "\"\"\"\n",
    "from __future__ import absolute_import, division, print_function\n",
    "import sys\n",
    "from copy import deepcopy\n",
    "\n",
    "from dataset_base import _DEFAULT_DS_TRAIN_OPTIONS\n",
    "from dataset_flyingchairs import FlyingChairsDataset\n",
    "from dataset_flyingthings3d import FlyingThings3DHalfResDataset\n",
    "from dataset_mixer import MixedDataset\n",
    "from model_pwcnet import ModelPWCNet, _DEFAULT_PWCNET_TRAIN_OPTIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Set this first!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: You MUST set dataset_root to the correct path on your machine!\n",
    "if sys.platform.startswith(\"win\"):\n",
    "    _DATASET_ROOT = 'E:/datasets/'\n",
    "else:\n",
    "    _DATASET_ROOT = '/media/EDrive/datasets/'\n",
    "_FLYINGCHAIRS_ROOT = _DATASET_ROOT + 'FlyingChairs_release'\n",
    "_FLYINGTHINGS3DHALFRES_ROOT = _DATASET_ROOT + 'FlyingThings3D_HalfRes'\n",
    "    \n",
    "# TODO: You MUST adjust the settings below based on the number of GPU(s) used for training\n",
    "# Set controller device and devices\n",
    "# A one-gpu setup would be something like controller='/device:GPU:0' and gpu_devices=['/device:GPU:0']\n",
    "# Here, we use a dual-GPU setup, as shown below\n",
    "gpu_devices = ['/device:GPU:0', '/device:GPU:1']\n",
    "controller = '/device:CPU:0'\n",
    "\n",
    "# TODO: You MUST adjust this setting below based on the amount of memory on your GPU(s)\n",
    "# Batch size\n",
    "batch_size = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-train on `FlyingChairs+FlyingThings3DHalfRes` mix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: You MUST set the batch size based on the capabilities of your GPU(s) \n",
    "#  Load train dataset\n",
    "ds_opts = deepcopy(_DEFAULT_DS_TRAIN_OPTIONS)\n",
    "ds_opts['in_memory'] = False                          # Too many samples to keep in memory at once, so don't preload them\n",
    "ds_opts['aug_type'] = 'heavy'                         # Apply all supported augmentations\n",
    "ds_opts['batch_size'] = batch_size * len(gpu_devices) # Use a multiple of 8; here, 16 for dual-GPU mode (Titan X & 1080 Ti)\n",
    "ds_opts['crop_preproc'] = (256, 448)                  # Crop to a smaller input size\n",
    "ds1 = FlyingChairsDataset(mode='train_with_val', ds_root=_FLYINGCHAIRS_ROOT, options=ds_opts)\n",
    "ds_opts['type'] = 'into_future'\n",
    "ds2 = FlyingThings3DHalfResDataset(mode='train_with_val', ds_root=_FLYINGTHINGS3DHALFRES_ROOT, options=ds_opts)\n",
    "ds = MixedDataset(mode='train_with_val', datasets=[ds1, ds2], options=ds_opts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dataset Configuration:\n",
      "  verbose              False\n",
      "  in_memory            False\n",
      "  crop_preproc         (256, 448)\n",
      "  scale_preproc        None\n",
      "  input_channels       3\n",
      "  tb_test_imgs         False\n",
      "  random_seed          1969\n",
      "  val_split            0.03\n",
      "  aug_type             heavy\n",
      "  aug_labels           True\n",
      "  fliplr               0.5\n",
      "  flipud               0.5\n",
      "  translate            (0.5, 0.05)\n",
      "  scale                (0.5, 0.05)\n",
      "  batch_size           16\n",
      "  type                 into_future\n",
      "  mode                 train_with_val\n",
      "  train size           41282\n",
      "  val size             1230\n"
     ]
    }
   ],
   "source": [
    "# Display dataset configuration\n",
    "ds.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start from the default options\n",
    "nn_opts = deepcopy(_DEFAULT_PWCNET_TRAIN_OPTIONS)\n",
    "nn_opts['verbose'] = True\n",
    "nn_opts['ckpt_dir'] = './pwcnet-sm-6-2-multisteps-chairsthingsmix/'\n",
    "nn_opts['batch_size'] = ds_opts['batch_size']\n",
    "nn_opts['x_shape'] = [2, ds_opts['crop_preproc'][0], ds_opts['crop_preproc'][1], 3]\n",
    "nn_opts['y_shape'] = [ds_opts['crop_preproc'][0], ds_opts['crop_preproc'][1], 2]\n",
    "nn_opts['use_tf_data'] = True # Use tf.data reader\n",
    "nn_opts['gpu_devices'] = gpu_devices\n",
    "nn_opts['controller'] = controller\n",
    "\n",
    "# Use the PWC-Net-small model in quarter-resolution mode\n",
    "nn_opts['use_dense_cx'] = False\n",
    "nn_opts['use_res_cx'] = False\n",
    "nn_opts['pyr_lvls'] = 6\n",
    "nn_opts['flow_pred_lvl'] = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the learning rate schedule. This schedule is for a single GPU using a batch size of 8.\n",
    "# Below,we adjust the schedule to the size of the batch and the number of GPUs.\n",
    "nn_opts['lr_policy'] = 'multisteps'\n",
    "nn_opts['lr_boundaries'] = [400000, 600000, 800000, 1000000, 1200000]\n",
    "nn_opts['lr_values'] = [0.0001, 5e-05, 2.5e-05, 1.25e-05, 6.25e-06, 3.125e-06]\n",
    "nn_opts['max_steps'] = 1200000\n",
    "\n",
    "# Below, we adjust the schedule to the size of the batch and our number of GPUs (2).\n",
    "nn_opts['max_steps'] = int(nn_opts['max_steps'] * 8 / ds_opts['batch_size'])\n",
    "nn_opts['lr_boundaries'] = [int(boundary * 8 / ds_opts['batch_size']) for boundary in nn_opts['lr_boundaries']]\n",
    "\n",
    "# More options\n",
    "nn_opts['max_to_keep'] = 50\n",
    "nn_opts['display_step'] = 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building model towers...\n",
      "  Building tower_0...\n",
      "WARNING:tensorflow:From e:\\toolkits.win\\anaconda3-5.2.0\\envs\\dlwin36\\lib\\site-packages\\tensorflow\\python\\keras\\initializers.py:104: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with distribution=normal is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "`normal` is a deprecated alias for `truncated_normal`\n",
      "  ...tower_0 built.\n",
      "  Building tower_1...\n",
      "  ...tower_1 built.\n",
      "... model towers built.\n",
      "Initializing model from previous checkpoint ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-467000 to resume training...\n",
      "\n",
      "INFO:tensorflow:Restoring parameters from ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-467000\n",
      "... model initialized\n",
      "\n",
      "Model Configuration:\n",
      "  verbose                True\n",
      "  ckpt_dir               ./pwcnet-sm-6-2-multisteps-chairsthingsmix/\n",
      "  max_to_keep            50\n",
      "  x_dtype                <dtype: 'float32'>\n",
      "  x_shape                [2, 256, 448, 3]\n",
      "  y_dtype                <dtype: 'float32'>\n",
      "  y_shape                [256, 448, 2]\n",
      "  train_mode             train\n",
      "  display_step           1000\n",
      "  snapshot_step          1000\n",
      "  val_step               1000\n",
      "  val_batch_size         -1\n",
      "  tb_val_imgs            pyramid\n",
      "  tb_test_imgs           None\n",
      "  gpu_devices            ['/device:GPU:0', '/device:GPU:1']\n",
      "  controller             /device:CPU:0\n",
      "  use_tf_data            True\n",
      "  use_mixed_precision    False\n",
      "  loss_scaler            8.0\n",
      "  batch_size             16\n",
      "  lr_policy              multisteps\n",
      "  max_steps              600000\n",
      "  lr_boundaries          [200000, 300000, 400000, 500000, 600000]\n",
      "  lr_values              [0.0001, 5e-05, 2.5e-05, 1.25e-05, 6.25e-06, 3.125e-06]\n",
      "  loss_fn                loss_multiscale\n",
      "  alphas                 [0.32, 0.08, 0.02, 0.01, 0.005, 0.0025]\n",
      "  gamma                  0.0004\n",
      "  q                      1.0\n",
      "  epsilon                0.0\n",
      "  pyr_lvls               6\n",
      "  flow_pred_lvl          2\n",
      "  search_range           4\n",
      "  use_dense_cx           False\n",
      "  use_res_cx             False\n",
      "  mode                   train_with_val\n",
      "  trainable params       0.0\n"
     ]
    }
   ],
   "source": [
    "# Instantiate the model and display the model configuration\n",
    "nn = ModelPWCNet(mode='train_with_val', options=nn_opts, dataset=ds)\n",
    "nn.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training from scratch...\n",
      "2018-09-20 20:45:09 Iter 1000 [Train]: loss=199.75, epe=15.70, lr=0.000100, samples/sec=45.5, sec/step=0.704, eta=4 days, 21:07:47\n",
      "2018-09-20 20:45:30 Iter 1000 [Val]: loss=169.35, epe=13.20\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-1000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-1000\n",
      "2018-09-20 20:58:01 Iter 2000 [Train]: loss=164.27, epe=12.41, lr=0.000100, samples/sec=50.3, sec/step=0.637, eta=4 days, 9:44:55\n",
      "2018-09-20 20:58:13 Iter 2000 [Val]: loss=151.78, epe=11.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-2000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-2000\n",
      "2018-09-20 21:10:43 Iter 3000 [Train]: loss=141.97, epe=10.86, lr=0.000100, samples/sec=50.0, sec/step=0.640, eta=4 days, 10:08:27\n",
      "2018-09-20 21:10:55 Iter 3000 [Val]: loss=130.76, epe=10.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-3000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-3000\n",
      "2018-09-20 21:23:31 Iter 4000 [Train]: loss=128.71, epe=9.85, lr=0.000100, samples/sec=49.6, sec/step=0.645, eta=4 days, 10:46:17\n",
      "2018-09-20 21:23:43 Iter 4000 [Val]: loss=122.97, epe=9.52\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-4000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-4000\n",
      "2018-09-20 21:36:20 Iter 5000 [Train]: loss=119.12, epe=9.10, lr=0.000100, samples/sec=49.5, sec/step=0.646, eta=4 days, 10:49:34\n",
      "2018-09-20 21:36:32 Iter 5000 [Val]: loss=113.13, epe=8.68\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-5000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-5000\n",
      "2018-09-20 21:49:08 Iter 6000 [Train]: loss=115.17, epe=8.80, lr=0.000100, samples/sec=49.5, sec/step=0.647, eta=4 days, 10:41:51\n",
      "2018-09-20 21:49:21 Iter 6000 [Val]: loss=113.06, epe=8.68\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-6000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-6000\n",
      "2018-09-20 22:02:01 Iter 7000 [Train]: loss=110.77, epe=8.43, lr=0.000100, samples/sec=49.2, sec/step=0.650, eta=4 days, 11:05:02\n",
      "2018-09-20 22:02:13 Iter 7000 [Val]: loss=108.67, epe=8.33\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-7000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-7000\n",
      "2018-09-20 22:14:51 Iter 8000 [Train]: loss=107.15, epe=8.12, lr=0.000100, samples/sec=49.4, sec/step=0.647, eta=4 days, 10:26:25\n",
      "2018-09-20 22:15:03 Iter 8000 [Val]: loss=105.40, epe=8.03\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-8000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-8000\n",
      "2018-09-20 22:27:46 Iter 9000 [Train]: loss=105.69, epe=8.00, lr=0.000100, samples/sec=49.0, sec/step=0.653, eta=4 days, 11:09:49\n",
      "2018-09-20 22:27:58 Iter 9000 [Val]: loss=102.56, epe=7.83\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-9000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-9000\n",
      "2018-09-20 22:40:43 Iter 10000 [Train]: loss=102.49, epe=7.73, lr=0.000100, samples/sec=48.8, sec/step=0.655, eta=4 days, 11:21:49\n",
      "2018-09-20 22:40:56 Iter 10000 [Val]: loss=101.80, epe=7.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-10000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-10000\n",
      "2018-09-20 22:53:48 Iter 11000 [Train]: loss=99.81, epe=7.50, lr=0.000100, samples/sec=48.3, sec/step=0.663, eta=4 days, 12:26:30\n",
      "2018-09-20 22:54:01 Iter 11000 [Val]: loss=97.78, epe=7.39\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-11000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-11000\n",
      "2018-09-20 23:06:58 Iter 12000 [Train]: loss=98.29, epe=7.36, lr=0.000100, samples/sec=48.1, sec/step=0.666, eta=4 days, 12:42:43\n",
      "2018-09-20 23:07:11 Iter 12000 [Val]: loss=98.82, epe=7.49\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-12000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-12000\n",
      "2018-09-20 23:20:09 Iter 13000 [Train]: loss=96.10, epe=7.19, lr=0.000100, samples/sec=48.0, sec/step=0.667, eta=4 days, 12:46:31\n",
      "2018-09-20 23:20:22 Iter 13000 [Val]: loss=93.50, epe=7.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-13000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-13000\n",
      "2018-09-20 23:33:28 Iter 14000 [Train]: loss=95.07, epe=7.09, lr=0.000100, samples/sec=47.4, sec/step=0.675, eta=4 days, 13:51:14\n",
      "2018-09-20 23:33:41 Iter 14000 [Val]: loss=92.29, epe=6.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-14000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-14000\n",
      "2018-09-20 23:46:49 Iter 15000 [Train]: loss=92.16, epe=6.84, lr=0.000100, samples/sec=47.3, sec/step=0.677, eta=4 days, 13:59:54\n",
      "2018-09-20 23:47:02 Iter 15000 [Val]: loss=93.60, epe=7.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-15000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-15000\n",
      "2018-09-21 00:00:22 Iter 16000 [Train]: loss=91.92, epe=6.82, lr=0.000100, samples/sec=46.6, sec/step=0.686, eta=4 days, 15:20:26\n",
      "2018-09-21 00:00:35 Iter 16000 [Val]: loss=93.26, epe=7.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-16000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-16000\n",
      "2018-09-21 00:13:56 Iter 17000 [Train]: loss=90.40, epe=6.69, lr=0.000100, samples/sec=46.4, sec/step=0.689, eta=4 days, 15:36:59\n",
      "2018-09-21 00:14:10 Iter 17000 [Val]: loss=92.28, epe=6.88\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-17000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-17000\n",
      "2018-09-21 00:27:39 Iter 18000 [Train]: loss=89.12, epe=6.58, lr=0.000100, samples/sec=45.9, sec/step=0.697, eta=4 days, 16:38:45\n",
      "2018-09-21 00:27:53 Iter 18000 [Val]: loss=85.80, epe=6.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-18000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-18000\n",
      "2018-09-21 00:41:31 Iter 19000 [Train]: loss=88.01, epe=6.48, lr=0.000100, samples/sec=45.3, sec/step=0.706, eta=4 days, 17:53:24\n",
      "2018-09-21 00:41:45 Iter 19000 [Val]: loss=89.78, epe=6.64\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-19000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-19000\n",
      "2018-09-21 00:55:29 Iter 20000 [Train]: loss=86.63, epe=6.37, lr=0.000100, samples/sec=45.1, sec/step=0.710, eta=4 days, 18:25:23\n",
      "2018-09-21 00:55:42 Iter 20000 [Val]: loss=84.92, epe=6.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-20000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-20000\n",
      "2018-09-21 01:09:32 Iter 21000 [Train]: loss=84.90, epe=6.22, lr=0.000100, samples/sec=44.7, sec/step=0.716, eta=4 days, 19:07:03\n",
      "2018-09-21 01:09:46 Iter 21000 [Val]: loss=82.99, epe=6.13\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-21000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-21000\n",
      "2018-09-21 01:23:44 Iter 22000 [Train]: loss=85.83, epe=6.29, lr=0.000100, samples/sec=44.4, sec/step=0.721, eta=4 days, 19:41:06\n",
      "2018-09-21 01:23:58 Iter 22000 [Val]: loss=84.92, epe=6.28\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-22000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-22000\n",
      "2018-09-21 01:37:58 Iter 23000 [Train]: loss=83.80, epe=6.12, lr=0.000100, samples/sec=44.1, sec/step=0.726, eta=4 days, 20:24:12\n",
      "2018-09-21 01:38:13 Iter 23000 [Val]: loss=81.13, epe=5.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-23000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-23000\n",
      "2018-09-21 01:52:21 Iter 24000 [Train]: loss=82.52, epe=6.01, lr=0.000100, samples/sec=43.6, sec/step=0.733, eta=4 days, 21:18:16\n",
      "2018-09-21 01:52:35 Iter 24000 [Val]: loss=86.37, epe=6.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-24000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-24000\n",
      "2018-09-21 02:06:51 Iter 25000 [Train]: loss=83.32, epe=6.06, lr=0.000100, samples/sec=43.2, sec/step=0.741, eta=4 days, 22:24:55\n",
      "2018-09-21 02:07:06 Iter 25000 [Val]: loss=81.30, epe=5.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-25000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-25000\n",
      "2018-09-21 02:21:34 Iter 26000 [Train]: loss=80.82, epe=5.86, lr=0.000100, samples/sec=42.5, sec/step=0.752, eta=4 days, 23:57:42\n",
      "2018-09-21 02:21:49 Iter 26000 [Val]: loss=85.58, epe=6.33\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-26000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-26000\n",
      "2018-09-21 02:36:19 Iter 27000 [Train]: loss=80.42, epe=5.83, lr=0.000100, samples/sec=42.5, sec/step=0.753, eta=4 days, 23:54:47\n",
      "2018-09-21 02:36:35 Iter 27000 [Val]: loss=79.27, epe=5.78\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-27000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-27000\n",
      "2018-09-21 02:51:14 Iter 28000 [Train]: loss=79.87, epe=5.78, lr=0.000100, samples/sec=42.2, sec/step=0.759, eta=5 days, 0:36:46\n",
      "2018-09-21 02:51:29 Iter 28000 [Val]: loss=89.01, epe=6.59\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-28000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-28000\n",
      "2018-09-21 03:06:05 Iter 29000 [Train]: loss=79.14, epe=5.72, lr=0.000100, samples/sec=42.2, sec/step=0.758, eta=5 days, 0:13:00\n",
      "2018-09-21 03:06:20 Iter 29000 [Val]: loss=79.44, epe=5.79\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-29000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-29000\n",
      "2018-09-21 03:21:03 Iter 30000 [Train]: loss=77.60, epe=5.59, lr=0.000100, samples/sec=41.8, sec/step=0.766, eta=5 days, 1:15:52\n",
      "2018-09-21 03:21:18 Iter 30000 [Val]: loss=80.27, epe=5.87\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-30000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-30000\n",
      "2018-09-21 03:36:04 Iter 31000 [Train]: loss=77.48, epe=5.58, lr=0.000100, samples/sec=41.7, sec/step=0.768, eta=5 days, 1:22:23\n",
      "2018-09-21 03:36:19 Iter 31000 [Val]: loss=76.78, epe=5.59\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-31000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-31000\n",
      "2018-09-21 03:51:12 Iter 32000 [Train]: loss=76.99, epe=5.54, lr=0.000100, samples/sec=41.3, sec/step=0.775, eta=5 days, 2:12:39\n",
      "2018-09-21 03:51:27 Iter 32000 [Val]: loss=79.57, epe=5.78\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-32000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-32000\n",
      "2018-09-21 04:06:26 Iter 33000 [Train]: loss=76.79, epe=5.51, lr=0.000100, samples/sec=41.0, sec/step=0.781, eta=5 days, 2:59:53\n",
      "2018-09-21 04:06:42 Iter 33000 [Val]: loss=81.17, epe=5.93\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-33000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-33000\n",
      "2018-09-21 04:21:50 Iter 34000 [Train]: loss=76.45, epe=5.49, lr=0.000100, samples/sec=40.6, sec/step=0.789, eta=5 days, 4:03:28\n",
      "2018-09-21 04:22:06 Iter 34000 [Val]: loss=76.81, epe=5.56\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-34000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-34000\n",
      "2018-09-21 04:37:22 Iter 35000 [Train]: loss=74.63, epe=5.34, lr=0.000100, samples/sec=40.2, sec/step=0.797, eta=5 days, 5:02:17\n",
      "2018-09-21 04:37:39 Iter 35000 [Val]: loss=85.63, epe=6.29\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-35000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-35000\n",
      "2018-09-21 04:52:58 Iter 36000 [Train]: loss=75.77, epe=5.43, lr=0.000100, samples/sec=40.0, sec/step=0.800, eta=5 days, 5:17:28\n",
      "2018-09-21 04:53:14 Iter 36000 [Val]: loss=69.19, epe=4.93\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-36000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-36000\n",
      "2018-09-21 05:08:37 Iter 37000 [Train]: loss=73.77, epe=5.27, lr=0.000100, samples/sec=40.1, sec/step=0.798, eta=5 days, 4:52:35\n",
      "2018-09-21 05:08:53 Iter 37000 [Val]: loss=77.44, epe=5.60\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-37000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-37000\n",
      "2018-09-21 05:24:20 Iter 38000 [Train]: loss=73.22, epe=5.21, lr=0.000100, samples/sec=39.7, sec/step=0.807, eta=5 days, 5:58:51\n",
      "2018-09-21 05:24:37 Iter 38000 [Val]: loss=78.36, epe=5.69\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-38000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-38000\n",
      "2018-09-21 05:40:15 Iter 39000 [Train]: loss=74.38, epe=5.31, lr=0.000100, samples/sec=39.2, sec/step=0.817, eta=5 days, 7:18:20\n",
      "2018-09-21 05:40:32 Iter 39000 [Val]: loss=76.53, epe=5.50\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-39000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-39000\n",
      "2018-09-21 05:56:08 Iter 40000 [Train]: loss=72.41, epe=5.15, lr=0.000100, samples/sec=39.3, sec/step=0.815, eta=5 days, 6:44:10\n",
      "2018-09-21 05:56:25 Iter 40000 [Val]: loss=70.69, epe=5.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-40000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-40000\n",
      "2018-09-21 06:12:04 Iter 41000 [Train]: loss=71.94, epe=5.11, lr=0.000100, samples/sec=39.1, sec/step=0.819, eta=5 days, 7:09:19\n",
      "2018-09-21 06:12:21 Iter 41000 [Val]: loss=75.74, epe=5.44\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-41000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-41000\n",
      "2018-09-21 06:28:03 Iter 42000 [Train]: loss=72.37, epe=5.14, lr=0.000100, samples/sec=38.9, sec/step=0.822, eta=5 days, 7:23:19\n",
      "2018-09-21 06:28:20 Iter 42000 [Val]: loss=69.71, epe=4.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-42000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-42000\n",
      "2018-09-21 06:44:07 Iter 43000 [Train]: loss=71.36, epe=5.06, lr=0.000100, samples/sec=38.8, sec/step=0.825, eta=5 days, 7:41:41\n",
      "2018-09-21 06:44:24 Iter 43000 [Val]: loss=70.47, epe=5.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-43000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-43000\n",
      "2018-09-21 07:00:16 Iter 44000 [Train]: loss=70.63, epe=4.99, lr=0.000100, samples/sec=38.6, sec/step=0.830, eta=5 days, 8:10:40\n",
      "2018-09-21 07:00:33 Iter 44000 [Val]: loss=76.03, epe=5.48\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-44000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-44000\n",
      "2018-09-21 07:16:26 Iter 45000 [Train]: loss=70.69, epe=5.00, lr=0.000100, samples/sec=38.5, sec/step=0.831, eta=5 days, 8:09:41\n",
      "2018-09-21 07:16:43 Iter 45000 [Val]: loss=67.68, epe=4.80\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-45000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-45000\n",
      "2018-09-21 07:32:42 Iter 46000 [Train]: loss=69.45, epe=4.90, lr=0.000100, samples/sec=38.4, sec/step=0.832, eta=5 days, 8:06:21\n",
      "2018-09-21 07:32:59 Iter 46000 [Val]: loss=74.83, epe=5.39\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-46000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-46000\n",
      "2018-09-21 07:48:56 Iter 47000 [Train]: loss=70.64, epe=4.98, lr=0.000100, samples/sec=38.3, sec/step=0.836, eta=5 days, 8:24:20\n",
      "2018-09-21 07:49:13 Iter 47000 [Val]: loss=69.58, epe=4.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-47000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-47000\n",
      "2018-09-21 08:05:16 Iter 48000 [Train]: loss=69.23, epe=4.88, lr=0.000100, samples/sec=38.0, sec/step=0.841, eta=5 days, 9:00:38\n",
      "2018-09-21 08:05:33 Iter 48000 [Val]: loss=70.99, epe=5.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-48000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-48000\n",
      "2018-09-21 08:21:37 Iter 49000 [Train]: loss=69.62, epe=4.90, lr=0.000100, samples/sec=38.0, sec/step=0.841, eta=5 days, 8:45:52\n",
      "2018-09-21 08:21:55 Iter 49000 [Val]: loss=70.20, epe=4.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-49000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-49000\n",
      "2018-09-21 08:37:58 Iter 50000 [Train]: loss=68.93, epe=4.84, lr=0.000100, samples/sec=38.1, sec/step=0.840, eta=5 days, 8:21:17\n",
      "2018-09-21 08:38:15 Iter 50000 [Val]: loss=72.96, epe=5.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-50000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-50000\n",
      "2018-09-21 08:54:28 Iter 51000 [Train]: loss=68.12, epe=4.78, lr=0.000100, samples/sec=37.6, sec/step=0.851, eta=5 days, 9:46:17\n",
      "2018-09-21 08:54:46 Iter 51000 [Val]: loss=65.87, epe=4.64\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-51000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-51000\n",
      "2018-09-21 09:11:01 Iter 52000 [Train]: loss=69.63, epe=4.89, lr=0.000100, samples/sec=37.5, sec/step=0.854, eta=5 days, 10:01:25\n",
      "2018-09-21 09:11:19 Iter 52000 [Val]: loss=70.92, epe=5.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-52000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-52000\n",
      "2018-09-21 09:27:35 Iter 53000 [Train]: loss=67.25, epe=4.71, lr=0.000100, samples/sec=37.5, sec/step=0.853, eta=5 days, 9:36:39\n",
      "2018-09-21 09:27:53 Iter 53000 [Val]: loss=68.40, epe=4.87\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-53000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-53000\n",
      "2018-09-21 09:44:13 Iter 54000 [Train]: loss=68.06, epe=4.77, lr=0.000100, samples/sec=37.4, sec/step=0.856, eta=5 days, 9:50:52\n",
      "2018-09-21 09:44:30 Iter 54000 [Val]: loss=67.23, epe=4.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-54000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-54000\n",
      "2018-09-21 10:00:53 Iter 55000 [Train]: loss=67.52, epe=4.72, lr=0.000100, samples/sec=37.2, sec/step=0.860, eta=5 days, 10:10:20\n",
      "2018-09-21 10:01:11 Iter 55000 [Val]: loss=66.63, epe=4.69\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-55000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-55000\n",
      "2018-09-21 10:17:41 Iter 56000 [Train]: loss=66.55, epe=4.64, lr=0.000100, samples/sec=37.0, sec/step=0.866, eta=5 days, 10:47:34\n",
      "2018-09-21 10:17:59 Iter 56000 [Val]: loss=75.50, epe=5.44\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-56000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-56000\n",
      "2018-09-21 10:34:28 Iter 57000 [Train]: loss=67.43, epe=nan, lr=0.000100, samples/sec=36.9, sec/step=0.866, eta=5 days, 10:40:22\n",
      "2018-09-21 10:34:47 Iter 57000 [Val]: loss=71.48, epe=5.03\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-57000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-57000\n",
      "2018-09-21 10:51:27 Iter 58000 [Train]: loss=66.90, epe=4.66, lr=0.000100, samples/sec=36.9, sec/step=0.868, eta=5 days, 10:43:39\n",
      "2018-09-21 10:51:45 Iter 58000 [Val]: loss=68.23, epe=4.79\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-58000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-58000\n",
      "2018-09-21 11:08:26 Iter 59000 [Train]: loss=66.31, epe=4.61, lr=0.000100, samples/sec=36.5, sec/step=0.877, eta=5 days, 11:51:48\n",
      "2018-09-21 11:08:45 Iter 59000 [Val]: loss=63.15, epe=4.39\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-59000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-59000\n",
      "2018-09-21 11:25:28 Iter 60000 [Train]: loss=66.23, epe=4.61, lr=0.000100, samples/sec=36.4, sec/step=0.879, eta=5 days, 11:48:12\n",
      "2018-09-21 11:25:47 Iter 60000 [Val]: loss=69.78, epe=4.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-60000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-60000\n",
      "2018-09-21 11:42:30 Iter 61000 [Train]: loss=66.20, epe=4.60, lr=0.000100, samples/sec=36.4, sec/step=0.879, eta=5 days, 11:40:36\n",
      "2018-09-21 11:42:49 Iter 61000 [Val]: loss=67.12, epe=4.71\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-61000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-61000\n",
      "2018-09-21 11:59:38 Iter 62000 [Train]: loss=65.29, epe=4.53, lr=0.000100, samples/sec=36.2, sec/step=0.884, eta=5 days, 12:10:10\n",
      "2018-09-21 11:59:56 Iter 62000 [Val]: loss=66.94, epe=4.70\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-62000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-62000\n",
      "2018-09-21 12:16:49 Iter 63000 [Train]: loss=64.89, epe=4.50, lr=0.000100, samples/sec=36.1, sec/step=0.887, eta=5 days, 12:16:15\n",
      "2018-09-21 12:17:08 Iter 63000 [Val]: loss=74.51, epe=5.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-63000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-63000\n",
      "2018-09-21 12:34:13 Iter 64000 [Train]: loss=66.02, epe=4.58, lr=0.000100, samples/sec=35.6, sec/step=0.900, eta=5 days, 13:58:48\n",
      "2018-09-21 12:34:32 Iter 64000 [Val]: loss=61.80, epe=4.32\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-64000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-64000\n",
      "2018-09-21 12:51:32 Iter 65000 [Train]: loss=65.07, epe=4.50, lr=0.000100, samples/sec=35.8, sec/step=0.894, eta=5 days, 12:49:39\n",
      "2018-09-21 12:51:51 Iter 65000 [Val]: loss=68.84, epe=4.85\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-65000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-65000\n",
      "2018-09-21 13:08:53 Iter 66000 [Train]: loss=65.09, epe=4.50, lr=0.000100, samples/sec=35.7, sec/step=0.896, eta=5 days, 12:54:16\n",
      "2018-09-21 13:09:11 Iter 66000 [Val]: loss=64.07, epe=4.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-66000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-66000\n",
      "2018-09-21 13:26:21 Iter 67000 [Train]: loss=63.95, epe=4.41, lr=0.000100, samples/sec=35.4, sec/step=0.905, eta=5 days, 13:55:50\n",
      "2018-09-21 13:26:40 Iter 67000 [Val]: loss=69.40, epe=4.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-67000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-67000\n",
      "2018-09-21 13:43:49 Iter 68000 [Train]: loss=65.02, epe=4.50, lr=0.000100, samples/sec=35.4, sec/step=0.904, eta=5 days, 13:35:59\n",
      "2018-09-21 13:44:09 Iter 68000 [Val]: loss=66.80, epe=4.66\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-68000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-68000\n",
      "2018-09-21 14:01:20 Iter 69000 [Train]: loss=64.14, epe=4.41, lr=0.000100, samples/sec=35.3, sec/step=0.906, eta=5 days, 13:40:50\n",
      "2018-09-21 14:01:40 Iter 69000 [Val]: loss=67.00, epe=4.69\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-69000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-69000\n",
      "2018-09-21 14:18:53 Iter 70000 [Train]: loss=63.85, epe=4.40, lr=0.000100, samples/sec=35.2, sec/step=0.908, eta=5 days, 13:39:57\n",
      "2018-09-21 14:19:13 Iter 70000 [Val]: loss=63.80, epe=4.44\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-70000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-70000\n",
      "2018-09-21 14:36:32 Iter 71000 [Train]: loss=63.20, epe=4.34, lr=0.000100, samples/sec=35.1, sec/step=0.912, eta=5 days, 14:03:35\n",
      "2018-09-21 14:36:51 Iter 71000 [Val]: loss=65.05, epe=4.54\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-71000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-71000\n",
      "2018-09-21 14:54:42 Iter 72000 [Train]: loss=62.68, epe=4.30, lr=0.000100, samples/sec=34.8, sec/step=0.919, eta=5 days, 14:51:12\n",
      "2018-09-21 14:55:02 Iter 72000 [Val]: loss=67.49, epe=4.73\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-72000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-72000\n",
      "2018-09-21 15:12:26 Iter 73000 [Train]: loss=63.12, epe=4.33, lr=0.000100, samples/sec=34.8, sec/step=0.919, eta=5 days, 14:30:35\n",
      "2018-09-21 15:12:46 Iter 73000 [Val]: loss=64.49, epe=4.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-73000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-73000\n",
      "2018-09-21 15:30:16 Iter 74000 [Train]: loss=63.80, epe=4.39, lr=0.000100, samples/sec=34.6, sec/step=0.925, eta=5 days, 15:09:42\n",
      "2018-09-21 15:30:36 Iter 74000 [Val]: loss=63.51, epe=4.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-74000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-74000\n",
      "2018-09-21 15:48:07 Iter 75000 [Train]: loss=63.86, epe=4.39, lr=0.000100, samples/sec=34.6, sec/step=0.924, eta=5 days, 14:46:42\n",
      "2018-09-21 15:48:27 Iter 75000 [Val]: loss=63.88, epe=4.42\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-75000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-75000\n",
      "2018-09-21 16:06:02 Iter 76000 [Train]: loss=63.58, epe=4.36, lr=0.000100, samples/sec=34.5, sec/step=0.929, eta=5 days, 15:11:40\n",
      "2018-09-21 16:06:22 Iter 76000 [Val]: loss=62.04, epe=4.30\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-76000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-76000\n",
      "2018-09-21 16:24:02 Iter 77000 [Train]: loss=62.62, epe=4.29, lr=0.000100, samples/sec=34.2, sec/step=0.937, eta=5 days, 16:03:29\n",
      "2018-09-21 16:24:23 Iter 77000 [Val]: loss=65.41, epe=4.52\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-77000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-77000\n",
      "2018-09-21 16:42:08 Iter 78000 [Train]: loss=61.49, epe=4.20, lr=0.000100, samples/sec=34.0, sec/step=0.941, eta=5 days, 16:22:31\n",
      "2018-09-21 16:42:28 Iter 78000 [Val]: loss=64.47, epe=4.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-78000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-78000\n",
      "2018-09-21 17:00:13 Iter 79000 [Train]: loss=63.10, epe=4.32, lr=0.000100, samples/sec=34.0, sec/step=0.941, eta=5 days, 16:09:50\n",
      "2018-09-21 17:00:34 Iter 79000 [Val]: loss=63.42, epe=4.40\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-79000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-79000\n",
      "2018-09-21 17:18:22 Iter 80000 [Train]: loss=62.41, epe=4.27, lr=0.000100, samples/sec=33.9, sec/step=0.943, eta=5 days, 16:14:02\n",
      "2018-09-21 17:18:42 Iter 80000 [Val]: loss=66.30, epe=4.59\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-80000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-80000\n",
      "2018-09-21 17:36:34 Iter 81000 [Train]: loss=61.55, epe=4.19, lr=0.000100, samples/sec=33.8, sec/step=0.945, eta=5 days, 16:18:17\n",
      "2018-09-21 17:36:55 Iter 81000 [Val]: loss=62.50, epe=4.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-81000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-81000\n",
      "2018-09-21 17:54:44 Iter 82000 [Train]: loss=62.03, epe=nan, lr=0.000100, samples/sec=33.9, sec/step=0.944, eta=5 days, 15:46:44\n",
      "2018-09-21 17:55:04 Iter 82000 [Val]: loss=65.69, epe=4.55\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-82000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-82000\n",
      "2018-09-21 18:12:59 Iter 83000 [Train]: loss=61.31, epe=4.17, lr=0.000100, samples/sec=33.7, sec/step=0.949, eta=5 days, 16:14:02\n",
      "2018-09-21 18:13:20 Iter 83000 [Val]: loss=62.30, epe=4.27\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-83000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-83000\n",
      "\n",
      "Resume training from step 83001...\n",
      "2018-09-21 21:46:13 Iter 84000 [Train]: loss=61.14, epe=4.16, lr=0.000100, samples/sec=45.9, sec/step=0.698, eta=4 days, 3:59:57\n",
      "2018-09-21 21:46:33 Iter 84000 [Val]: loss=63.07, epe=4.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-84000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-84000\n",
      "2018-09-21 21:59:01 Iter 85000 [Train]: loss=62.97, epe=4.31, lr=0.000100, samples/sec=50.5, sec/step=0.634, eta=3 days, 18:40:33\n",
      "2018-09-21 21:59:13 Iter 85000 [Val]: loss=61.10, epe=4.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-85000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-85000\n",
      "2018-09-21 22:11:39 Iter 86000 [Train]: loss=60.16, epe=4.08, lr=0.000100, samples/sec=50.3, sec/step=0.636, eta=3 days, 18:49:04\n",
      "2018-09-21 22:11:51 Iter 86000 [Val]: loss=63.22, epe=4.34\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-86000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-86000\n",
      "2018-09-21 22:24:22 Iter 87000 [Train]: loss=61.81, epe=4.21, lr=0.000100, samples/sec=49.9, sec/step=0.641, eta=3 days, 19:17:30\n",
      "2018-09-21 22:24:34 Iter 87000 [Val]: loss=67.03, epe=4.62\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-87000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-87000\n",
      "2018-09-21 22:37:07 Iter 88000 [Train]: loss=60.69, epe=4.12, lr=0.000100, samples/sec=49.8, sec/step=0.642, eta=3 days, 19:19:18\n",
      "2018-09-21 22:37:19 Iter 88000 [Val]: loss=59.75, epe=4.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-88000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-88000\n",
      "2018-09-21 22:49:55 Iter 89000 [Train]: loss=60.81, epe=nan, lr=0.000100, samples/sec=49.6, sec/step=0.645, eta=3 days, 19:36:42\n",
      "2018-09-21 22:50:07 Iter 89000 [Val]: loss=62.67, epe=4.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-89000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-89000\n",
      "2018-09-21 23:02:41 Iter 90000 [Train]: loss=61.19, epe=4.16, lr=0.000100, samples/sec=49.4, sec/step=0.648, eta=3 days, 19:43:49\n",
      "2018-09-21 23:02:54 Iter 90000 [Val]: loss=61.50, epe=4.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-90000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-90000\n",
      "2018-09-21 23:15:28 Iter 91000 [Train]: loss=60.53, epe=4.10, lr=0.000100, samples/sec=49.4, sec/step=0.647, eta=3 days, 19:32:18\n",
      "2018-09-21 23:15:41 Iter 91000 [Val]: loss=62.13, epe=4.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-91000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-91000\n",
      "2018-09-21 23:28:19 Iter 92000 [Train]: loss=60.99, epe=4.13, lr=0.000100, samples/sec=49.1, sec/step=0.652, eta=3 days, 20:00:12\n",
      "2018-09-21 23:28:32 Iter 92000 [Val]: loss=63.39, epe=4.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-92000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-92000\n",
      "2018-09-21 23:41:10 Iter 93000 [Train]: loss=59.25, epe=3.99, lr=0.000100, samples/sec=49.1, sec/step=0.652, eta=3 days, 19:49:22\n",
      "2018-09-21 23:41:22 Iter 93000 [Val]: loss=59.39, epe=4.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-93000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-93000\n",
      "2018-09-21 23:54:04 Iter 94000 [Train]: loss=60.13, epe=4.07, lr=0.000100, samples/sec=48.7, sec/step=0.657, eta=3 days, 20:19:13\n",
      "2018-09-21 23:54:17 Iter 94000 [Val]: loss=62.38, epe=4.27\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-94000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-94000\n",
      "2018-09-22 00:07:07 Iter 95000 [Train]: loss=59.81, epe=4.04, lr=0.000100, samples/sec=48.3, sec/step=0.662, eta=3 days, 20:53:34\n",
      "2018-09-22 00:07:20 Iter 95000 [Val]: loss=58.63, epe=3.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-95000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-95000\n",
      "\n",
      "Resume training from step 95001...\n",
      "2018-09-22 08:23:47 Iter 96000 [Train]: loss=59.74, epe=4.03, lr=0.000100, samples/sec=43.7, sec/step=0.732, eta=4 days, 6:26:42\n",
      "2018-09-22 08:24:08 Iter 96000 [Val]: loss=60.65, epe=4.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-96000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-96000\n",
      "2018-09-22 08:36:41 Iter 97000 [Train]: loss=60.26, epe=4.07, lr=0.000100, samples/sec=50.0, sec/step=0.641, eta=3 days, 17:29:49\n",
      "2018-09-22 08:36:53 Iter 97000 [Val]: loss=62.47, epe=4.29\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-97000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-97000\n",
      "2018-09-22 08:49:22 Iter 98000 [Train]: loss=59.09, epe=3.98, lr=0.000100, samples/sec=49.9, sec/step=0.641, eta=3 days, 17:25:41\n",
      "2018-09-22 08:49:35 Iter 98000 [Val]: loss=61.05, epe=4.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-98000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-98000\n",
      "2018-09-22 09:02:08 Iter 99000 [Train]: loss=59.51, epe=4.01, lr=0.000100, samples/sec=49.6, sec/step=0.645, eta=3 days, 17:41:59\n",
      "2018-09-22 09:02:20 Iter 99000 [Val]: loss=60.84, epe=4.17\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-99000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-99000\n",
      "2018-09-22 09:14:59 Iter 100000 [Train]: loss=60.11, epe=4.05, lr=0.000100, samples/sec=49.3, sec/step=0.649, eta=3 days, 18:06:36\n",
      "2018-09-22 09:15:11 Iter 100000 [Val]: loss=60.79, epe=4.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-100000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-100000\n",
      "2018-09-22 09:27:50 Iter 101000 [Train]: loss=58.96, epe=3.96, lr=0.000100, samples/sec=49.2, sec/step=0.651, eta=3 days, 18:14:21\n",
      "2018-09-22 09:28:03 Iter 101000 [Val]: loss=60.18, epe=4.11\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-101000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-101000\n",
      "2018-09-22 09:40:44 Iter 102000 [Train]: loss=59.69, epe=4.02, lr=0.000100, samples/sec=48.8, sec/step=0.655, eta=3 days, 18:38:13\n",
      "2018-09-22 09:40:56 Iter 102000 [Val]: loss=59.26, epe=4.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-102000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-102000\n",
      "2018-09-22 09:53:34 Iter 103000 [Train]: loss=58.30, epe=3.91, lr=0.000100, samples/sec=49.1, sec/step=0.652, eta=3 days, 17:58:14\n",
      "2018-09-22 09:53:46 Iter 103000 [Val]: loss=60.47, epe=4.11\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-103000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-103000\n",
      "2018-09-22 10:06:30 Iter 104000 [Train]: loss=58.98, epe=3.96, lr=0.000100, samples/sec=48.6, sec/step=0.659, eta=3 days, 18:46:41\n",
      "2018-09-22 10:06:43 Iter 104000 [Val]: loss=59.78, epe=4.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-104000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-104000\n",
      "2018-09-22 10:19:29 Iter 105000 [Train]: loss=58.33, epe=3.91, lr=0.000100, samples/sec=48.4, sec/step=0.661, eta=3 days, 18:51:58\n",
      "2018-09-22 10:19:42 Iter 105000 [Val]: loss=60.21, epe=4.10\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-105000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-105000\n",
      "2018-09-22 10:32:32 Iter 106000 [Train]: loss=59.07, epe=3.97, lr=0.000100, samples/sec=48.1, sec/step=0.665, eta=3 days, 19:12:02\n",
      "2018-09-22 10:32:45 Iter 106000 [Val]: loss=62.95, epe=4.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-106000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-106000\n",
      "2018-09-22 10:45:47 Iter 107000 [Train]: loss=57.52, epe=3.84, lr=0.000100, samples/sec=47.4, sec/step=0.674, eta=3 days, 20:22:07\n",
      "2018-09-22 10:46:00 Iter 107000 [Val]: loss=59.30, epe=4.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-107000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-107000\n",
      "2018-09-22 10:59:08 Iter 108000 [Train]: loss=58.47, epe=3.92, lr=0.000100, samples/sec=47.0, sec/step=0.681, eta=3 days, 21:06:54\n",
      "2018-09-22 10:59:22 Iter 108000 [Val]: loss=60.48, epe=4.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-108000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-108000\n",
      "2018-09-22 11:12:41 Iter 109000 [Train]: loss=58.89, epe=3.95, lr=0.000100, samples/sec=46.3, sec/step=0.691, eta=3 days, 22:10:55\n",
      "2018-09-22 11:12:54 Iter 109000 [Val]: loss=59.40, epe=4.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-109000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-109000\n",
      "2018-09-22 11:26:21 Iter 110000 [Train]: loss=58.30, epe=3.91, lr=0.000100, samples/sec=45.9, sec/step=0.696, eta=3 days, 22:47:51\n",
      "2018-09-22 11:26:34 Iter 110000 [Val]: loss=59.65, epe=4.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-110000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-110000\n",
      "2018-09-22 11:40:07 Iter 111000 [Train]: loss=57.49, epe=3.84, lr=0.000100, samples/sec=45.7, sec/step=0.699, eta=3 days, 23:00:53\n",
      "2018-09-22 11:40:20 Iter 111000 [Val]: loss=60.95, epe=4.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-111000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-111000\n",
      "2018-09-22 11:53:49 Iter 112000 [Train]: loss=58.28, epe=3.90, lr=0.000100, samples/sec=45.7, sec/step=0.699, eta=3 days, 22:49:00\n",
      "2018-09-22 11:54:03 Iter 112000 [Val]: loss=60.33, epe=4.10\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-112000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-112000\n",
      "2018-09-22 12:07:36 Iter 113000 [Train]: loss=57.10, epe=3.80, lr=0.000100, samples/sec=45.5, sec/step=0.704, eta=3 days, 23:11:32\n",
      "2018-09-22 12:07:49 Iter 113000 [Val]: loss=58.73, epe=3.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-113000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-113000\n",
      "2018-09-22 12:21:30 Iter 114000 [Train]: loss=57.78, epe=3.86, lr=0.000100, samples/sec=45.0, sec/step=0.711, eta=4 days, 0:03:08\n",
      "2018-09-22 12:21:44 Iter 114000 [Val]: loss=61.99, epe=4.25\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-114000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-114000\n",
      "2018-09-22 12:35:34 Iter 115000 [Train]: loss=57.53, epe=3.84, lr=0.000100, samples/sec=44.4, sec/step=0.721, eta=4 days, 1:08:24\n",
      "2018-09-22 12:35:49 Iter 115000 [Val]: loss=57.58, epe=3.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-115000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-115000\n",
      "2018-09-22 12:50:09 Iter 116000 [Train]: loss=57.78, epe=3.85, lr=0.000100, samples/sec=42.9, sec/step=0.746, eta=4 days, 4:19:42\n",
      "2018-09-22 12:50:23 Iter 116000 [Val]: loss=62.11, epe=4.21\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-116000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-116000\n",
      "2018-09-22 13:04:23 Iter 117000 [Train]: loss=56.76, epe=3.78, lr=0.000100, samples/sec=44.0, sec/step=0.727, eta=4 days, 1:35:42\n",
      "2018-09-22 13:04:37 Iter 117000 [Val]: loss=61.01, epe=4.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-117000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-117000\n",
      "2018-09-22 13:18:39 Iter 118000 [Train]: loss=57.30, epe=3.81, lr=0.000100, samples/sec=43.7, sec/step=0.732, eta=4 days, 2:02:14\n",
      "2018-09-22 13:18:53 Iter 118000 [Val]: loss=57.17, epe=3.85\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-118000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-118000\n",
      "2018-09-22 13:33:06 Iter 119000 [Train]: loss=57.91, epe=3.87, lr=0.000100, samples/sec=43.1, sec/step=0.743, eta=4 days, 3:17:34\n",
      "2018-09-22 13:33:21 Iter 119000 [Val]: loss=57.57, epe=3.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-119000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-119000\n",
      "2018-09-22 13:47:35 Iter 120000 [Train]: loss=57.75, epe=3.85, lr=0.000100, samples/sec=43.2, sec/step=0.742, eta=4 days, 2:52:13\n",
      "2018-09-22 13:47:50 Iter 120000 [Val]: loss=60.99, epe=4.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-120000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-120000\n",
      "2018-09-22 14:02:16 Iter 121000 [Train]: loss=56.59, epe=3.76, lr=0.000100, samples/sec=42.5, sec/step=0.753, eta=4 days, 4:08:18\n",
      "2018-09-22 14:02:31 Iter 121000 [Val]: loss=57.88, epe=3.93\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-121000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-121000\n",
      "2018-09-22 14:17:08 Iter 122000 [Train]: loss=56.93, epe=3.79, lr=0.000100, samples/sec=42.1, sec/step=0.761, eta=4 days, 5:01:56\n",
      "2018-09-22 14:17:23 Iter 122000 [Val]: loss=65.48, epe=4.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-122000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-122000\n",
      "2018-09-22 14:32:10 Iter 123000 [Train]: loss=57.18, epe=3.80, lr=0.000100, samples/sec=41.8, sec/step=0.766, eta=4 days, 5:31:20\n",
      "2018-09-22 14:32:25 Iter 123000 [Val]: loss=55.30, epe=3.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-123000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-123000\n",
      "2018-09-22 14:47:13 Iter 124000 [Train]: loss=56.34, epe=3.73, lr=0.000100, samples/sec=41.5, sec/step=0.770, eta=4 days, 5:52:12\n",
      "2018-09-22 14:47:29 Iter 124000 [Val]: loss=61.48, epe=4.13\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-124000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-124000\n",
      "2018-09-22 15:02:14 Iter 125000 [Train]: loss=57.72, epe=3.84, lr=0.000100, samples/sec=41.7, sec/step=0.768, eta=4 days, 5:20:18\n",
      "2018-09-22 15:02:29 Iter 125000 [Val]: loss=57.87, epe=nan\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-125000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-125000\n",
      "2018-09-22 15:17:22 Iter 126000 [Train]: loss=56.48, epe=3.74, lr=0.000100, samples/sec=41.3, sec/step=0.775, eta=4 days, 5:59:43\n",
      "2018-09-22 15:17:38 Iter 126000 [Val]: loss=58.77, epe=3.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-126000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-126000\n",
      "2018-09-22 15:32:38 Iter 127000 [Train]: loss=56.20, epe=3.72, lr=0.000100, samples/sec=40.9, sec/step=0.783, eta=4 days, 6:54:46\n",
      "2018-09-22 15:32:54 Iter 127000 [Val]: loss=56.09, epe=3.75\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-127000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-127000\n",
      "2018-09-22 15:47:52 Iter 128000 [Train]: loss=56.75, epe=3.76, lr=0.000100, samples/sec=41.0, sec/step=0.780, eta=4 days, 6:18:43\n",
      "2018-09-22 15:48:08 Iter 128000 [Val]: loss=60.00, epe=4.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-128000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-128000\n",
      "2018-09-22 16:03:20 Iter 129000 [Train]: loss=56.47, epe=nan, lr=0.000100, samples/sec=40.3, sec/step=0.793, eta=4 days, 7:46:25\n",
      "2018-09-22 16:03:36 Iter 129000 [Val]: loss=61.29, epe=4.16\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-129000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-129000\n",
      "2018-09-22 16:18:43 Iter 130000 [Train]: loss=55.85, epe=3.69, lr=0.000100, samples/sec=40.6, sec/step=0.788, eta=4 days, 6:53:10\n",
      "2018-09-22 16:19:00 Iter 130000 [Val]: loss=56.62, epe=3.80\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-130000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-130000\n",
      "2018-09-22 16:34:23 Iter 131000 [Train]: loss=56.29, epe=3.73, lr=0.000100, samples/sec=40.0, sec/step=0.799, eta=4 days, 8:06:21\n",
      "2018-09-22 16:34:40 Iter 131000 [Val]: loss=58.14, epe=3.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-131000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-131000\n",
      "2018-09-22 16:50:09 Iter 132000 [Train]: loss=58.45, epe=3.89, lr=0.000100, samples/sec=39.5, sec/step=0.810, eta=4 days, 9:15:46\n",
      "2018-09-22 16:50:26 Iter 132000 [Val]: loss=57.17, epe=3.81\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-132000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-132000\n",
      "2018-09-22 17:05:56 Iter 133000 [Train]: loss=55.67, epe=nan, lr=0.000100, samples/sec=39.5, sec/step=0.810, eta=4 days, 9:03:50\n",
      "2018-09-22 17:06:13 Iter 133000 [Val]: loss=56.24, epe=3.77\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-133000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-133000\n",
      "2018-09-22 17:21:49 Iter 134000 [Train]: loss=55.52, epe=3.67, lr=0.000100, samples/sec=39.2, sec/step=0.816, eta=4 days, 9:38:39\n",
      "2018-09-22 17:22:06 Iter 134000 [Val]: loss=59.23, epe=3.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-134000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-134000\n",
      "2018-09-22 17:37:46 Iter 135000 [Train]: loss=55.66, epe=3.68, lr=0.000100, samples/sec=39.0, sec/step=0.820, eta=4 days, 9:53:35\n",
      "2018-09-22 17:38:03 Iter 135000 [Val]: loss=57.20, epe=3.82\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-135000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-135000\n",
      "2018-09-22 17:53:47 Iter 136000 [Train]: loss=56.04, epe=3.70, lr=0.000100, samples/sec=38.8, sec/step=0.824, eta=4 days, 10:11:48\n",
      "2018-09-22 17:54:04 Iter 136000 [Val]: loss=55.49, epe=3.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-136000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-136000\n",
      "2018-09-22 18:09:55 Iter 137000 [Train]: loss=56.66, epe=3.74, lr=0.000100, samples/sec=38.6, sec/step=0.830, eta=4 days, 10:42:56\n",
      "2018-09-22 18:10:13 Iter 137000 [Val]: loss=59.29, epe=3.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-137000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-137000\n",
      "2018-09-22 18:26:03 Iter 138000 [Train]: loss=55.82, epe=3.69, lr=0.000100, samples/sec=38.6, sec/step=0.829, eta=4 days, 10:24:01\n",
      "2018-09-22 18:26:20 Iter 138000 [Val]: loss=58.88, epe=3.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-138000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-138000\n",
      "2018-09-22 18:42:20 Iter 139000 [Train]: loss=55.11, epe=3.62, lr=0.000100, samples/sec=38.2, sec/step=0.838, eta=4 days, 11:17:02\n",
      "2018-09-22 18:42:37 Iter 139000 [Val]: loss=57.59, epe=3.88\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-139000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-139000\n",
      "2018-09-22 18:58:38 Iter 140000 [Train]: loss=54.66, epe=3.60, lr=0.000100, samples/sec=38.2, sec/step=0.838, eta=4 days, 11:03:16\n",
      "2018-09-22 18:58:55 Iter 140000 [Val]: loss=53.87, epe=3.54\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-140000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-140000\n",
      "2018-09-22 19:15:08 Iter 141000 [Train]: loss=54.73, epe=3.60, lr=0.000100, samples/sec=37.9, sec/step=0.845, eta=4 days, 11:41:58\n",
      "2018-09-22 19:15:26 Iter 141000 [Val]: loss=57.98, epe=3.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-141000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-141000\n",
      "2018-09-22 19:31:33 Iter 142000 [Train]: loss=56.05, epe=3.70, lr=0.000100, samples/sec=37.9, sec/step=0.845, eta=4 days, 11:30:26\n",
      "2018-09-22 19:31:51 Iter 142000 [Val]: loss=55.01, epe=3.67\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-142000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-142000\n",
      "2018-09-22 19:48:11 Iter 143000 [Train]: loss=55.69, epe=3.67, lr=0.000100, samples/sec=37.3, sec/step=0.858, eta=4 days, 12:51:21\n",
      "2018-09-22 19:48:28 Iter 143000 [Val]: loss=57.92, epe=3.88\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-143000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-143000\n",
      "2018-09-22 20:04:41 Iter 144000 [Train]: loss=54.62, epe=3.59, lr=0.000100, samples/sec=37.4, sec/step=0.856, eta=4 days, 12:27:50\n",
      "2018-09-22 20:04:59 Iter 144000 [Val]: loss=55.95, epe=3.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-144000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-144000\n",
      "2018-09-22 20:21:08 Iter 145000 [Train]: loss=55.29, epe=3.64, lr=0.000100, samples/sec=37.5, sec/step=0.853, eta=4 days, 11:50:53\n",
      "2018-09-22 20:21:26 Iter 145000 [Val]: loss=56.16, epe=3.77\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-145000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-145000\n",
      "2018-09-22 20:37:40 Iter 146000 [Train]: loss=55.02, epe=3.62, lr=0.000100, samples/sec=37.3, sec/step=0.859, eta=4 days, 12:16:13\n",
      "2018-09-22 20:37:58 Iter 146000 [Val]: loss=57.23, epe=3.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-146000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-146000\n",
      "2018-09-22 20:54:11 Iter 147000 [Train]: loss=54.93, epe=3.61, lr=0.000100, samples/sec=37.3, sec/step=0.858, eta=4 days, 11:56:37\n",
      "2018-09-22 20:54:29 Iter 147000 [Val]: loss=55.81, epe=3.71\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-147000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-147000\n",
      "2018-09-22 21:10:46 Iter 148000 [Train]: loss=55.20, epe=3.63, lr=0.000100, samples/sec=37.2, sec/step=0.861, eta=4 days, 12:02:53\n",
      "2018-09-22 21:11:04 Iter 148000 [Val]: loss=57.46, epe=3.83\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-148000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-148000\n",
      "2018-09-22 21:27:26 Iter 149000 [Train]: loss=54.31, epe=3.56, lr=0.000100, samples/sec=37.0, sec/step=0.866, eta=4 days, 12:27:46\n",
      "2018-09-22 21:27:44 Iter 149000 [Val]: loss=56.48, epe=3.76\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-149000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-149000\n",
      "2018-09-22 21:44:06 Iter 150000 [Train]: loss=55.26, epe=3.63, lr=0.000100, samples/sec=37.0, sec/step=0.866, eta=4 days, 12:15:06\n",
      "2018-09-22 21:44:25 Iter 150000 [Val]: loss=56.60, epe=3.81\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-150000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-150000\n",
      "2018-09-22 22:00:53 Iter 151000 [Train]: loss=54.81, epe=3.59, lr=0.000100, samples/sec=36.7, sec/step=0.872, eta=4 days, 12:44:57\n",
      "2018-09-22 22:01:11 Iter 151000 [Val]: loss=56.44, epe=3.79\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-151000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-151000\n",
      "2018-09-22 22:17:39 Iter 152000 [Train]: loss=54.13, epe=3.55, lr=0.000100, samples/sec=36.8, sec/step=0.871, eta=4 days, 12:20:26\n",
      "2018-09-22 22:17:58 Iter 152000 [Val]: loss=57.84, epe=3.85\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-152000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-152000\n",
      "2018-09-22 22:34:33 Iter 153000 [Train]: loss=53.95, epe=3.53, lr=0.000100, samples/sec=36.7, sec/step=0.871, eta=4 days, 12:10:08\n",
      "2018-09-22 22:34:51 Iter 153000 [Val]: loss=56.87, epe=3.83\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-153000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-153000\n",
      "2018-09-22 22:51:25 Iter 154000 [Train]: loss=55.44, epe=3.64, lr=0.000100, samples/sec=36.5, sec/step=0.877, eta=4 days, 12:39:50\n",
      "2018-09-22 22:51:44 Iter 154000 [Val]: loss=55.85, epe=3.70\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-154000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-154000\n",
      "2018-09-22 23:08:18 Iter 155000 [Train]: loss=54.69, epe=3.58, lr=0.000100, samples/sec=36.5, sec/step=0.878, eta=4 days, 12:28:55\n",
      "2018-09-22 23:08:37 Iter 155000 [Val]: loss=57.26, epe=3.82\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-155000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-155000\n",
      "2018-09-22 23:25:14 Iter 156000 [Train]: loss=55.54, epe=3.65, lr=0.000100, samples/sec=36.4, sec/step=0.880, eta=4 days, 12:34:09\n",
      "2018-09-22 23:25:33 Iter 156000 [Val]: loss=53.21, epe=3.51\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-156000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-156000\n",
      "2018-09-22 23:42:14 Iter 157000 [Train]: loss=54.26, epe=3.54, lr=0.000100, samples/sec=36.2, sec/step=0.883, eta=4 days, 12:42:13\n",
      "2018-09-22 23:42:32 Iter 157000 [Val]: loss=57.95, epe=3.89\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-157000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-157000\n",
      "2018-09-22 23:59:16 Iter 158000 [Train]: loss=53.76, epe=3.51, lr=0.000100, samples/sec=36.1, sec/step=0.886, eta=4 days, 12:48:13\n",
      "2018-09-22 23:59:35 Iter 158000 [Val]: loss=56.98, epe=3.84\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-158000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-158000\n",
      "2018-09-23 00:16:20 Iter 159000 [Train]: loss=53.09, epe=3.46, lr=0.000100, samples/sec=36.0, sec/step=0.888, eta=4 days, 12:45:33\n",
      "2018-09-23 00:16:39 Iter 159000 [Val]: loss=56.68, epe=3.76\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-159000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-159000\n",
      "2018-09-23 00:33:22 Iter 160000 [Train]: loss=54.63, epe=3.58, lr=0.000100, samples/sec=36.1, sec/step=0.886, eta=4 days, 12:17:22\n",
      "2018-09-23 00:33:41 Iter 160000 [Val]: loss=53.91, epe=3.58\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-160000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-160000\n",
      "2018-09-23 00:50:32 Iter 161000 [Train]: loss=54.21, epe=3.54, lr=0.000100, samples/sec=35.8, sec/step=0.894, eta=4 days, 12:57:51\n",
      "2018-09-23 00:50:51 Iter 161000 [Val]: loss=55.96, epe=3.71\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-161000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-161000\n",
      "2018-09-23 01:07:47 Iter 162000 [Train]: loss=53.84, epe=3.51, lr=0.000100, samples/sec=35.6, sec/step=0.898, eta=4 days, 13:14:02\n",
      "2018-09-23 01:08:06 Iter 162000 [Val]: loss=56.76, epe=3.81\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-162000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-162000\n",
      "2018-09-23 01:25:07 Iter 163000 [Train]: loss=54.43, epe=3.56, lr=0.000100, samples/sec=35.4, sec/step=0.904, eta=4 days, 13:41:01\n",
      "2018-09-23 01:25:26 Iter 163000 [Val]: loss=56.64, epe=3.76\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-163000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-163000\n",
      "2018-09-23 01:42:25 Iter 164000 [Train]: loss=53.31, epe=3.48, lr=0.000100, samples/sec=35.5, sec/step=0.901, eta=4 days, 13:10:25\n",
      "2018-09-23 01:42:44 Iter 164000 [Val]: loss=56.49, epe=3.77\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-164000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-164000\n",
      "2018-09-23 01:59:41 Iter 165000 [Train]: loss=54.04, epe=3.53, lr=0.000100, samples/sec=35.6, sec/step=0.899, eta=4 days, 12:37:15\n",
      "2018-09-23 02:00:00 Iter 165000 [Val]: loss=56.18, epe=3.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-165000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-165000\n",
      "2018-09-23 02:17:01 Iter 166000 [Train]: loss=54.17, epe=3.54, lr=0.000100, samples/sec=35.5, sec/step=0.902, eta=4 days, 12:42:36\n",
      "2018-09-23 02:17:20 Iter 166000 [Val]: loss=56.24, epe=3.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-166000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-166000\n",
      "2018-09-23 02:34:32 Iter 167000 [Train]: loss=53.71, epe=3.50, lr=0.000100, samples/sec=35.3, sec/step=0.906, eta=4 days, 12:56:26\n",
      "2018-09-23 02:34:51 Iter 167000 [Val]: loss=55.35, epe=3.69\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-167000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-167000\n",
      "2018-09-23 02:52:07 Iter 168000 [Train]: loss=53.74, epe=3.50, lr=0.000100, samples/sec=35.1, sec/step=0.913, eta=4 days, 13:32:52\n",
      "2018-09-23 02:52:26 Iter 168000 [Val]: loss=57.49, epe=3.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-168000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-168000\n",
      "2018-09-23 03:09:38 Iter 169000 [Train]: loss=53.39, epe=3.48, lr=0.000100, samples/sec=35.2, sec/step=0.908, eta=4 days, 12:45:18\n",
      "2018-09-23 03:09:58 Iter 169000 [Val]: loss=57.03, epe=3.80\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-169000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-169000\n",
      "2018-09-23 03:27:11 Iter 170000 [Train]: loss=52.50, epe=3.41, lr=0.000100, samples/sec=35.2, sec/step=0.909, eta=4 days, 12:36:48\n",
      "2018-09-23 03:27:31 Iter 170000 [Val]: loss=53.43, epe=3.52\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-170000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-170000\n",
      "2018-09-23 03:44:50 Iter 171000 [Train]: loss=53.51, epe=3.48, lr=0.000100, samples/sec=35.0, sec/step=0.915, eta=4 days, 13:00:53\n",
      "2018-09-23 03:45:10 Iter 171000 [Val]: loss=56.51, epe=3.76\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-171000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-171000\n",
      "2018-09-23 04:02:33 Iter 172000 [Train]: loss=53.54, epe=3.49, lr=0.000100, samples/sec=34.8, sec/step=0.920, eta=4 days, 13:20:39\n",
      "2018-09-23 04:02:53 Iter 172000 [Val]: loss=56.16, epe=3.71\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-172000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-172000\n",
      "2018-09-23 04:20:27 Iter 173000 [Train]: loss=53.53, epe=3.48, lr=0.000100, samples/sec=34.4, sec/step=0.930, eta=4 days, 14:18:46\n",
      "2018-09-23 04:20:47 Iter 173000 [Val]: loss=55.81, epe=3.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-173000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-173000\n",
      "2018-09-23 04:38:20 Iter 174000 [Train]: loss=52.71, epe=3.42, lr=0.000100, samples/sec=34.4, sec/step=0.930, eta=4 days, 14:01:49\n",
      "2018-09-23 04:38:40 Iter 174000 [Val]: loss=53.27, epe=3.51\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-174000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-174000\n",
      "2018-09-23 04:56:03 Iter 175000 [Train]: loss=53.96, epe=3.52, lr=0.000100, samples/sec=34.6, sec/step=0.925, eta=4 days, 13:10:59\n",
      "2018-09-23 04:56:23 Iter 175000 [Val]: loss=56.18, epe=3.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-175000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-175000\n",
      "2018-09-23 05:13:50 Iter 176000 [Train]: loss=53.15, epe=3.45, lr=0.000100, samples/sec=34.4, sec/step=0.930, eta=4 days, 13:31:38\n",
      "2018-09-23 05:14:10 Iter 176000 [Val]: loss=55.95, epe=3.70\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-176000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-176000\n",
      "2018-09-23 05:31:40 Iter 177000 [Train]: loss=53.60, epe=3.49, lr=0.000100, samples/sec=34.3, sec/step=0.933, eta=4 days, 13:36:02\n",
      "2018-09-23 05:32:00 Iter 177000 [Val]: loss=55.28, epe=3.65\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-177000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-177000\n",
      "2018-09-23 05:49:29 Iter 178000 [Train]: loss=52.89, epe=3.43, lr=0.000100, samples/sec=34.4, sec/step=0.930, eta=4 days, 13:04:21\n",
      "2018-09-23 05:49:49 Iter 178000 [Val]: loss=54.54, epe=3.61\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-178000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-178000\n",
      "2018-09-23 06:07:24 Iter 179000 [Train]: loss=52.55, epe=3.41, lr=0.000100, samples/sec=34.1, sec/step=0.937, eta=4 days, 13:36:33\n",
      "2018-09-23 06:07:45 Iter 179000 [Val]: loss=54.44, epe=3.60\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-179000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-179000\n",
      "2018-09-23 06:25:21 Iter 180000 [Train]: loss=53.09, epe=3.45, lr=0.000100, samples/sec=34.1, sec/step=0.939, eta=4 days, 13:32:05\n",
      "2018-09-23 06:25:42 Iter 180000 [Val]: loss=54.70, epe=3.63\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-180000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-180000\n",
      "2018-09-23 06:43:15 Iter 181000 [Train]: loss=53.58, epe=3.49, lr=0.000100, samples/sec=34.2, sec/step=0.936, eta=4 days, 12:58:35\n",
      "2018-09-23 06:43:36 Iter 181000 [Val]: loss=56.10, epe=3.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-181000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-181000\n",
      "2018-09-23 07:01:14 Iter 182000 [Train]: loss=52.79, epe=3.42, lr=0.000100, samples/sec=34.0, sec/step=0.940, eta=4 days, 13:11:10\n",
      "2018-09-23 07:01:35 Iter 182000 [Val]: loss=54.59, epe=3.63\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-182000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-182000\n",
      "2018-09-23 07:19:18 Iter 183000 [Train]: loss=53.20, epe=3.46, lr=0.000100, samples/sec=33.8, sec/step=0.946, eta=4 days, 13:32:34\n",
      "2018-09-23 07:19:39 Iter 183000 [Val]: loss=55.47, epe=3.71\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-183000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-183000\n",
      "2018-09-23 07:37:35 Iter 184000 [Train]: loss=52.05, epe=3.37, lr=0.000100, samples/sec=33.8, sec/step=0.947, eta=4 days, 13:27:39\n",
      "2018-09-23 07:37:56 Iter 184000 [Val]: loss=55.31, epe=3.67\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-184000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-184000\n",
      "2018-09-23 07:55:43 Iter 185000 [Train]: loss=52.87, epe=3.43, lr=0.000100, samples/sec=33.7, sec/step=0.951, eta=4 days, 13:34:42\n",
      "2018-09-23 07:56:04 Iter 185000 [Val]: loss=55.88, epe=3.75\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-185000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-185000\n",
      "2018-09-23 08:13:54 Iter 186000 [Train]: loss=52.92, epe=3.44, lr=0.000100, samples/sec=33.6, sec/step=0.952, eta=4 days, 13:28:46\n",
      "2018-09-23 08:14:15 Iter 186000 [Val]: loss=54.84, epe=3.63\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-186000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-186000\n",
      "2018-09-23 08:32:08 Iter 187000 [Train]: loss=52.49, epe=3.39, lr=0.000100, samples/sec=33.5, sec/step=0.955, eta=4 days, 13:34:11\n",
      "2018-09-23 08:32:29 Iter 187000 [Val]: loss=53.41, epe=3.51\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-187000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-187000\n",
      "2018-09-23 08:50:25 Iter 188000 [Train]: loss=52.88, epe=3.43, lr=0.000100, samples/sec=33.4, sec/step=0.957, eta=4 days, 13:32:25\n",
      "2018-09-23 08:50:46 Iter 188000 [Val]: loss=55.25, epe=3.66\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-188000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-188000\n",
      "2018-09-23 09:08:44 Iter 189000 [Train]: loss=51.82, epe=3.35, lr=0.000100, samples/sec=33.4, sec/step=0.959, eta=4 days, 13:30:58\n",
      "2018-09-23 09:09:05 Iter 189000 [Val]: loss=55.99, epe=3.73\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-189000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-189000\n",
      "2018-09-23 09:27:05 Iter 190000 [Train]: loss=53.10, epe=3.44, lr=0.000100, samples/sec=33.3, sec/step=0.962, eta=4 days, 13:35:27\n",
      "2018-09-23 09:27:27 Iter 190000 [Val]: loss=53.25, epe=3.48\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-190000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-190000\n",
      "2018-09-23 09:45:26 Iter 191000 [Train]: loss=52.65, epe=3.41, lr=0.000100, samples/sec=33.3, sec/step=0.961, eta=4 days, 13:13:30\n",
      "2018-09-23 09:45:48 Iter 191000 [Val]: loss=52.41, epe=3.45\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-191000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-191000\n",
      "2018-09-23 10:03:47 Iter 192000 [Train]: loss=52.44, epe=3.39, lr=0.000100, samples/sec=33.3, sec/step=0.961, eta=4 days, 12:57:21\n",
      "2018-09-23 10:04:09 Iter 192000 [Val]: loss=54.06, epe=3.55\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-192000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-192000\n",
      "2018-09-23 10:22:09 Iter 193000 [Train]: loss=51.58, epe=3.33, lr=0.000100, samples/sec=33.3, sec/step=0.961, eta=4 days, 12:38:33\n",
      "2018-09-23 10:22:31 Iter 193000 [Val]: loss=52.77, epe=3.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-193000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-193000\n",
      "2018-09-23 10:40:36 Iter 194000 [Train]: loss=51.49, epe=3.32, lr=0.000100, samples/sec=33.1, sec/step=0.967, eta=4 days, 13:00:39\n",
      "2018-09-23 10:40:58 Iter 194000 [Val]: loss=56.52, epe=3.76\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-194000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-194000\n",
      "2018-09-23 10:59:02 Iter 195000 [Train]: loss=52.62, epe=3.40, lr=0.000100, samples/sec=33.2, sec/step=0.964, eta=4 days, 12:27:07\n",
      "2018-09-23 10:59:23 Iter 195000 [Val]: loss=55.16, epe=3.66\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-195000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-195000\n",
      "2018-09-23 11:17:35 Iter 196000 [Train]: loss=52.46, epe=3.39, lr=0.000100, samples/sec=33.1, sec/step=0.968, eta=4 days, 12:39:24\n",
      "2018-09-23 11:17:57 Iter 196000 [Val]: loss=53.91, epe=3.54\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-196000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-196000\n",
      "2018-09-23 11:36:09 Iter 197000 [Train]: loss=51.53, epe=3.33, lr=0.000100, samples/sec=33.1, sec/step=0.966, eta=4 days, 12:08:22\n",
      "2018-09-23 11:36:31 Iter 197000 [Val]: loss=52.39, epe=3.43\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-197000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-197000\n",
      "2018-09-23 11:54:44 Iter 198000 [Train]: loss=53.18, epe=3.45, lr=0.000100, samples/sec=33.1, sec/step=0.968, eta=4 days, 12:03:00\n",
      "2018-09-23 11:55:06 Iter 198000 [Val]: loss=56.24, epe=3.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-198000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-198000\n",
      "2018-09-23 12:13:14 Iter 199000 [Train]: loss=52.09, epe=3.37, lr=0.000100, samples/sec=33.3, sec/step=0.962, eta=4 days, 11:11:01\n",
      "2018-09-23 12:13:36 Iter 199000 [Val]: loss=54.65, epe=3.62\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-199000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-199000\n",
      "2018-09-23 12:31:50 Iter 200000 [Train]: loss=52.64, epe=3.40, lr=0.000100, samples/sec=33.1, sec/step=0.968, eta=4 days, 11:32:19\n",
      "2018-09-23 12:32:12 Iter 200000 [Val]: loss=53.97, epe=3.54\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-200000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-200000\n",
      "2018-09-23 12:50:28 Iter 201000 [Train]: loss=49.48, epe=nan, lr=0.000050, samples/sec=33.0, sec/step=0.971, eta=4 days, 11:37:31\n",
      "2018-09-23 12:50:50 Iter 201000 [Val]: loss=51.99, epe=3.43\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-201000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-201000\n",
      "2018-09-23 13:09:02 Iter 202000 [Train]: loss=50.08, epe=3.21, lr=0.000050, samples/sec=32.9, sec/step=0.974, eta=4 days, 11:41:13\n",
      "2018-09-23 13:09:24 Iter 202000 [Val]: loss=52.78, epe=3.46\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-202000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-202000\n",
      "2018-09-23 13:27:40 Iter 203000 [Train]: loss=49.32, epe=3.15, lr=0.000050, samples/sec=32.8, sec/step=0.977, eta=4 days, 11:44:11\n",
      "2018-09-23 13:28:03 Iter 203000 [Val]: loss=53.49, epe=3.54\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-203000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-203000\n",
      "2018-09-23 13:46:31 Iter 204000 [Train]: loss=49.88, epe=3.19, lr=0.000050, samples/sec=32.8, sec/step=0.977, eta=4 days, 11:26:58\n",
      "2018-09-23 13:46:53 Iter 204000 [Val]: loss=50.22, epe=3.24\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-204000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-204000\n",
      "2018-09-23 14:05:11 Iter 205000 [Train]: loss=48.83, epe=3.11, lr=0.000050, samples/sec=32.7, sec/step=0.978, eta=4 days, 11:18:00\n",
      "2018-09-23 14:05:33 Iter 205000 [Val]: loss=52.92, epe=3.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-205000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-205000\n",
      "2018-09-23 14:23:51 Iter 206000 [Train]: loss=49.75, epe=nan, lr=0.000050, samples/sec=32.7, sec/step=0.978, eta=4 days, 11:02:10\n",
      "2018-09-23 14:24:13 Iter 206000 [Val]: loss=51.45, epe=3.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-206000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-206000\n",
      "2018-09-23 14:42:32 Iter 207000 [Train]: loss=49.15, epe=3.13, lr=0.000050, samples/sec=32.7, sec/step=0.979, eta=4 days, 10:50:19\n",
      "2018-09-23 14:42:54 Iter 207000 [Val]: loss=52.49, epe=3.44\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-207000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-207000\n",
      "2018-09-23 15:01:16 Iter 208000 [Train]: loss=48.65, epe=3.10, lr=0.000050, samples/sec=32.5, sec/step=0.983, eta=4 days, 11:05:14\n",
      "2018-09-23 15:01:39 Iter 208000 [Val]: loss=53.34, epe=3.49\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-208000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-208000\n",
      "2018-09-23 15:20:04 Iter 209000 [Train]: loss=49.39, epe=3.15, lr=0.000050, samples/sec=32.5, sec/step=0.985, eta=4 days, 11:00:18\n",
      "2018-09-23 15:20:26 Iter 209000 [Val]: loss=51.27, epe=3.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-209000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-209000\n",
      "2018-09-23 15:38:47 Iter 210000 [Train]: loss=48.60, epe=3.10, lr=0.000050, samples/sec=32.6, sec/step=0.981, eta=4 days, 10:16:07\n",
      "2018-09-23 15:39:10 Iter 210000 [Val]: loss=52.21, epe=3.41\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-210000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-210000\n",
      "2018-09-23 15:57:31 Iter 211000 [Train]: loss=49.09, epe=3.13, lr=0.000050, samples/sec=32.7, sec/step=0.980, eta=4 days, 9:54:03\n",
      "2018-09-23 15:57:53 Iter 211000 [Val]: loss=51.07, epe=3.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-211000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-211000\n",
      "2018-09-23 16:16:14 Iter 212000 [Train]: loss=48.93, epe=3.12, lr=0.000050, samples/sec=32.6, sec/step=0.981, eta=4 days, 9:41:39\n",
      "2018-09-23 16:16:36 Iter 212000 [Val]: loss=53.09, epe=3.49\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-212000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-212000\n",
      "2018-09-23 16:35:00 Iter 213000 [Train]: loss=49.24, epe=3.14, lr=0.000050, samples/sec=32.5, sec/step=0.984, eta=4 days, 9:45:40\n",
      "2018-09-23 16:35:22 Iter 213000 [Val]: loss=51.43, epe=3.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-213000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-213000\n",
      "2018-09-23 16:53:44 Iter 214000 [Train]: loss=48.72, epe=3.11, lr=0.000050, samples/sec=32.6, sec/step=0.981, eta=4 days, 9:13:06\n",
      "2018-09-23 16:54:06 Iter 214000 [Val]: loss=53.31, epe=3.52\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-214000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-214000\n",
      "2018-09-23 17:12:31 Iter 215000 [Train]: loss=48.64, epe=3.10, lr=0.000050, samples/sec=32.5, sec/step=0.984, eta=4 days, 9:14:25\n",
      "2018-09-23 17:12:53 Iter 215000 [Val]: loss=51.38, epe=3.40\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-215000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-215000\n",
      "2018-09-23 17:31:25 Iter 216000 [Train]: loss=48.69, epe=nan, lr=0.000050, samples/sec=32.4, sec/step=0.989, eta=4 days, 9:29:19\n",
      "2018-09-23 17:31:48 Iter 216000 [Val]: loss=52.44, epe=3.42\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-216000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-216000\n",
      "2018-09-23 17:50:17 Iter 217000 [Train]: loss=48.66, epe=3.10, lr=0.000050, samples/sec=32.6, sec/step=0.982, eta=4 days, 8:28:23\n",
      "2018-09-23 17:50:40 Iter 217000 [Val]: loss=48.74, epe=3.17\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-217000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-217000\n",
      "2018-09-23 18:09:11 Iter 218000 [Train]: loss=49.08, epe=3.13, lr=0.000050, samples/sec=32.5, sec/step=0.984, eta=4 days, 8:25:31\n",
      "2018-09-23 18:09:33 Iter 218000 [Val]: loss=50.98, epe=3.30\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-218000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-218000\n",
      "2018-09-23 18:28:02 Iter 219000 [Train]: loss=48.53, epe=3.09, lr=0.000050, samples/sec=32.6, sec/step=0.980, eta=4 days, 7:44:42\n",
      "2018-09-23 18:28:24 Iter 219000 [Val]: loss=52.41, epe=3.43\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-219000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-219000\n",
      "2018-09-23 18:46:52 Iter 220000 [Train]: loss=48.49, epe=nan, lr=0.000050, samples/sec=32.6, sec/step=0.981, eta=4 days, 7:34:19\n",
      "2018-09-23 18:47:15 Iter 220000 [Val]: loss=49.96, epe=3.25\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-220000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-220000\n",
      "2018-09-23 19:05:46 Iter 221000 [Train]: loss=48.65, epe=3.10, lr=0.000050, samples/sec=32.5, sec/step=0.985, eta=4 days, 7:41:06\n",
      "2018-09-23 19:06:09 Iter 221000 [Val]: loss=52.22, epe=3.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-221000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-221000\n",
      "2018-09-23 19:24:41 Iter 222000 [Train]: loss=48.61, epe=3.09, lr=0.000050, samples/sec=32.3, sec/step=0.990, eta=4 days, 7:54:52\n",
      "2018-09-23 19:25:04 Iter 222000 [Val]: loss=50.76, epe=3.30\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-222000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-222000\n",
      "2018-09-23 19:43:25 Iter 223000 [Train]: loss=48.35, epe=3.08, lr=0.000050, samples/sec=32.6, sec/step=0.982, eta=4 days, 6:48:27\n",
      "2018-09-23 19:43:48 Iter 223000 [Val]: loss=53.79, epe=3.55\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-223000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-223000\n",
      "2018-09-23 20:02:07 Iter 224000 [Train]: loss=48.55, epe=3.09, lr=0.000050, samples/sec=32.7, sec/step=0.980, eta=4 days, 6:21:26\n",
      "2018-09-23 20:02:30 Iter 224000 [Val]: loss=49.48, epe=3.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-224000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-224000\n",
      "2018-09-23 20:20:53 Iter 225000 [Train]: loss=48.46, epe=3.08, lr=0.000050, samples/sec=32.5, sec/step=0.984, eta=4 days, 6:27:27\n",
      "2018-09-23 20:21:15 Iter 225000 [Val]: loss=50.40, epe=3.28\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-225000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-225000\n",
      "2018-09-23 20:39:36 Iter 226000 [Train]: loss=48.40, epe=3.08, lr=0.000050, samples/sec=32.7, sec/step=0.979, eta=4 days, 5:42:48\n",
      "2018-09-23 20:39:58 Iter 226000 [Val]: loss=53.31, epe=3.48\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-226000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-226000\n",
      "2018-09-23 20:58:19 Iter 227000 [Train]: loss=47.92, epe=3.04, lr=0.000050, samples/sec=32.6, sec/step=0.982, eta=4 days, 5:43:01\n",
      "2018-09-23 20:58:42 Iter 227000 [Val]: loss=50.38, epe=3.32\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-227000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-227000\n",
      "2018-09-23 21:17:18 Iter 228000 [Train]: loss=48.72, epe=3.10, lr=0.000050, samples/sec=32.7, sec/step=0.979, eta=4 days, 5:09:31\n",
      "2018-09-23 21:17:41 Iter 228000 [Val]: loss=51.01, epe=3.35\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-228000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-228000\n",
      "2018-09-23 21:36:05 Iter 229000 [Train]: loss=48.47, epe=3.08, lr=0.000050, samples/sec=32.5, sec/step=0.983, eta=4 days, 5:19:21\n",
      "2018-09-23 21:36:28 Iter 229000 [Val]: loss=52.42, epe=3.43\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-229000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-229000\n",
      "2018-09-23 21:54:53 Iter 230000 [Train]: loss=47.89, epe=3.04, lr=0.000050, samples/sec=32.7, sec/step=0.977, eta=4 days, 4:27:32\n",
      "2018-09-23 21:55:16 Iter 230000 [Val]: loss=49.41, epe=3.21\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-230000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-230000\n",
      "2018-09-23 22:13:37 Iter 231000 [Train]: loss=47.98, epe=3.05, lr=0.000050, samples/sec=32.8, sec/step=0.974, eta=4 days, 3:51:21\n",
      "2018-09-23 22:14:00 Iter 231000 [Val]: loss=48.29, epe=3.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-231000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-231000\n",
      "2018-09-23 22:32:23 Iter 232000 [Train]: loss=48.19, epe=nan, lr=0.000050, samples/sec=32.8, sec/step=0.977, eta=4 days, 3:50:36\n",
      "2018-09-23 22:32:46 Iter 232000 [Val]: loss=53.07, epe=3.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-232000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-232000\n",
      "2018-09-23 22:51:08 Iter 233000 [Train]: loss=48.33, epe=3.07, lr=0.000050, samples/sec=32.8, sec/step=0.974, eta=4 days, 3:20:38\n",
      "2018-09-23 22:51:31 Iter 233000 [Val]: loss=51.45, epe=3.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-233000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-233000\n",
      "2018-09-23 23:09:56 Iter 234000 [Train]: loss=48.19, epe=3.06, lr=0.000050, samples/sec=32.7, sec/step=0.978, eta=4 days, 3:22:48\n",
      "2018-09-23 23:10:19 Iter 234000 [Val]: loss=51.30, epe=3.33\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-234000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-234000\n",
      "2018-09-23 23:28:45 Iter 235000 [Train]: loss=48.06, epe=3.05, lr=0.000050, samples/sec=32.7, sec/step=0.979, eta=4 days, 3:15:59\n",
      "2018-09-23 23:29:08 Iter 235000 [Val]: loss=49.13, epe=3.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-235000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-235000\n",
      "2018-09-23 23:47:36 Iter 236000 [Train]: loss=47.96, epe=3.04, lr=0.000050, samples/sec=32.6, sec/step=0.982, eta=4 days, 3:15:27\n",
      "2018-09-23 23:47:59 Iter 236000 [Val]: loss=50.13, epe=3.28\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-236000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-236000\n",
      "2018-09-24 00:06:22 Iter 237000 [Train]: loss=47.99, epe=3.05, lr=0.000050, samples/sec=32.7, sec/step=0.977, eta=4 days, 2:32:39\n",
      "2018-09-24 00:06:45 Iter 237000 [Val]: loss=52.63, epe=3.46\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-237000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-237000\n",
      "2018-09-24 00:25:11 Iter 238000 [Train]: loss=47.82, epe=3.04, lr=0.000050, samples/sec=32.7, sec/step=0.979, eta=4 days, 2:27:45\n",
      "2018-09-24 00:25:34 Iter 238000 [Val]: loss=50.64, epe=3.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-238000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-238000\n",
      "2018-09-24 00:43:59 Iter 239000 [Train]: loss=48.10, epe=3.06, lr=0.000050, samples/sec=32.7, sec/step=0.978, eta=4 days, 2:04:45\n",
      "2018-09-24 00:44:22 Iter 239000 [Val]: loss=53.25, epe=3.50\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-239000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-239000\n",
      "2018-09-24 01:02:48 Iter 240000 [Train]: loss=48.23, epe=3.06, lr=0.000050, samples/sec=32.7, sec/step=0.978, eta=4 days, 1:50:33\n",
      "2018-09-24 01:03:11 Iter 240000 [Val]: loss=49.51, epe=3.21\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-240000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-240000\n",
      "2018-09-24 01:21:34 Iter 241000 [Train]: loss=47.13, epe=2.98, lr=0.000050, samples/sec=32.8, sec/step=0.975, eta=4 days, 1:16:44\n",
      "2018-09-24 01:21:57 Iter 241000 [Val]: loss=50.99, epe=3.35\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-241000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-241000\n",
      "2018-09-24 01:40:27 Iter 242000 [Train]: loss=48.15, epe=nan, lr=0.000050, samples/sec=32.6, sec/step=0.982, eta=4 days, 1:39:31\n",
      "2018-09-24 01:40:50 Iter 242000 [Val]: loss=51.93, epe=3.41\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-242000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-242000\n",
      "2018-09-24 01:59:14 Iter 243000 [Train]: loss=47.33, epe=3.00, lr=0.000050, samples/sec=32.8, sec/step=0.977, eta=4 days, 0:52:10\n",
      "2018-09-24 01:59:37 Iter 243000 [Val]: loss=50.77, epe=3.30\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-243000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-243000\n",
      "2018-09-24 02:18:05 Iter 244000 [Train]: loss=48.42, epe=3.08, lr=0.000050, samples/sec=32.6, sec/step=0.981, eta=4 days, 0:59:08\n",
      "2018-09-24 02:18:28 Iter 244000 [Val]: loss=47.28, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-244000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-244000\n",
      "2018-09-24 02:36:54 Iter 245000 [Train]: loss=47.34, epe=3.00, lr=0.000050, samples/sec=32.7, sec/step=0.979, eta=4 days, 0:32:29\n",
      "2018-09-24 02:37:17 Iter 245000 [Val]: loss=51.34, epe=3.34\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-245000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-245000\n",
      "2018-09-24 02:55:42 Iter 246000 [Train]: loss=47.77, epe=3.03, lr=0.000050, samples/sec=32.8, sec/step=0.977, eta=4 days, 0:03:23\n",
      "2018-09-24 02:56:05 Iter 246000 [Val]: loss=51.13, epe=3.34\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-246000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-246000\n",
      "2018-09-24 03:14:41 Iter 247000 [Train]: loss=47.90, epe=nan, lr=0.000050, samples/sec=32.3, sec/step=0.989, eta=4 days, 1:00:40\n",
      "2018-09-24 03:15:05 Iter 247000 [Val]: loss=51.45, epe=3.35\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-247000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-247000\n",
      "2018-09-24 03:34:18 Iter 248000 [Train]: loss=47.78, epe=3.03, lr=0.000050, samples/sec=31.3, sec/step=1.023, eta=4 days, 4:03:46\n",
      "2018-09-24 03:34:44 Iter 248000 [Val]: loss=50.93, epe=3.35\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-248000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-248000\n",
      "2018-09-24 03:54:05 Iter 249000 [Train]: loss=47.29, epe=2.99, lr=0.000050, samples/sec=31.0, sec/step=1.032, eta=4 days, 4:34:18\n",
      "2018-09-24 03:54:31 Iter 249000 [Val]: loss=49.26, epe=3.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-249000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-249000\n",
      "2018-09-24 04:14:00 Iter 250000 [Train]: loss=47.93, epe=nan, lr=0.000050, samples/sec=30.9, sec/step=1.036, eta=4 days, 4:43:57\n",
      "2018-09-24 04:14:26 Iter 250000 [Val]: loss=51.72, epe=3.38\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-250000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-250000\n",
      "2018-09-24 04:33:48 Iter 251000 [Train]: loss=47.62, epe=3.02, lr=0.000050, samples/sec=31.1, sec/step=1.030, eta=4 days, 3:51:00\n",
      "2018-09-24 04:34:14 Iter 251000 [Val]: loss=52.43, epe=3.45\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-251000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-251000\n",
      "2018-09-24 04:53:44 Iter 252000 [Train]: loss=47.71, epe=3.03, lr=0.000050, samples/sec=30.8, sec/step=1.037, eta=4 days, 4:17:18\n",
      "2018-09-24 04:54:12 Iter 252000 [Val]: loss=48.66, epe=3.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-252000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-252000\n",
      "2018-09-24 05:13:49 Iter 253000 [Train]: loss=47.73, epe=3.03, lr=0.000050, samples/sec=30.6, sec/step=1.045, eta=4 days, 4:43:09\n",
      "2018-09-24 05:14:16 Iter 253000 [Val]: loss=51.04, epe=3.34\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-253000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-253000\n",
      "2018-09-24 05:33:44 Iter 254000 [Train]: loss=47.53, epe=3.01, lr=0.000050, samples/sec=30.9, sec/step=1.036, eta=4 days, 3:33:40\n",
      "2018-09-24 05:34:11 Iter 254000 [Val]: loss=51.15, epe=3.35\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-254000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-254000\n",
      "2018-09-24 05:53:46 Iter 255000 [Train]: loss=47.24, epe=2.99, lr=0.000050, samples/sec=30.7, sec/step=1.042, eta=4 days, 3:52:48\n",
      "2018-09-24 05:54:13 Iter 255000 [Val]: loss=51.24, epe=3.36\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-255000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-255000\n",
      "2018-09-24 06:13:46 Iter 256000 [Train]: loss=47.49, epe=3.01, lr=0.000050, samples/sec=30.7, sec/step=1.041, eta=4 days, 3:27:36\n",
      "2018-09-24 06:14:13 Iter 256000 [Val]: loss=51.03, epe=3.34\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-256000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-256000\n",
      "2018-09-24 06:34:10 Iter 257000 [Train]: loss=47.43, epe=3.01, lr=0.000050, samples/sec=30.7, sec/step=1.043, eta=4 days, 3:21:59\n",
      "2018-09-24 06:34:37 Iter 257000 [Val]: loss=48.79, epe=3.17\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-257000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-257000\n",
      "2018-09-24 06:54:20 Iter 258000 [Train]: loss=47.42, epe=3.00, lr=0.000050, samples/sec=30.5, sec/step=1.050, eta=4 days, 3:43:59\n",
      "2018-09-24 06:54:48 Iter 258000 [Val]: loss=50.84, epe=3.32\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-258000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-258000\n",
      "2018-09-24 07:14:24 Iter 259000 [Train]: loss=47.81, epe=3.04, lr=0.000050, samples/sec=30.6, sec/step=1.044, eta=4 days, 2:55:39\n",
      "2018-09-24 07:14:51 Iter 259000 [Val]: loss=50.67, epe=3.29\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-259000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-259000\n",
      "2018-09-24 07:34:37 Iter 260000 [Train]: loss=47.24, epe=2.99, lr=0.000050, samples/sec=30.4, sec/step=1.054, eta=4 days, 3:30:10\n",
      "2018-09-24 07:35:05 Iter 260000 [Val]: loss=49.42, epe=3.23\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-260000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-260000\n",
      "2018-09-24 07:54:54 Iter 261000 [Train]: loss=46.91, epe=nan, lr=0.000050, samples/sec=30.3, sec/step=1.057, eta=4 days, 3:32:04\n",
      "2018-09-24 07:55:21 Iter 261000 [Val]: loss=49.11, epe=3.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-261000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-261000\n",
      "2018-09-24 08:15:15 Iter 262000 [Train]: loss=47.59, epe=3.02, lr=0.000050, samples/sec=30.2, sec/step=1.061, eta=4 days, 3:36:34\n",
      "2018-09-24 08:15:43 Iter 262000 [Val]: loss=52.81, epe=3.47\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.47) doesn't outperform other checkpoints\n",
      "2018-09-24 08:35:25 Iter 263000 [Train]: loss=47.32, epe=2.99, lr=0.000050, samples/sec=30.3, sec/step=1.055, eta=4 days, 2:43:56\n",
      "2018-09-24 08:35:52 Iter 263000 [Val]: loss=50.45, epe=3.29\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-263000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-263000\n",
      "2018-09-24 08:55:54 Iter 264000 [Train]: loss=47.06, epe=2.97, lr=0.000050, samples/sec=30.1, sec/step=1.065, eta=4 days, 3:23:05\n",
      "2018-09-24 08:56:22 Iter 264000 [Val]: loss=49.76, epe=3.23\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-264000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-264000\n",
      "2018-09-24 09:16:14 Iter 265000 [Train]: loss=47.09, epe=2.98, lr=0.000050, samples/sec=30.3, sec/step=1.058, eta=4 days, 2:25:34\n",
      "2018-09-24 09:16:41 Iter 265000 [Val]: loss=51.32, epe=3.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-265000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-265000\n",
      "2018-09-24 09:36:34 Iter 266000 [Train]: loss=47.39, epe=3.00, lr=0.000050, samples/sec=30.2, sec/step=1.059, eta=4 days, 2:16:48\n",
      "2018-09-24 09:37:02 Iter 266000 [Val]: loss=49.87, epe=3.25\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-266000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-266000\n",
      "2018-09-24 09:57:14 Iter 267000 [Train]: loss=47.10, epe=2.97, lr=0.000050, samples/sec=29.7, sec/step=1.077, eta=4 days, 3:36:42\n",
      "2018-09-24 09:57:42 Iter 267000 [Val]: loss=49.75, epe=3.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-267000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-267000\n",
      "2018-09-24 10:17:33 Iter 268000 [Train]: loss=47.33, epe=3.00, lr=0.000050, samples/sec=30.2, sec/step=1.060, eta=4 days, 1:46:27\n",
      "2018-09-24 10:18:00 Iter 268000 [Val]: loss=49.65, epe=3.25\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-268000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-268000\n",
      "2018-09-24 10:38:05 Iter 269000 [Train]: loss=46.90, epe=2.96, lr=0.000050, samples/sec=29.9, sec/step=1.071, eta=4 days, 2:27:29\n",
      "2018-09-24 10:38:33 Iter 269000 [Val]: loss=49.77, epe=3.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-269000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-269000\n",
      "2018-09-24 10:58:39 Iter 270000 [Train]: loss=46.87, epe=2.96, lr=0.000050, samples/sec=29.8, sec/step=1.073, eta=4 days, 2:22:54\n",
      "2018-09-24 10:59:08 Iter 270000 [Val]: loss=49.22, epe=3.18\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-270000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-270000\n",
      "2018-09-24 11:19:23 Iter 271000 [Train]: loss=47.39, epe=3.00, lr=0.000050, samples/sec=29.6, sec/step=1.081, eta=4 days, 2:49:43\n",
      "2018-09-24 11:19:51 Iter 271000 [Val]: loss=50.12, epe=3.27\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-271000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-271000\n",
      "2018-09-24 11:40:27 Iter 272000 [Train]: loss=46.91, epe=2.96, lr=0.000050, samples/sec=29.1, sec/step=1.099, eta=4 days, 4:09:26\n",
      "2018-09-24 11:40:56 Iter 272000 [Val]: loss=50.03, epe=3.24\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-272000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-272000\n",
      "2018-09-24 12:01:09 Iter 273000 [Train]: loss=47.08, epe=2.97, lr=0.000050, samples/sec=29.8, sec/step=1.075, eta=4 days, 1:40:22\n",
      "2018-09-24 12:01:37 Iter 273000 [Val]: loss=49.57, epe=3.20\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-273000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-273000\n",
      "2018-09-24 12:21:57 Iter 274000 [Train]: loss=46.67, epe=2.94, lr=0.000050, samples/sec=29.5, sec/step=1.083, eta=4 days, 2:04:08\n",
      "2018-09-24 12:22:26 Iter 274000 [Val]: loss=49.27, epe=3.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-274000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-274000\n",
      "2018-09-24 12:42:37 Iter 275000 [Train]: loss=47.31, epe=2.99, lr=0.000050, samples/sec=29.7, sec/step=1.076, eta=4 days, 1:09:20\n",
      "2018-09-24 12:43:05 Iter 275000 [Val]: loss=50.70, epe=3.30\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-275000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-275000\n",
      "2018-09-24 13:03:16 Iter 276000 [Train]: loss=46.97, epe=2.97, lr=0.000050, samples/sec=29.8, sec/step=1.075, eta=4 days, 0:43:36\n",
      "2018-09-24 13:03:45 Iter 276000 [Val]: loss=48.96, epe=3.17\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-276000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-276000\n",
      "2018-09-24 13:24:15 Iter 277000 [Train]: loss=46.66, epe=2.95, lr=0.000050, samples/sec=29.3, sec/step=1.091, eta=4 days, 1:52:11\n",
      "2018-09-24 13:24:44 Iter 277000 [Val]: loss=50.52, epe=3.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-277000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-277000\n",
      "2018-09-24 13:45:14 Iter 278000 [Train]: loss=46.99, epe=2.97, lr=0.000050, samples/sec=29.3, sec/step=1.091, eta=4 days, 1:35:40\n",
      "2018-09-24 13:45:43 Iter 278000 [Val]: loss=50.57, epe=3.29\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-278000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-278000\n",
      "2018-09-24 14:05:54 Iter 279000 [Train]: loss=47.27, epe=3.00, lr=0.000050, samples/sec=29.7, sec/step=1.078, eta=4 days, 0:05:12\n",
      "2018-09-24 14:06:22 Iter 279000 [Val]: loss=47.82, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-279000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-279000\n",
      "2018-09-24 14:26:21 Iter 280000 [Train]: loss=46.89, epe=2.96, lr=0.000050, samples/sec=29.9, sec/step=1.072, eta=3 days, 23:14:54\n",
      "2018-09-24 14:26:49 Iter 280000 [Val]: loss=52.17, epe=3.42\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.42) doesn't outperform other checkpoints\n",
      "2018-09-24 14:46:55 Iter 281000 [Train]: loss=46.65, epe=2.95, lr=0.000050, samples/sec=29.6, sec/step=1.080, eta=3 days, 23:41:09\n",
      "2018-09-24 14:47:24 Iter 281000 [Val]: loss=49.02, epe=3.18\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-281000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-281000\n",
      "2018-09-24 15:07:55 Iter 282000 [Train]: loss=47.04, epe=2.97, lr=0.000050, samples/sec=29.3, sec/step=1.093, eta=4 days, 0:32:43\n",
      "2018-09-24 15:08:24 Iter 282000 [Val]: loss=50.28, epe=3.28\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-282000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-282000\n",
      "2018-09-24 15:28:37 Iter 283000 [Train]: loss=46.79, epe=nan, lr=0.000050, samples/sec=29.6, sec/step=1.081, eta=3 days, 23:10:46\n",
      "2018-09-24 15:29:06 Iter 283000 [Val]: loss=50.68, epe=3.32\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-283000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-283000\n",
      "2018-09-24 15:49:30 Iter 284000 [Train]: loss=46.87, epe=2.97, lr=0.000050, samples/sec=29.3, sec/step=1.091, eta=3 days, 23:45:04\n",
      "2018-09-24 15:49:59 Iter 284000 [Val]: loss=49.46, epe=3.20\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-284000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-284000\n",
      "2018-09-24 16:10:10 Iter 285000 [Train]: loss=47.17, epe=2.98, lr=0.000050, samples/sec=29.6, sec/step=1.080, eta=3 days, 22:29:32\n",
      "2018-09-24 16:10:38 Iter 285000 [Val]: loss=50.58, epe=3.28\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-285000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-285000\n",
      "2018-09-24 16:30:59 Iter 286000 [Train]: loss=46.36, epe=2.92, lr=0.000050, samples/sec=29.4, sec/step=1.087, eta=3 days, 22:47:07\n",
      "2018-09-24 16:31:28 Iter 286000 [Val]: loss=50.52, epe=3.28\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-286000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-286000\n",
      "2018-09-24 16:52:05 Iter 287000 [Train]: loss=46.89, epe=2.97, lr=0.000050, samples/sec=29.1, sec/step=1.099, eta=3 days, 23:31:11\n",
      "2018-09-24 16:52:35 Iter 287000 [Val]: loss=50.40, epe=3.30\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-287000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-287000\n",
      "2018-09-24 17:13:18 Iter 288000 [Train]: loss=46.64, epe=2.95, lr=0.000050, samples/sec=29.0, sec/step=1.104, eta=3 days, 23:43:11\n",
      "2018-09-24 17:13:48 Iter 288000 [Val]: loss=47.05, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-288000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-288000\n",
      "2018-09-24 17:34:24 Iter 289000 [Train]: loss=47.04, epe=2.97, lr=0.000050, samples/sec=29.2, sec/step=1.097, eta=3 days, 22:47:26\n",
      "2018-09-24 17:34:53 Iter 289000 [Val]: loss=50.65, epe=3.29\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-289000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-289000\n",
      "2018-09-24 17:55:15 Iter 290000 [Train]: loss=46.32, epe=2.92, lr=0.000050, samples/sec=29.4, sec/step=1.088, eta=3 days, 21:42:00\n",
      "2018-09-24 17:55:44 Iter 290000 [Val]: loss=49.90, epe=3.23\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-290000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-290000\n",
      "2018-09-24 18:16:08 Iter 291000 [Train]: loss=46.73, epe=2.95, lr=0.000050, samples/sec=29.4, sec/step=1.090, eta=3 days, 21:32:17\n",
      "2018-09-24 18:16:37 Iter 291000 [Val]: loss=51.37, epe=3.37\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.37) doesn't outperform other checkpoints\n",
      "2018-09-24 18:36:45 Iter 292000 [Train]: loss=46.34, epe=2.91, lr=0.000050, samples/sec=29.6, sec/step=1.081, eta=3 days, 20:27:08\n",
      "2018-09-24 18:37:14 Iter 292000 [Val]: loss=47.02, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-292000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-292000\n",
      "2018-09-24 18:58:17 Iter 293000 [Train]: loss=46.94, epe=2.97, lr=0.000050, samples/sec=29.9, sec/step=1.071, eta=3 days, 19:22:06\n",
      "2018-09-24 18:58:44 Iter 293000 [Val]: loss=51.23, epe=3.36\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.36) doesn't outperform other checkpoints\n",
      "2018-09-24 19:18:49 Iter 294000 [Train]: loss=46.92, epe=2.96, lr=0.000050, samples/sec=29.7, sec/step=1.077, eta=3 days, 19:32:53\n",
      "2018-09-24 19:19:18 Iter 294000 [Val]: loss=49.13, epe=3.18\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-294000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-294000\n",
      "2018-09-24 19:39:53 Iter 295000 [Train]: loss=46.29, epe=2.92, lr=0.000050, samples/sec=29.1, sec/step=1.100, eta=3 days, 21:12:06\n",
      "2018-09-24 19:40:23 Iter 295000 [Val]: loss=47.05, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-295000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-295000\n",
      "2018-09-24 20:00:51 Iter 296000 [Train]: loss=46.49, epe=2.93, lr=0.000050, samples/sec=29.3, sec/step=1.093, eta=3 days, 20:17:56\n",
      "2018-09-24 20:01:21 Iter 296000 [Val]: loss=52.35, epe=3.40\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.40) doesn't outperform other checkpoints\n",
      "2018-09-24 20:21:40 Iter 297000 [Train]: loss=46.79, epe=2.95, lr=0.000050, samples/sec=29.4, sec/step=1.090, eta=3 days, 19:44:43\n",
      "2018-09-24 20:22:09 Iter 297000 [Val]: loss=49.48, epe=3.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-297000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-297000\n",
      "2018-09-24 20:42:18 Iter 298000 [Train]: loss=46.19, epe=2.91, lr=0.000050, samples/sec=29.7, sec/step=1.076, eta=3 days, 18:14:14\n",
      "2018-09-24 20:42:46 Iter 298000 [Val]: loss=46.71, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-298000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-298000\n",
      "2018-09-24 21:03:06 Iter 299000 [Train]: loss=46.91, epe=2.96, lr=0.000050, samples/sec=29.5, sec/step=1.086, eta=3 days, 18:47:19\n",
      "2018-09-24 21:03:35 Iter 299000 [Val]: loss=53.66, epe=3.53\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.53) doesn't outperform other checkpoints\n",
      "2018-09-24 21:23:48 Iter 300000 [Train]: loss=46.22, epe=2.91, lr=0.000050, samples/sec=29.5, sec/step=1.085, eta=3 days, 18:27:21\n",
      "2018-09-24 21:24:17 Iter 300000 [Val]: loss=47.68, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-300000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-300000\n",
      "2018-09-24 21:45:14 Iter 301000 [Train]: loss=46.08, epe=2.90, lr=0.000025, samples/sec=28.7, sec/step=1.117, eta=3 days, 20:44:31\n",
      "2018-09-24 21:45:44 Iter 301000 [Val]: loss=48.53, epe=3.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-301000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-301000\n",
      "2018-09-24 22:06:45 Iter 302000 [Train]: loss=45.53, epe=2.85, lr=0.000025, samples/sec=28.6, sec/step=1.117, eta=3 days, 20:28:13\n",
      "2018-09-24 22:07:16 Iter 302000 [Val]: loss=50.03, epe=3.23\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-302000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-302000\n",
      "2018-09-24 22:28:07 Iter 303000 [Train]: loss=44.66, epe=2.79, lr=0.000025, samples/sec=28.8, sec/step=1.109, eta=3 days, 19:31:41\n",
      "2018-09-24 22:28:37 Iter 303000 [Val]: loss=46.46, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-303000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-303000\n",
      "2018-09-24 22:49:15 Iter 304000 [Train]: loss=45.24, epe=2.83, lr=0.000025, samples/sec=29.2, sec/step=1.097, eta=3 days, 18:11:12\n",
      "2018-09-24 22:49:45 Iter 304000 [Val]: loss=47.07, epe=3.03\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-304000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-304000\n",
      "2018-09-24 23:10:35 Iter 305000 [Train]: loss=45.37, epe=2.84, lr=0.000025, samples/sec=28.9, sec/step=1.108, eta=3 days, 18:49:13\n",
      "2018-09-24 23:11:05 Iter 305000 [Val]: loss=52.12, epe=3.41\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.41) doesn't outperform other checkpoints\n",
      "2018-09-24 23:31:39 Iter 306000 [Train]: loss=44.94, epe=2.81, lr=0.000025, samples/sec=29.1, sec/step=1.099, eta=3 days, 17:47:16\n",
      "2018-09-24 23:32:09 Iter 306000 [Val]: loss=46.50, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-306000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-306000\n",
      "2018-09-24 23:53:08 Iter 307000 [Train]: loss=45.36, epe=2.84, lr=0.000025, samples/sec=28.7, sec/step=1.116, eta=3 days, 18:48:47\n",
      "2018-09-24 23:53:38 Iter 307000 [Val]: loss=48.72, epe=3.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-307000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-307000\n",
      "2018-09-25 00:14:41 Iter 308000 [Train]: loss=45.25, epe=2.83, lr=0.000025, samples/sec=28.6, sec/step=1.119, eta=3 days, 18:43:48\n",
      "2018-09-25 00:15:11 Iter 308000 [Val]: loss=47.40, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-308000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-308000\n",
      "2018-09-25 00:36:20 Iter 309000 [Train]: loss=44.88, epe=2.81, lr=0.000025, samples/sec=28.5, sec/step=1.124, eta=3 days, 18:49:56\n",
      "2018-09-25 00:36:50 Iter 309000 [Val]: loss=49.54, epe=3.17\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-309000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-309000\n",
      "2018-09-25 00:58:11 Iter 310000 [Train]: loss=45.59, epe=2.86, lr=0.000025, samples/sec=28.2, sec/step=1.134, eta=3 days, 19:23:20\n",
      "2018-09-25 00:58:43 Iter 310000 [Val]: loss=47.03, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-310000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-310000\n",
      "2018-09-25 01:19:49 Iter 311000 [Train]: loss=44.89, epe=2.81, lr=0.000025, samples/sec=28.5, sec/step=1.121, eta=3 days, 18:01:47\n",
      "2018-09-25 01:20:20 Iter 311000 [Val]: loss=48.45, epe=3.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-311000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-311000\n",
      "2018-09-25 01:41:35 Iter 312000 [Train]: loss=45.06, epe=2.82, lr=0.000025, samples/sec=28.3, sec/step=1.130, eta=3 days, 18:24:17\n",
      "2018-09-25 01:42:06 Iter 312000 [Val]: loss=46.92, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-312000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-312000\n",
      "2018-09-25 02:03:23 Iter 313000 [Train]: loss=44.84, epe=2.80, lr=0.000025, samples/sec=28.3, sec/step=1.132, eta=3 days, 18:13:53\n",
      "2018-09-25 02:03:54 Iter 313000 [Val]: loss=48.80, epe=3.16\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-313000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-313000\n",
      "2018-09-25 02:25:03 Iter 314000 [Train]: loss=44.74, epe=2.80, lr=0.000025, samples/sec=28.5, sec/step=1.124, eta=3 days, 17:17:44\n",
      "2018-09-25 02:25:34 Iter 314000 [Val]: loss=47.96, epe=3.10\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-314000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-314000\n",
      "2018-09-25 02:46:48 Iter 315000 [Train]: loss=44.96, epe=2.81, lr=0.000025, samples/sec=28.4, sec/step=1.128, eta=3 days, 17:19:21\n",
      "2018-09-25 02:47:20 Iter 315000 [Val]: loss=47.92, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-315000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-315000\n",
      "2018-09-25 03:08:41 Iter 316000 [Train]: loss=45.26, epe=2.83, lr=0.000025, samples/sec=28.2, sec/step=1.135, eta=3 days, 17:32:43\n",
      "2018-09-25 03:09:13 Iter 316000 [Val]: loss=49.48, epe=3.20\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-316000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-316000\n",
      "2018-09-25 03:30:24 Iter 317000 [Train]: loss=44.85, epe=2.81, lr=0.000025, samples/sec=28.4, sec/step=1.126, eta=3 days, 16:32:27\n",
      "2018-09-25 03:30:56 Iter 317000 [Val]: loss=46.89, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-317000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-317000\n",
      "2018-09-25 03:52:17 Iter 318000 [Train]: loss=44.48, epe=2.78, lr=0.000025, samples/sec=28.2, sec/step=1.135, eta=3 days, 16:52:30\n",
      "2018-09-25 03:52:49 Iter 318000 [Val]: loss=47.97, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-318000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-318000\n",
      "2018-09-25 04:14:16 Iter 319000 [Train]: loss=45.06, epe=2.82, lr=0.000025, samples/sec=28.0, sec/step=1.142, eta=3 days, 17:06:10\n",
      "2018-09-25 04:14:47 Iter 319000 [Val]: loss=49.39, epe=3.21\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-319000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-319000\n",
      "2018-09-25 04:35:49 Iter 320000 [Train]: loss=44.96, epe=2.81, lr=0.000025, samples/sec=28.5, sec/step=1.123, eta=3 days, 15:21:10\n",
      "2018-09-25 04:36:20 Iter 320000 [Val]: loss=48.39, epe=3.12\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-320000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-320000\n",
      "2018-09-25 04:57:46 Iter 321000 [Train]: loss=44.53, epe=2.78, lr=0.000025, samples/sec=28.0, sec/step=1.141, eta=3 days, 16:25:06\n",
      "2018-09-25 04:58:18 Iter 321000 [Val]: loss=48.02, epe=3.10\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-321000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-321000\n",
      "2018-09-25 05:19:44 Iter 322000 [Train]: loss=45.13, epe=2.82, lr=0.000025, samples/sec=28.1, sec/step=1.139, eta=3 days, 15:57:42\n",
      "2018-09-25 05:20:16 Iter 322000 [Val]: loss=47.78, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-322000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-322000\n",
      "2018-09-25 05:41:51 Iter 323000 [Train]: loss=44.60, epe=2.79, lr=0.000025, samples/sec=27.9, sec/step=1.147, eta=3 days, 16:15:29\n",
      "2018-09-25 05:42:24 Iter 323000 [Val]: loss=46.54, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-323000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-323000\n",
      "2018-09-25 06:03:59 Iter 324000 [Train]: loss=44.95, epe=2.81, lr=0.000025, samples/sec=27.9, sec/step=1.146, eta=3 days, 15:53:27\n",
      "2018-09-25 06:04:32 Iter 324000 [Val]: loss=48.25, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-324000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-324000\n",
      "2018-09-25 06:26:22 Iter 325000 [Train]: loss=44.57, epe=nan, lr=0.000025, samples/sec=27.6, sec/step=1.160, eta=3 days, 16:38:45\n",
      "2018-09-25 06:26:56 Iter 325000 [Val]: loss=46.90, epe=3.03\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-325000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-325000\n",
      "2018-09-25 06:48:32 Iter 326000 [Train]: loss=44.68, epe=2.79, lr=0.000025, samples/sec=27.9, sec/step=1.148, eta=3 days, 15:24:31\n",
      "2018-09-25 06:49:05 Iter 326000 [Val]: loss=49.04, epe=3.18\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-326000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-326000\n",
      "2018-09-25 07:10:59 Iter 327000 [Train]: loss=44.72, epe=2.80, lr=0.000025, samples/sec=27.5, sec/step=1.165, eta=3 days, 16:18:30\n",
      "2018-09-25 07:11:32 Iter 327000 [Val]: loss=47.83, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-327000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-327000\n",
      "2018-09-25 07:33:12 Iter 328000 [Train]: loss=44.83, epe=2.80, lr=0.000025, samples/sec=27.8, sec/step=1.151, eta=3 days, 14:59:46\n",
      "2018-09-25 07:33:45 Iter 328000 [Val]: loss=50.60, epe=3.32\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.32) doesn't outperform other checkpoints\n",
      "2018-09-25 07:55:18 Iter 329000 [Train]: loss=44.70, epe=2.79, lr=0.000025, samples/sec=27.8, sec/step=1.150, eta=3 days, 14:33:46\n",
      "2018-09-25 07:55:50 Iter 329000 [Val]: loss=48.80, epe=3.16\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-329000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-329000\n",
      "2018-09-25 08:17:37 Iter 330000 [Train]: loss=44.72, epe=2.80, lr=0.000025, samples/sec=27.6, sec/step=1.158, eta=3 days, 14:48:45\n",
      "2018-09-25 08:18:11 Iter 330000 [Val]: loss=48.26, epe=3.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-330000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-330000\n",
      "2018-09-25 08:40:17 Iter 331000 [Train]: loss=44.59, epe=2.78, lr=0.000025, samples/sec=27.2, sec/step=1.175, eta=3 days, 15:49:29\n",
      "2018-09-25 08:40:51 Iter 331000 [Val]: loss=46.64, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-331000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-331000\n",
      "2018-09-25 09:02:50 Iter 332000 [Train]: loss=44.76, epe=2.80, lr=0.000025, samples/sec=27.4, sec/step=1.169, eta=3 days, 15:02:08\n",
      "2018-09-25 09:03:23 Iter 332000 [Val]: loss=47.63, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-332000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-332000\n",
      "2018-09-25 09:25:29 Iter 333000 [Train]: loss=44.56, epe=2.78, lr=0.000025, samples/sec=27.2, sec/step=1.177, eta=3 days, 15:16:18\n",
      "2018-09-25 09:26:03 Iter 333000 [Val]: loss=48.37, epe=3.10\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-333000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-333000\n",
      "2018-09-25 09:48:26 Iter 334000 [Train]: loss=44.79, epe=2.80, lr=0.000025, samples/sec=26.8, sec/step=1.192, eta=3 days, 16:04:21\n",
      "2018-09-25 09:49:01 Iter 334000 [Val]: loss=48.12, epe=3.11\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-334000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-334000\n",
      "2018-09-25 10:11:07 Iter 335000 [Train]: loss=44.32, epe=2.77, lr=0.000025, samples/sec=27.2, sec/step=1.174, eta=3 days, 14:26:36\n",
      "2018-09-25 10:11:41 Iter 335000 [Val]: loss=46.29, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-335000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-335000\n",
      "2018-09-25 10:35:25 Iter 336000 [Train]: loss=44.61, epe=2.79, lr=0.000025, samples/sec=26.9, sec/step=1.191, eta=3 days, 15:18:31\n",
      "2018-09-25 10:35:59 Iter 336000 [Val]: loss=49.38, epe=3.17\n",
      "\n",
      "Resume training from step 335001...\n",
      "2018-09-25 12:51:11 Iter 336000 [Train]: loss=44.52, epe=2.78, lr=0.000025, samples/sec=45.1, sec/step=0.709, eta=2 days, 4:00:32\n",
      "2018-09-25 12:51:32 Iter 336000 [Val]: loss=47.82, epe=3.08\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-336000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-336000\n",
      "2018-09-25 13:04:01 Iter 337000 [Train]: loss=44.53, epe=2.78, lr=0.000025, samples/sec=50.3, sec/step=0.637, eta=1 day, 22:30:37\n",
      "2018-09-25 13:04:14 Iter 337000 [Val]: loss=46.76, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-337000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-337000\n",
      "2018-09-25 13:16:43 Iter 338000 [Train]: loss=44.69, epe=2.79, lr=0.000025, samples/sec=50.0, sec/step=0.640, eta=1 day, 22:35:00\n",
      "2018-09-25 13:16:55 Iter 338000 [Val]: loss=49.11, epe=3.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-338000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-338000\n",
      "2018-09-25 13:29:29 Iter 339000 [Train]: loss=44.57, epe=nan, lr=0.000025, samples/sec=49.6, sec/step=0.645, eta=1 day, 22:46:58\n",
      "2018-09-25 13:29:41 Iter 339000 [Val]: loss=47.24, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-339000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-339000\n",
      "2018-09-25 13:42:16 Iter 340000 [Train]: loss=44.57, epe=2.78, lr=0.000025, samples/sec=49.6, sec/step=0.645, eta=1 day, 22:34:04\n",
      "2018-09-25 13:42:28 Iter 340000 [Val]: loss=47.95, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-340000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-340000\n",
      "2018-09-25 13:55:06 Iter 341000 [Train]: loss=44.32, epe=2.77, lr=0.000025, samples/sec=49.3, sec/step=0.650, eta=1 day, 22:44:24\n",
      "2018-09-25 13:55:19 Iter 341000 [Val]: loss=46.84, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-341000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-341000\n",
      "2018-09-25 14:08:03 Iter 342000 [Train]: loss=44.45, epe=2.78, lr=0.000025, samples/sec=48.8, sec/step=0.656, eta=1 day, 23:02:03\n",
      "2018-09-25 14:08:16 Iter 342000 [Val]: loss=48.59, epe=3.16\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-342000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-342000\n",
      "2018-09-25 14:21:06 Iter 343000 [Train]: loss=44.29, epe=2.77, lr=0.000025, samples/sec=48.7, sec/step=0.658, eta=1 day, 22:56:48\n",
      "2018-09-25 14:21:18 Iter 343000 [Val]: loss=48.25, epe=3.12\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-343000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-343000\n",
      "2018-09-25 14:34:07 Iter 344000 [Train]: loss=44.86, epe=2.81, lr=0.000025, samples/sec=48.5, sec/step=0.660, eta=1 day, 22:54:25\n",
      "2018-09-25 14:34:20 Iter 344000 [Val]: loss=48.12, epe=3.11\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-344000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "\n",
      "Resume training from step 344001...\n",
      "2018-09-25 15:37:10 Iter 345000 [Train]: loss=44.35, epe=2.77, lr=0.000025, samples/sec=45.4, sec/step=0.704, eta=2 days, 1:52:37\n",
      "2018-09-25 15:37:30 Iter 345000 [Val]: loss=47.77, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-345000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-345000\n",
      "2018-09-25 15:50:01 Iter 346000 [Train]: loss=44.48, epe=2.77, lr=0.000025, samples/sec=50.3, sec/step=0.637, eta=1 day, 20:55:35\n",
      "2018-09-25 15:50:13 Iter 346000 [Val]: loss=48.09, epe=3.12\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-346000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-346000\n",
      "2018-09-25 16:02:47 Iter 347000 [Train]: loss=44.62, epe=2.79, lr=0.000025, samples/sec=49.7, sec/step=0.643, eta=1 day, 21:13:04\n",
      "2018-09-25 16:02:59 Iter 347000 [Val]: loss=48.65, epe=3.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-347000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-347000\n",
      "2018-09-25 16:15:32 Iter 348000 [Train]: loss=44.53, epe=2.78, lr=0.000025, samples/sec=49.8, sec/step=0.642, eta=1 day, 20:58:23\n",
      "2018-09-25 16:15:44 Iter 348000 [Val]: loss=48.04, epe=3.12\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-348000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-348000\n",
      "2018-09-25 16:28:22 Iter 349000 [Train]: loss=44.40, epe=2.77, lr=0.000025, samples/sec=49.4, sec/step=0.647, eta=1 day, 21:07:40\n",
      "2018-09-25 16:28:35 Iter 349000 [Val]: loss=47.80, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-349000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-349000\n",
      "2018-09-25 16:41:13 Iter 350000 [Train]: loss=44.19, epe=2.76, lr=0.000025, samples/sec=49.3, sec/step=0.649, eta=1 day, 21:03:51\n",
      "2018-09-25 16:41:26 Iter 350000 [Val]: loss=47.33, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-350000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-350000\n",
      "2018-09-25 16:54:01 Iter 351000 [Train]: loss=44.50, epe=2.77, lr=0.000025, samples/sec=49.5, sec/step=0.647, eta=1 day, 20:43:10\n",
      "2018-09-25 16:54:13 Iter 351000 [Val]: loss=48.05, epe=3.10\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-351000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-351000\n",
      "2018-09-25 17:06:55 Iter 352000 [Train]: loss=44.43, epe=2.78, lr=0.000025, samples/sec=49.1, sec/step=0.652, eta=1 day, 20:55:28\n",
      "2018-09-25 17:07:08 Iter 352000 [Val]: loss=47.44, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-352000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-352000\n",
      "2018-09-25 17:19:52 Iter 353000 [Train]: loss=44.75, epe=2.79, lr=0.000025, samples/sec=48.9, sec/step=0.655, eta=1 day, 20:55:58\n",
      "2018-09-25 17:20:05 Iter 353000 [Val]: loss=47.36, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-353000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-353000\n",
      "2018-09-25 17:32:54 Iter 354000 [Train]: loss=44.33, epe=2.77, lr=0.000025, samples/sec=48.6, sec/step=0.659, eta=1 day, 21:02:08\n",
      "2018-09-25 17:33:07 Iter 354000 [Val]: loss=50.35, epe=3.28\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-354000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-354000\n",
      "2018-09-25 17:46:00 Iter 355000 [Train]: loss=44.04, epe=2.75, lr=0.000025, samples/sec=48.2, sec/step=0.664, eta=1 day, 21:11:44\n",
      "2018-09-25 17:46:13 Iter 355000 [Val]: loss=46.54, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-355000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-355000\n",
      "2018-09-25 17:59:10 Iter 356000 [Train]: loss=44.68, epe=2.79, lr=0.000025, samples/sec=48.1, sec/step=0.665, eta=1 day, 21:04:49\n",
      "2018-09-25 17:59:23 Iter 356000 [Val]: loss=48.08, epe=3.08\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-356000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-356000\n",
      "2018-09-25 18:12:28 Iter 357000 [Train]: loss=44.41, epe=2.77, lr=0.000025, samples/sec=47.4, sec/step=0.675, eta=1 day, 21:33:24\n",
      "2018-09-25 18:12:42 Iter 357000 [Val]: loss=46.06, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-357000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-357000\n",
      "2018-09-25 18:25:50 Iter 358000 [Train]: loss=43.95, epe=2.74, lr=0.000025, samples/sec=47.2, sec/step=0.678, eta=1 day, 21:35:18\n",
      "2018-09-25 18:26:03 Iter 358000 [Val]: loss=48.63, epe=3.16\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-358000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-358000\n",
      "2018-09-25 18:39:16 Iter 359000 [Train]: loss=44.48, epe=2.78, lr=0.000025, samples/sec=46.9, sec/step=0.682, eta=1 day, 21:41:22\n",
      "2018-09-25 18:39:29 Iter 359000 [Val]: loss=48.45, epe=3.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-359000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-359000\n",
      "2018-09-25 18:52:46 Iter 360000 [Train]: loss=44.33, epe=2.76, lr=0.000025, samples/sec=46.8, sec/step=0.684, eta=1 day, 21:35:25\n",
      "2018-09-25 18:52:59 Iter 360000 [Val]: loss=48.47, epe=3.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-360000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-360000\n",
      "2018-09-25 19:06:19 Iter 361000 [Train]: loss=43.91, epe=2.74, lr=0.000025, samples/sec=46.5, sec/step=0.689, eta=1 day, 21:43:00\n",
      "2018-09-25 19:06:33 Iter 361000 [Val]: loss=48.56, epe=3.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-361000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-361000\n",
      "2018-09-25 19:20:00 Iter 362000 [Train]: loss=44.32, epe=2.77, lr=0.000025, samples/sec=46.0, sec/step=0.696, eta=1 day, 22:00:19\n",
      "2018-09-25 19:20:14 Iter 362000 [Val]: loss=45.90, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-362000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-362000\n",
      "2018-09-25 19:33:42 Iter 363000 [Train]: loss=43.98, epe=2.74, lr=0.000025, samples/sec=46.0, sec/step=0.696, eta=1 day, 21:48:31\n",
      "2018-09-25 19:33:55 Iter 363000 [Val]: loss=47.73, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-363000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-363000\n",
      "2018-09-25 19:47:29 Iter 364000 [Train]: loss=44.23, epe=2.76, lr=0.000025, samples/sec=45.6, sec/step=0.702, eta=1 day, 22:02:03\n",
      "2018-09-25 19:47:43 Iter 364000 [Val]: loss=47.48, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-364000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-364000\n",
      "2018-09-25 20:01:25 Iter 365000 [Train]: loss=44.42, epe=2.78, lr=0.000025, samples/sec=45.1, sec/step=0.709, eta=1 day, 22:18:19\n",
      "2018-09-25 20:01:39 Iter 365000 [Val]: loss=47.50, epe=3.06\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-365000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-365000\n",
      "2018-09-25 20:15:30 Iter 366000 [Train]: loss=43.98, epe=2.74, lr=0.000025, samples/sec=44.8, sec/step=0.715, eta=1 day, 22:28:16\n",
      "2018-09-25 20:15:44 Iter 366000 [Val]: loss=49.10, epe=3.19\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.19) doesn't outperform other checkpoints\n",
      "2018-09-25 20:29:31 Iter 367000 [Train]: loss=44.10, epe=nan, lr=0.000025, samples/sec=44.6, sec/step=0.718, eta=1 day, 22:27:30\n",
      "2018-09-25 20:29:45 Iter 367000 [Val]: loss=46.40, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-367000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-367000\n",
      "\n",
      "Resume training from step 367001...\n",
      "2018-09-25 22:18:26 Iter 368000 [Train]: loss=44.22, epe=2.76, lr=0.000025, samples/sec=45.7, sec/step=0.700, eta=1 day, 21:05:49\n",
      "2018-09-25 22:18:46 Iter 368000 [Val]: loss=47.62, epe=3.06\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-368000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-368000\n",
      "2018-09-25 22:31:12 Iter 369000 [Train]: loss=44.11, epe=2.75, lr=0.000025, samples/sec=50.6, sec/step=0.632, eta=1 day, 16:32:53\n",
      "2018-09-25 22:31:24 Iter 369000 [Val]: loss=47.03, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-369000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-369000\n",
      "2018-09-25 22:43:49 Iter 370000 [Train]: loss=43.91, epe=2.74, lr=0.000025, samples/sec=50.3, sec/step=0.636, eta=1 day, 16:38:25\n",
      "2018-09-25 22:44:01 Iter 370000 [Val]: loss=47.83, epe=3.09\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-370000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-370000\n",
      "2018-09-25 22:56:28 Iter 371000 [Train]: loss=44.47, epe=2.78, lr=0.000025, samples/sec=50.2, sec/step=0.637, eta=1 day, 16:32:39\n",
      "2018-09-25 22:56:40 Iter 371000 [Val]: loss=47.22, epe=3.03\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-371000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-371000\n",
      "2018-09-25 23:09:11 Iter 372000 [Train]: loss=44.01, epe=2.74, lr=0.000025, samples/sec=49.9, sec/step=0.642, eta=1 day, 16:38:46\n",
      "2018-09-25 23:09:24 Iter 372000 [Val]: loss=46.75, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-372000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-372000\n",
      "2018-09-25 23:21:55 Iter 373000 [Train]: loss=44.26, epe=2.76, lr=0.000025, samples/sec=49.8, sec/step=0.643, eta=1 day, 16:31:54\n",
      "2018-09-25 23:22:08 Iter 373000 [Val]: loss=49.09, epe=3.17\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-373000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-373000\n",
      "2018-09-25 23:34:46 Iter 374000 [Train]: loss=43.91, epe=2.74, lr=0.000025, samples/sec=49.3, sec/step=0.649, eta=1 day, 16:45:07\n",
      "2018-09-25 23:34:58 Iter 374000 [Val]: loss=47.23, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-374000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-374000\n",
      "2018-09-25 23:47:37 Iter 375000 [Train]: loss=44.18, epe=2.75, lr=0.000025, samples/sec=49.3, sec/step=0.649, eta=1 day, 16:34:51\n",
      "2018-09-25 23:47:50 Iter 375000 [Val]: loss=48.32, epe=3.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-375000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-375000\n",
      "2018-09-26 00:00:36 Iter 376000 [Train]: loss=44.08, epe=2.75, lr=0.000025, samples/sec=48.8, sec/step=0.656, eta=1 day, 16:48:39\n",
      "2018-09-26 00:00:48 Iter 376000 [Val]: loss=48.00, epe=3.08\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-376000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-376000\n",
      "2018-09-26 00:13:35 Iter 377000 [Train]: loss=43.98, epe=2.74, lr=0.000025, samples/sec=48.7, sec/step=0.657, eta=1 day, 16:40:40\n",
      "2018-09-26 00:13:47 Iter 377000 [Val]: loss=45.57, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-377000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-377000\n",
      "2018-09-26 00:26:41 Iter 378000 [Train]: loss=44.22, epe=2.76, lr=0.000025, samples/sec=48.2, sec/step=0.664, eta=1 day, 16:55:46\n",
      "2018-09-26 00:26:54 Iter 378000 [Val]: loss=47.43, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-378000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-378000\n",
      "2018-09-26 00:39:49 Iter 379000 [Train]: loss=44.07, epe=2.74, lr=0.000025, samples/sec=48.2, sec/step=0.664, eta=1 day, 16:43:57\n",
      "2018-09-26 00:40:02 Iter 379000 [Val]: loss=48.59, epe=3.13\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-379000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-379000\n",
      "2018-09-26 00:53:04 Iter 380000 [Train]: loss=43.55, epe=2.71, lr=0.000025, samples/sec=47.7, sec/step=0.671, eta=1 day, 17:00:01\n",
      "2018-09-26 00:53:17 Iter 380000 [Val]: loss=47.46, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-380000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-380000\n",
      "2018-09-26 01:06:23 Iter 381000 [Train]: loss=44.23, epe=2.76, lr=0.000025, samples/sec=47.4, sec/step=0.675, eta=1 day, 17:04:15\n",
      "2018-09-26 01:06:36 Iter 381000 [Val]: loss=46.21, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-381000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-381000\n",
      "2018-09-26 01:19:46 Iter 382000 [Train]: loss=44.18, epe=2.75, lr=0.000025, samples/sec=47.1, sec/step=0.679, eta=1 day, 17:06:56\n",
      "2018-09-26 01:20:00 Iter 382000 [Val]: loss=47.56, epe=3.06\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-382000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-382000\n",
      "2018-09-26 01:33:19 Iter 383000 [Train]: loss=43.78, epe=2.73, lr=0.000025, samples/sec=46.7, sec/step=0.685, eta=1 day, 17:16:42\n",
      "2018-09-26 01:33:32 Iter 383000 [Val]: loss=46.15, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-383000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-383000\n",
      "2018-09-26 01:46:53 Iter 384000 [Train]: loss=44.42, epe=2.77, lr=0.000025, samples/sec=46.5, sec/step=0.689, eta=1 day, 17:19:50\n",
      "2018-09-26 01:47:07 Iter 384000 [Val]: loss=48.25, epe=3.12\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.12) doesn't outperform other checkpoints\n",
      "2018-09-26 02:00:22 Iter 385000 [Train]: loss=43.56, epe=2.71, lr=0.000025, samples/sec=46.5, sec/step=0.688, eta=1 day, 17:05:04\n",
      "2018-09-26 02:00:36 Iter 385000 [Val]: loss=45.99, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-385000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-385000\n",
      "2018-09-26 02:14:06 Iter 386000 [Train]: loss=44.06, epe=2.74, lr=0.000025, samples/sec=45.9, sec/step=0.697, eta=1 day, 17:27:37\n",
      "2018-09-26 02:14:19 Iter 386000 [Val]: loss=47.22, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-386000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-386000\n",
      "2018-09-26 02:28:01 Iter 387000 [Train]: loss=44.38, epe=2.77, lr=0.000025, samples/sec=45.2, sec/step=0.709, eta=1 day, 17:55:17\n",
      "2018-09-26 02:28:15 Iter 387000 [Val]: loss=49.24, epe=3.20\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.20) doesn't outperform other checkpoints\n",
      "2018-09-26 02:41:51 Iter 388000 [Train]: loss=43.82, epe=2.73, lr=0.000025, samples/sec=45.2, sec/step=0.708, eta=1 day, 17:40:26\n",
      "2018-09-26 02:42:05 Iter 388000 [Val]: loss=46.30, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-388000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-388000\n",
      "2018-09-26 02:55:47 Iter 389000 [Train]: loss=43.87, epe=2.73, lr=0.000025, samples/sec=45.3, sec/step=0.706, eta=1 day, 17:23:43\n",
      "2018-09-26 02:56:01 Iter 389000 [Val]: loss=46.83, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-389000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-389000\n",
      "2018-09-26 03:09:52 Iter 390000 [Train]: loss=43.62, epe=2.71, lr=0.000025, samples/sec=44.7, sec/step=0.716, eta=1 day, 17:46:58\n",
      "2018-09-26 03:10:06 Iter 390000 [Val]: loss=46.94, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-390000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-390000\n",
      "2018-09-26 03:24:00 Iter 391000 [Train]: loss=43.90, epe=2.73, lr=0.000025, samples/sec=44.4, sec/step=0.720, eta=1 day, 17:48:19\n",
      "2018-09-26 03:24:15 Iter 391000 [Val]: loss=47.50, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-391000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-391000\n",
      "2018-09-26 03:38:19 Iter 392000 [Train]: loss=43.64, epe=2.72, lr=0.000025, samples/sec=43.8, sec/step=0.730, eta=1 day, 18:10:02\n",
      "2018-09-26 03:38:34 Iter 392000 [Val]: loss=48.17, epe=3.11\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.11) doesn't outperform other checkpoints\n",
      "2018-09-26 03:52:30 Iter 393000 [Train]: loss=44.10, epe=2.75, lr=0.000025, samples/sec=44.1, sec/step=0.726, eta=1 day, 17:45:27\n",
      "2018-09-26 03:52:45 Iter 393000 [Val]: loss=49.17, epe=3.19\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.19) doesn't outperform other checkpoints\n",
      "2018-09-26 04:06:39 Iter 394000 [Train]: loss=43.71, epe=2.72, lr=0.000025, samples/sec=44.2, sec/step=0.724, eta=1 day, 17:25:58\n",
      "2018-09-26 04:06:54 Iter 394000 [Val]: loss=46.98, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-394000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-394000\n",
      "2018-09-26 04:21:13 Iter 395000 [Train]: loss=43.63, epe=2.72, lr=0.000025, samples/sec=43.0, sec/step=0.743, eta=1 day, 18:20:11\n",
      "2018-09-26 04:21:28 Iter 395000 [Val]: loss=47.10, epe=3.03\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-395000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-395000\n",
      "2018-09-26 04:35:52 Iter 396000 [Train]: loss=43.93, epe=2.73, lr=0.000025, samples/sec=43.0, sec/step=0.745, eta=1 day, 18:11:52\n",
      "2018-09-26 04:36:07 Iter 396000 [Val]: loss=47.93, epe=3.10\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.10) doesn't outperform other checkpoints\n",
      "2018-09-26 04:50:23 Iter 397000 [Train]: loss=43.79, epe=2.73, lr=0.000025, samples/sec=43.0, sec/step=0.744, eta=1 day, 17:57:53\n",
      "2018-09-26 04:50:37 Iter 397000 [Val]: loss=47.37, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-397000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-397000\n",
      "2018-09-26 05:05:00 Iter 398000 [Train]: loss=43.90, epe=2.74, lr=0.000025, samples/sec=42.9, sec/step=0.746, eta=1 day, 17:53:00\n",
      "2018-09-26 05:05:15 Iter 398000 [Val]: loss=45.66, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-398000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-398000\n",
      "2018-09-26 05:19:49 Iter 399000 [Train]: loss=44.02, epe=2.74, lr=0.000025, samples/sec=42.2, sec/step=0.758, eta=1 day, 18:18:20\n",
      "2018-09-26 05:20:05 Iter 399000 [Val]: loss=48.52, epe=3.16\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.16) doesn't outperform other checkpoints\n",
      "2018-09-26 05:34:33 Iter 400000 [Train]: loss=43.70, epe=2.72, lr=0.000025, samples/sec=42.2, sec/step=0.758, eta=1 day, 18:06:37\n",
      "2018-09-26 05:34:49 Iter 400000 [Val]: loss=46.75, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-400000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-400000\n",
      "2018-09-26 05:49:30 Iter 401000 [Train]: loss=43.43, epe=2.70, lr=0.000012, samples/sec=41.9, sec/step=0.764, eta=1 day, 18:15:13\n",
      "2018-09-26 05:49:46 Iter 401000 [Val]: loss=45.95, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-401000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-401000\n",
      "2018-09-26 06:04:39 Iter 402000 [Train]: loss=43.35, epe=2.69, lr=0.000012, samples/sec=41.3, sec/step=0.775, eta=1 day, 18:37:34\n",
      "2018-09-26 06:04:54 Iter 402000 [Val]: loss=47.40, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-402000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-402000\n",
      "2018-09-26 06:19:48 Iter 403000 [Train]: loss=43.12, epe=2.68, lr=0.000012, samples/sec=41.2, sec/step=0.776, eta=1 day, 18:28:27\n",
      "2018-09-26 06:20:04 Iter 403000 [Val]: loss=47.90, epe=3.10\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.10) doesn't outperform other checkpoints\n",
      "2018-09-26 06:34:54 Iter 404000 [Train]: loss=43.33, epe=2.69, lr=0.000012, samples/sec=41.2, sec/step=0.777, eta=1 day, 18:17:29\n",
      "2018-09-26 06:35:10 Iter 404000 [Val]: loss=45.30, epe=2.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-404000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-404000\n",
      "2018-09-26 06:50:20 Iter 405000 [Train]: loss=43.27, epe=2.68, lr=0.000012, samples/sec=40.6, sec/step=0.788, eta=1 day, 18:40:14\n",
      "2018-09-26 06:50:36 Iter 405000 [Val]: loss=47.66, epe=3.07\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.07) doesn't outperform other checkpoints\n",
      "2018-09-26 07:05:34 Iter 406000 [Train]: loss=42.75, epe=2.65, lr=0.000012, samples/sec=40.8, sec/step=0.784, eta=1 day, 18:15:13\n",
      "2018-09-26 07:05:50 Iter 406000 [Val]: loss=46.05, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-406000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-406000\n",
      "2018-09-26 07:21:00 Iter 407000 [Train]: loss=43.46, epe=2.70, lr=0.000012, samples/sec=40.5, sec/step=0.791, eta=1 day, 18:24:09\n",
      "2018-09-26 07:21:16 Iter 407000 [Val]: loss=47.92, epe=3.08\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.08) doesn't outperform other checkpoints\n",
      "2018-09-26 07:36:20 Iter 408000 [Train]: loss=43.02, epe=2.67, lr=0.000012, samples/sec=40.5, sec/step=0.789, eta=1 day, 18:05:44\n",
      "2018-09-26 07:36:36 Iter 408000 [Val]: loss=45.82, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-408000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-408000\n",
      "2018-09-26 07:51:53 Iter 409000 [Train]: loss=43.40, epe=2.70, lr=0.000012, samples/sec=40.1, sec/step=0.798, eta=1 day, 18:19:44\n",
      "2018-09-26 07:52:10 Iter 409000 [Val]: loss=48.18, epe=3.12\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.12) doesn't outperform other checkpoints\n",
      "2018-09-26 08:07:19 Iter 410000 [Train]: loss=43.19, epe=2.68, lr=0.000012, samples/sec=40.2, sec/step=0.795, eta=1 day, 17:58:45\n",
      "2018-09-26 08:07:36 Iter 410000 [Val]: loss=46.18, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-410000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-410000\n",
      "2018-09-26 08:23:02 Iter 411000 [Train]: loss=43.13, epe=2.68, lr=0.000012, samples/sec=39.7, sec/step=0.806, eta=1 day, 18:19:29\n",
      "2018-09-26 08:23:18 Iter 411000 [Val]: loss=46.39, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-411000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-411000\n",
      "2018-09-26 08:38:47 Iter 412000 [Train]: loss=43.09, epe=2.67, lr=0.000012, samples/sec=39.5, sec/step=0.809, eta=1 day, 18:15:12\n",
      "2018-09-26 08:39:04 Iter 412000 [Val]: loss=45.94, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-412000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-412000\n",
      "2018-09-26 08:54:34 Iter 413000 [Train]: loss=42.88, epe=2.66, lr=0.000012, samples/sec=39.5, sec/step=0.809, eta=1 day, 18:02:28\n",
      "2018-09-26 08:54:51 Iter 413000 [Val]: loss=46.22, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-413000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-413000\n",
      "2018-09-26 09:10:26 Iter 414000 [Train]: loss=43.09, epe=2.67, lr=0.000012, samples/sec=39.3, sec/step=0.815, eta=1 day, 18:05:23\n",
      "2018-09-26 09:10:43 Iter 414000 [Val]: loss=47.38, epe=3.06\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.06) doesn't outperform other checkpoints\n",
      "2018-09-26 09:26:20 Iter 415000 [Train]: loss=42.98, epe=2.66, lr=0.000012, samples/sec=39.3, sec/step=0.814, eta=1 day, 17:49:30\n",
      "2018-09-26 09:26:37 Iter 415000 [Val]: loss=48.20, epe=3.12\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.12) doesn't outperform other checkpoints\n",
      "2018-09-26 09:42:02 Iter 416000 [Train]: loss=43.06, epe=2.67, lr=0.000012, samples/sec=39.6, sec/step=0.809, eta=1 day, 17:20:57\n",
      "2018-09-26 09:42:19 Iter 416000 [Val]: loss=45.11, epe=2.87\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-416000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-416000\n",
      "2018-09-26 09:58:15 Iter 417000 [Train]: loss=42.94, epe=2.66, lr=0.000012, samples/sec=38.4, sec/step=0.834, eta=1 day, 18:24:39\n",
      "2018-09-26 09:58:33 Iter 417000 [Val]: loss=46.11, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-417000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-417000\n",
      "2018-09-26 10:14:27 Iter 418000 [Train]: loss=43.26, epe=2.68, lr=0.000012, samples/sec=38.5, sec/step=0.832, eta=1 day, 18:02:26\n",
      "2018-09-26 10:14:44 Iter 418000 [Val]: loss=48.18, epe=3.13\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.13) doesn't outperform other checkpoints\n",
      "2018-09-26 10:30:30 Iter 419000 [Train]: loss=42.91, epe=2.66, lr=0.000012, samples/sec=38.6, sec/step=0.829, eta=1 day, 17:42:09\n",
      "2018-09-26 10:30:48 Iter 419000 [Val]: loss=45.63, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-419000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-419000\n",
      "2018-09-26 10:46:46 Iter 420000 [Train]: loss=43.21, epe=2.68, lr=0.000012, samples/sec=38.3, sec/step=0.835, eta=1 day, 17:45:14\n",
      "2018-09-26 10:47:03 Iter 420000 [Val]: loss=46.69, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-420000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-420000\n",
      "2018-09-26 11:03:06 Iter 421000 [Train]: loss=42.85, epe=2.66, lr=0.000012, samples/sec=38.1, sec/step=0.841, eta=1 day, 17:48:34\n",
      "2018-09-26 11:03:24 Iter 421000 [Val]: loss=47.53, epe=3.05\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.05) doesn't outperform other checkpoints\n",
      "2018-09-26 11:19:25 Iter 422000 [Train]: loss=42.77, epe=2.65, lr=0.000012, samples/sec=37.9, sec/step=0.843, eta=1 day, 17:42:07\n",
      "2018-09-26 11:19:43 Iter 422000 [Val]: loss=48.47, epe=3.12\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.12) doesn't outperform other checkpoints\n",
      "2018-09-26 11:35:34 Iter 423000 [Train]: loss=43.39, epe=2.69, lr=0.000012, samples/sec=38.4, sec/step=0.834, eta=1 day, 16:59:51\n",
      "2018-09-26 11:35:52 Iter 423000 [Val]: loss=45.36, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-423000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-423000\n",
      "2018-09-26 11:52:04 Iter 424000 [Train]: loss=42.94, epe=2.66, lr=0.000012, samples/sec=37.7, sec/step=0.849, eta=1 day, 17:30:48\n",
      "2018-09-26 11:52:22 Iter 424000 [Val]: loss=46.65, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-424000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-424000\n",
      "2018-09-26 12:08:51 Iter 425000 [Train]: loss=42.79, epe=2.65, lr=0.000012, samples/sec=36.9, sec/step=0.867, eta=1 day, 18:08:14\n",
      "2018-09-26 12:09:10 Iter 425000 [Val]: loss=46.46, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-425000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-425000\n",
      "2018-09-26 12:25:31 Iter 426000 [Train]: loss=43.17, epe=2.68, lr=0.000012, samples/sec=37.3, sec/step=0.857, eta=1 day, 17:26:25\n",
      "2018-09-26 12:25:48 Iter 426000 [Val]: loss=46.88, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-426000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-426000\n",
      "2018-09-26 12:42:08 Iter 427000 [Train]: loss=43.23, epe=2.68, lr=0.000012, samples/sec=37.4, sec/step=0.857, eta=1 day, 17:09:41\n",
      "2018-09-26 12:42:26 Iter 427000 [Val]: loss=45.84, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-427000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-427000\n",
      "2018-09-26 12:58:55 Iter 428000 [Train]: loss=42.93, epe=2.66, lr=0.000012, samples/sec=37.3, sec/step=0.858, eta=1 day, 17:00:44\n",
      "2018-09-26 12:59:13 Iter 428000 [Val]: loss=46.94, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-428000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-428000\n",
      "2018-09-26 13:15:36 Iter 429000 [Train]: loss=42.89, epe=nan, lr=0.000012, samples/sec=37.2, sec/step=0.860, eta=1 day, 16:50:18\n",
      "2018-09-26 13:15:54 Iter 429000 [Val]: loss=46.43, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-429000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-429000\n",
      "2018-09-26 13:32:19 Iter 430000 [Train]: loss=43.13, epe=2.68, lr=0.000012, samples/sec=37.1, sec/step=0.862, eta=1 day, 16:42:16\n",
      "2018-09-26 13:32:37 Iter 430000 [Val]: loss=46.83, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-430000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-430000\n",
      "2018-09-26 13:49:04 Iter 431000 [Train]: loss=42.61, epe=2.64, lr=0.000012, samples/sec=37.1, sec/step=0.864, eta=1 day, 16:32:39\n",
      "2018-09-26 13:49:22 Iter 431000 [Val]: loss=48.02, epe=3.10\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.10) doesn't outperform other checkpoints\n",
      "2018-09-26 14:05:38 Iter 432000 [Train]: loss=43.08, epe=2.67, lr=0.000012, samples/sec=37.3, sec/step=0.859, eta=1 day, 16:03:49\n",
      "2018-09-26 14:05:56 Iter 432000 [Val]: loss=48.04, epe=3.11\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.11) doesn't outperform other checkpoints\n",
      "2018-09-26 14:22:08 Iter 433000 [Train]: loss=42.84, epe=2.65, lr=0.000012, samples/sec=37.5, sec/step=0.854, eta=1 day, 15:38:11\n",
      "2018-09-26 14:22:26 Iter 433000 [Val]: loss=45.19, epe=2.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-433000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-433000\n",
      "2018-09-26 14:39:03 Iter 434000 [Train]: loss=42.82, epe=2.65, lr=0.000012, samples/sec=36.6, sec/step=0.874, eta=1 day, 16:17:42\n",
      "2018-09-26 14:39:22 Iter 434000 [Val]: loss=47.41, epe=3.05\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.05) doesn't outperform other checkpoints\n",
      "2018-09-26 14:55:49 Iter 435000 [Train]: loss=43.09, epe=2.67, lr=0.000012, samples/sec=36.9, sec/step=0.867, eta=1 day, 15:45:30\n",
      "2018-09-26 14:56:07 Iter 435000 [Val]: loss=46.67, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-435000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-435000\n",
      "2018-09-26 15:12:35 Iter 436000 [Train]: loss=43.01, epe=2.67, lr=0.000012, samples/sec=37.0, sec/step=0.865, eta=1 day, 15:23:07\n",
      "2018-09-26 15:12:54 Iter 436000 [Val]: loss=47.70, epe=3.08\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.08) doesn't outperform other checkpoints\n",
      "2018-09-26 15:29:12 Iter 437000 [Train]: loss=42.72, epe=2.65, lr=0.000012, samples/sec=37.2, sec/step=0.861, eta=1 day, 14:58:12\n",
      "2018-09-26 15:29:30 Iter 437000 [Val]: loss=44.98, epe=2.88\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-437000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-437000\n",
      "2018-09-26 15:45:54 Iter 438000 [Train]: loss=42.88, epe=2.65, lr=0.000012, samples/sec=37.2, sec/step=0.861, eta=1 day, 14:43:38\n",
      "2018-09-26 15:46:12 Iter 438000 [Val]: loss=46.14, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-438000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-438000\n",
      "2018-09-26 16:03:01 Iter 439000 [Train]: loss=42.88, epe=2.65, lr=0.000012, samples/sec=36.1, sec/step=0.886, eta=1 day, 15:36:36\n",
      "2018-09-26 16:03:20 Iter 439000 [Val]: loss=47.52, epe=3.08\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.08) doesn't outperform other checkpoints\n",
      "2018-09-26 16:20:10 Iter 440000 [Train]: loss=42.96, epe=2.66, lr=0.000012, samples/sec=36.0, sec/step=0.890, eta=1 day, 15:33:28\n",
      "2018-09-26 16:20:29 Iter 440000 [Val]: loss=46.70, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-440000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-440000\n",
      "2018-09-26 16:37:27 Iter 441000 [Train]: loss=42.55, epe=2.63, lr=0.000012, samples/sec=35.8, sec/step=0.893, eta=1 day, 15:27:14\n",
      "2018-09-26 16:37:46 Iter 441000 [Val]: loss=46.05, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-441000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-441000\n",
      "2018-09-26 16:54:23 Iter 442000 [Train]: loss=43.03, epe=2.67, lr=0.000012, samples/sec=36.6, sec/step=0.874, eta=1 day, 14:21:04\n",
      "2018-09-26 16:54:42 Iter 442000 [Val]: loss=46.91, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "2018-09-26 17:11:11 Iter 443000 [Train]: loss=42.90, epe=2.66, lr=0.000012, samples/sec=36.7, sec/step=0.871, eta=1 day, 13:58:31\n",
      "2018-09-26 17:11:30 Iter 443000 [Val]: loss=46.15, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-443000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-443000\n",
      "2018-09-26 17:28:30 Iter 444000 [Train]: loss=42.50, epe=2.63, lr=0.000012, samples/sec=36.1, sec/step=0.886, eta=1 day, 14:22:44\n",
      "2018-09-26 17:28:49 Iter 444000 [Val]: loss=48.43, epe=3.14\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.14) doesn't outperform other checkpoints\n",
      "2018-09-26 17:45:12 Iter 445000 [Train]: loss=42.85, epe=2.66, lr=0.000012, samples/sec=37.0, sec/step=0.866, eta=1 day, 13:16:04\n",
      "2018-09-26 17:45:30 Iter 445000 [Val]: loss=46.17, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-445000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-445000\n",
      "2018-09-26 18:02:06 Iter 446000 [Train]: loss=42.95, epe=2.65, lr=0.000012, samples/sec=36.7, sec/step=0.873, eta=1 day, 13:20:38\n",
      "2018-09-26 18:02:25 Iter 446000 [Val]: loss=46.45, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-446000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-446000\n",
      "2018-09-26 18:19:03 Iter 447000 [Train]: loss=42.67, epe=2.64, lr=0.000012, samples/sec=36.6, sec/step=0.875, eta=1 day, 13:10:27\n",
      "2018-09-26 18:19:21 Iter 447000 [Val]: loss=46.08, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-447000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-447000\n",
      "2018-09-26 18:35:59 Iter 448000 [Train]: loss=42.71, epe=2.64, lr=0.000012, samples/sec=36.6, sec/step=0.874, eta=1 day, 12:55:01\n",
      "2018-09-26 18:36:18 Iter 448000 [Val]: loss=46.85, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-448000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-448000\n",
      "2018-09-26 18:52:55 Iter 449000 [Train]: loss=42.57, epe=2.64, lr=0.000012, samples/sec=36.7, sec/step=0.873, eta=1 day, 12:36:49\n",
      "2018-09-26 18:53:13 Iter 449000 [Val]: loss=46.93, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "2018-09-26 19:09:37 Iter 450000 [Train]: loss=43.02, epe=2.67, lr=0.000012, samples/sec=37.0, sec/step=0.865, eta=1 day, 12:01:31\n",
      "2018-09-26 19:09:55 Iter 450000 [Val]: loss=46.60, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-450000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-450000\n",
      "2018-09-26 19:26:41 Iter 451000 [Train]: loss=42.98, epe=2.66, lr=0.000012, samples/sec=36.3, sec/step=0.881, eta=1 day, 12:27:52\n",
      "2018-09-26 19:26:59 Iter 451000 [Val]: loss=47.16, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "2018-09-26 19:43:32 Iter 452000 [Train]: loss=42.82, epe=2.65, lr=0.000012, samples/sec=36.6, sec/step=0.874, eta=1 day, 11:55:20\n",
      "2018-09-26 19:43:51 Iter 452000 [Val]: loss=46.58, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-452000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-452000\n",
      "2018-09-26 20:00:35 Iter 453000 [Train]: loss=42.93, epe=nan, lr=0.000012, samples/sec=36.4, sec/step=0.880, eta=1 day, 11:56:06\n",
      "2018-09-26 20:00:54 Iter 453000 [Val]: loss=47.22, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "2018-09-26 20:17:29 Iter 454000 [Train]: loss=42.39, epe=2.61, lr=0.000012, samples/sec=36.5, sec/step=0.876, eta=1 day, 11:31:06\n",
      "2018-09-26 20:17:48 Iter 454000 [Val]: loss=45.77, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-454000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-454000\n",
      "2018-09-26 20:34:33 Iter 455000 [Train]: loss=43.14, epe=2.67, lr=0.000012, samples/sec=36.3, sec/step=0.881, eta=1 day, 11:28:01\n",
      "2018-09-26 20:34:52 Iter 455000 [Val]: loss=47.29, epe=3.05\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.05) doesn't outperform other checkpoints\n",
      "2018-09-26 20:51:24 Iter 456000 [Train]: loss=42.50, epe=2.63, lr=0.000012, samples/sec=36.7, sec/step=0.873, eta=1 day, 10:54:58\n",
      "2018-09-26 20:51:43 Iter 456000 [Val]: loss=45.61, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-456000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-456000\n",
      "2018-09-26 21:08:30 Iter 457000 [Train]: loss=42.92, epe=2.66, lr=0.000012, samples/sec=36.3, sec/step=0.882, eta=1 day, 11:03:16\n",
      "2018-09-26 21:08:48 Iter 457000 [Val]: loss=47.57, epe=3.07\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.07) doesn't outperform other checkpoints\n",
      "2018-09-26 21:25:27 Iter 458000 [Train]: loss=42.59, epe=2.63, lr=0.000012, samples/sec=36.4, sec/step=0.879, eta=1 day, 10:39:37\n",
      "2018-09-26 21:25:45 Iter 458000 [Val]: loss=45.38, epe=2.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-458000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-458000\n",
      "2018-09-26 21:42:35 Iter 459000 [Train]: loss=42.81, epe=2.66, lr=0.000012, samples/sec=36.2, sec/step=0.885, eta=1 day, 10:38:56\n",
      "2018-09-26 21:42:54 Iter 459000 [Val]: loss=47.04, epe=3.04\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.04) doesn't outperform other checkpoints\n",
      "2018-09-26 21:59:31 Iter 460000 [Train]: loss=42.79, epe=2.65, lr=0.000012, samples/sec=36.5, sec/step=0.878, eta=1 day, 10:07:50\n",
      "2018-09-26 21:59:50 Iter 460000 [Val]: loss=44.97, epe=2.87\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-460000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-460000\n",
      "2018-09-26 22:16:44 Iter 461000 [Train]: loss=42.50, epe=2.63, lr=0.000012, samples/sec=36.0, sec/step=0.889, eta=1 day, 10:20:40\n",
      "2018-09-26 22:17:03 Iter 461000 [Val]: loss=45.55, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-461000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-461000\n",
      "2018-09-26 22:33:52 Iter 462000 [Train]: loss=42.95, epe=2.66, lr=0.000012, samples/sec=36.2, sec/step=0.884, eta=1 day, 9:53:50\n",
      "2018-09-26 22:34:11 Iter 462000 [Val]: loss=47.71, epe=nan\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-462000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-462000\n",
      "2018-09-26 22:51:11 Iter 463000 [Train]: loss=42.66, epe=2.64, lr=0.000012, samples/sec=36.2, sec/step=0.883, eta=1 day, 9:36:07\n",
      "2018-09-26 22:51:30 Iter 463000 [Val]: loss=44.54, epe=2.85\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-463000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-463000\n",
      "2018-09-26 23:08:19 Iter 464000 [Train]: loss=42.77, epe=2.65, lr=0.000012, samples/sec=36.2, sec/step=0.883, eta=1 day, 9:22:10\n",
      "2018-09-26 23:08:38 Iter 464000 [Val]: loss=46.50, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-464000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-464000\n",
      "2018-09-26 23:25:29 Iter 465000 [Train]: loss=42.55, epe=2.63, lr=0.000012, samples/sec=36.1, sec/step=0.887, eta=1 day, 9:14:47\n",
      "2018-09-26 23:25:48 Iter 465000 [Val]: loss=46.88, epe=3.02\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.02) doesn't outperform other checkpoints\n",
      "2018-09-26 23:42:29 Iter 466000 [Train]: loss=42.73, epe=2.64, lr=0.000012, samples/sec=36.3, sec/step=0.881, eta=1 day, 8:46:32\n",
      "2018-09-26 23:42:48 Iter 466000 [Val]: loss=46.93, epe=3.02\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.02) doesn't outperform other checkpoints\n",
      "2018-09-26 23:59:27 Iter 467000 [Train]: loss=42.50, epe=2.63, lr=0.000012, samples/sec=36.3, sec/step=0.881, eta=1 day, 8:32:03\n",
      "2018-09-26 23:59:47 Iter 467000 [Val]: loss=46.02, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-467000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-467000\n",
      "\n",
      "Resume training from step 467001...\n",
      "2018-09-27 01:17:44 Iter 468000 [Train]: loss=42.53, epe=2.63, lr=0.000012, samples/sec=45.5, sec/step=0.703, eta=1 day, 1:47:02\n",
      "2018-09-27 01:18:04 Iter 468000 [Val]: loss=46.05, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-468000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-468000\n",
      "2018-09-27 01:30:31 Iter 469000 [Train]: loss=42.74, epe=2.65, lr=0.000012, samples/sec=50.5, sec/step=0.634, eta=23:04:29\n",
      "2018-09-27 01:30:44 Iter 469000 [Val]: loss=46.53, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-469000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-469000\n",
      "2018-09-27 01:43:16 Iter 470000 [Train]: loss=43.12, epe=2.67, lr=0.000012, samples/sec=49.9, sec/step=0.641, eta=23:09:32\n",
      "2018-09-27 01:43:29 Iter 470000 [Val]: loss=46.33, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-470000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-470000\n",
      "2018-09-27 01:56:09 Iter 471000 [Train]: loss=42.43, epe=2.63, lr=0.000012, samples/sec=49.3, sec/step=0.650, eta=23:16:41\n",
      "2018-09-27 01:56:22 Iter 471000 [Val]: loss=46.88, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-471000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-471000\n",
      "2018-09-27 02:09:01 Iter 472000 [Train]: loss=42.53, epe=2.63, lr=0.000012, samples/sec=49.3, sec/step=0.649, eta=23:04:24\n",
      "2018-09-27 02:09:13 Iter 472000 [Val]: loss=46.36, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-472000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-472000\n",
      "2018-09-27 02:21:53 Iter 473000 [Train]: loss=42.83, epe=2.65, lr=0.000012, samples/sec=49.2, sec/step=0.651, eta=22:58:06\n",
      "2018-09-27 02:22:06 Iter 473000 [Val]: loss=46.57, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-473000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-473000\n",
      "2018-09-27 02:34:52 Iter 474000 [Train]: loss=42.43, epe=2.62, lr=0.000012, samples/sec=48.7, sec/step=0.657, eta=22:59:16\n",
      "2018-09-27 02:35:05 Iter 474000 [Val]: loss=45.98, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-474000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-474000\n",
      "2018-09-27 02:47:50 Iter 475000 [Train]: loss=42.80, epe=2.66, lr=0.000012, samples/sec=48.8, sec/step=0.655, eta=22:45:35\n",
      "2018-09-27 02:48:03 Iter 475000 [Val]: loss=47.63, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-475000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-475000\n",
      "2018-09-27 03:00:59 Iter 476000 [Train]: loss=42.46, epe=2.62, lr=0.000012, samples/sec=48.0, sec/step=0.666, eta=22:56:54\n",
      "2018-09-27 03:01:12 Iter 476000 [Val]: loss=45.57, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-476000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-476000\n",
      "2018-09-27 03:14:02 Iter 477000 [Train]: loss=42.57, epe=2.64, lr=0.000012, samples/sec=48.4, sec/step=0.661, eta=22:34:40\n",
      "2018-09-27 03:14:15 Iter 477000 [Val]: loss=47.27, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-477000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-477000\n",
      "2018-09-27 03:27:15 Iter 478000 [Train]: loss=42.73, epe=2.65, lr=0.000012, samples/sec=47.8, sec/step=0.670, eta=22:42:27\n",
      "2018-09-27 03:27:28 Iter 478000 [Val]: loss=46.01, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-478000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-478000\n",
      "2018-09-27 03:40:34 Iter 479000 [Train]: loss=42.53, epe=2.63, lr=0.000012, samples/sec=47.4, sec/step=0.674, eta=22:40:08\n",
      "2018-09-27 03:40:47 Iter 479000 [Val]: loss=45.43, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-479000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-479000\n",
      "2018-09-27 03:53:59 Iter 480000 [Train]: loss=42.64, epe=2.64, lr=0.000012, samples/sec=47.0, sec/step=0.681, eta=22:41:40\n",
      "2018-09-27 03:54:13 Iter 480000 [Val]: loss=47.03, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-480000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-480000\n",
      "2018-09-27 04:07:28 Iter 481000 [Train]: loss=42.51, epe=2.63, lr=0.000012, samples/sec=46.7, sec/step=0.685, eta=22:37:43\n",
      "2018-09-27 04:07:42 Iter 481000 [Val]: loss=46.12, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-481000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-481000\n",
      "2018-09-27 04:21:05 Iter 482000 [Train]: loss=42.79, epe=2.65, lr=0.000012, samples/sec=46.3, sec/step=0.691, eta=22:39:43\n",
      "2018-09-27 04:21:18 Iter 482000 [Val]: loss=46.46, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-482000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-482000\n",
      "2018-09-27 04:34:45 Iter 483000 [Train]: loss=42.41, epe=2.62, lr=0.000012, samples/sec=46.2, sec/step=0.693, eta=22:31:00\n",
      "2018-09-27 04:34:59 Iter 483000 [Val]: loss=45.62, epe=2.93\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-483000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-483000\n",
      "2018-09-27 04:48:29 Iter 484000 [Train]: loss=42.56, epe=2.63, lr=0.000012, samples/sec=45.8, sec/step=0.699, eta=22:31:26\n",
      "2018-09-27 04:48:43 Iter 484000 [Val]: loss=47.23, epe=3.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-484000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-484000\n",
      "2018-09-27 05:02:25 Iter 485000 [Train]: loss=42.62, epe=2.63, lr=0.000012, samples/sec=45.1, sec/step=0.709, eta=22:39:33\n",
      "2018-09-27 05:02:39 Iter 485000 [Val]: loss=47.73, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-485000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-485000\n",
      "2018-09-27 05:16:28 Iter 486000 [Train]: loss=42.27, epe=2.62, lr=0.000012, samples/sec=44.7, sec/step=0.716, eta=22:40:31\n",
      "2018-09-27 05:16:42 Iter 486000 [Val]: loss=44.41, epe=2.82\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-486000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-486000\n",
      "2018-09-27 05:30:38 Iter 487000 [Train]: loss=42.51, epe=2.63, lr=0.000012, samples/sec=44.3, sec/step=0.722, eta=22:39:46\n",
      "2018-09-27 05:30:52 Iter 487000 [Val]: loss=48.30, epe=3.13\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-487000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-487000\n",
      "2018-09-27 05:44:56 Iter 488000 [Train]: loss=42.92, epe=2.66, lr=0.000012, samples/sec=43.8, sec/step=0.730, eta=22:43:13\n",
      "2018-09-27 05:45:11 Iter 488000 [Val]: loss=44.38, epe=2.84\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-488000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-488000\n",
      "2018-09-27 05:59:25 Iter 489000 [Train]: loss=42.36, epe=2.62, lr=0.000012, samples/sec=43.4, sec/step=0.737, eta=22:43:05\n",
      "2018-09-27 05:59:40 Iter 489000 [Val]: loss=47.18, epe=3.03\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-489000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-489000\n",
      "2018-09-27 06:14:00 Iter 490000 [Train]: loss=42.33, epe=2.62, lr=0.000012, samples/sec=43.0, sec/step=0.745, eta=22:45:17\n",
      "2018-09-27 06:14:15 Iter 490000 [Val]: loss=46.40, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-490000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-490000\n",
      "2018-09-27 06:28:38 Iter 491000 [Train]: loss=43.03, epe=2.66, lr=0.000012, samples/sec=42.8, sec/step=0.748, eta=22:38:20\n",
      "2018-09-27 06:28:53 Iter 491000 [Val]: loss=47.62, epe=3.07\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-491000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-491000\n",
      "2018-09-27 06:43:27 Iter 492000 [Train]: loss=42.35, epe=2.62, lr=0.000012, samples/sec=42.2, sec/step=0.759, eta=22:45:18\n",
      "2018-09-27 06:43:43 Iter 492000 [Val]: loss=46.47, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-492000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-492000\n",
      "2018-09-27 06:58:16 Iter 493000 [Train]: loss=42.48, epe=2.62, lr=0.000012, samples/sec=42.3, sec/step=0.757, eta=22:30:03\n",
      "2018-09-27 06:58:31 Iter 493000 [Val]: loss=43.08, epe=2.73\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-493000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-493000\n",
      "2018-09-27 07:13:10 Iter 494000 [Train]: loss=42.38, epe=2.62, lr=0.000012, samples/sec=42.0, sec/step=0.762, eta=22:26:53\n",
      "2018-09-27 07:13:25 Iter 494000 [Val]: loss=50.41, epe=3.31\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.31) doesn't outperform other checkpoints\n",
      "2018-09-27 07:27:56 Iter 495000 [Train]: loss=42.76, epe=nan, lr=0.000012, samples/sec=42.2, sec/step=0.759, eta=22:08:33\n",
      "2018-09-27 07:28:11 Iter 495000 [Val]: loss=47.06, epe=3.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-495000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-495000\n",
      "2018-09-27 07:42:54 Iter 496000 [Train]: loss=42.42, epe=2.62, lr=0.000012, samples/sec=42.0, sec/step=0.762, eta=22:01:33\n",
      "2018-09-27 07:43:10 Iter 496000 [Val]: loss=45.37, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-496000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-496000\n",
      "2018-09-27 07:57:58 Iter 497000 [Train]: loss=42.38, epe=2.62, lr=0.000012, samples/sec=41.4, sec/step=0.772, eta=22:05:37\n",
      "2018-09-27 07:58:14 Iter 497000 [Val]: loss=47.15, epe=3.01\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-497000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-497000\n",
      "2018-09-27 08:13:07 Iter 498000 [Train]: loss=42.59, epe=2.63, lr=0.000012, samples/sec=41.3, sec/step=0.775, eta=21:57:48\n",
      "2018-09-27 08:13:23 Iter 498000 [Val]: loss=44.82, epe=2.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-498000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-498000\n",
      "2018-09-27 08:28:28 Iter 499000 [Train]: loss=42.81, epe=2.65, lr=0.000012, samples/sec=40.7, sec/step=0.786, eta=22:02:51\n",
      "2018-09-27 08:28:44 Iter 499000 [Val]: loss=47.49, epe=3.06\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.06) doesn't outperform other checkpoints\n",
      "2018-09-27 08:43:41 Iter 500000 [Train]: loss=42.30, epe=2.61, lr=0.000012, samples/sec=40.8, sec/step=0.784, eta=21:46:32\n",
      "2018-09-27 08:43:58 Iter 500000 [Val]: loss=47.10, epe=3.05\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.05) doesn't outperform other checkpoints\n",
      "2018-09-27 08:58:51 Iter 501000 [Train]: loss=42.17, epe=2.60, lr=0.000006, samples/sec=41.0, sec/step=0.781, eta=21:29:17\n",
      "2018-09-27 08:59:07 Iter 501000 [Val]: loss=46.71, epe=3.00\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-501000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-501000\n",
      "2018-09-27 09:14:20 Iter 502000 [Train]: loss=42.40, epe=2.62, lr=0.000006, samples/sec=40.3, sec/step=0.794, eta=21:37:00\n",
      "2018-09-27 09:14:37 Iter 502000 [Val]: loss=46.07, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-502000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-502000\n",
      "2018-09-27 09:29:58 Iter 503000 [Train]: loss=42.09, epe=2.60, lr=0.000006, samples/sec=39.9, sec/step=0.802, eta=21:36:39\n",
      "2018-09-27 09:30:14 Iter 503000 [Val]: loss=46.17, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-503000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-503000\n",
      "2018-09-27 09:45:57 Iter 504000 [Train]: loss=42.53, epe=2.63, lr=0.000006, samples/sec=39.1, sec/step=0.817, eta=21:47:48\n",
      "2018-09-27 09:46:13 Iter 504000 [Val]: loss=45.83, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-504000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-504000\n",
      "2018-09-27 10:01:54 Iter 505000 [Train]: loss=42.25, epe=2.61, lr=0.000006, samples/sec=39.0, sec/step=0.821, eta=21:39:08\n",
      "2018-09-27 10:02:11 Iter 505000 [Val]: loss=46.65, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "2018-09-27 10:17:38 Iter 506000 [Train]: loss=41.70, epe=2.57, lr=0.000006, samples/sec=39.2, sec/step=0.817, eta=21:20:13\n",
      "2018-09-27 10:17:55 Iter 506000 [Val]: loss=43.08, epe=2.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-506000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-506000\n",
      "2018-09-27 10:33:35 Iter 507000 [Train]: loss=42.54, epe=2.63, lr=0.000006, samples/sec=39.0, sec/step=0.821, eta=21:13:11\n",
      "2018-09-27 10:33:52 Iter 507000 [Val]: loss=49.17, epe=3.19\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.19) doesn't outperform other checkpoints\n",
      "2018-09-27 10:49:22 Iter 508000 [Train]: loss=42.17, epe=2.60, lr=0.000006, samples/sec=39.3, sec/step=0.815, eta=20:49:49\n",
      "2018-09-27 10:49:39 Iter 508000 [Val]: loss=46.30, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-508000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-508000\n",
      "2018-09-27 11:05:20 Iter 509000 [Train]: loss=42.36, epe=2.61, lr=0.000006, samples/sec=39.0, sec/step=0.821, eta=20:45:15\n",
      "2018-09-27 11:05:37 Iter 509000 [Val]: loss=45.79, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-509000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-509000\n",
      "2018-09-27 11:21:30 Iter 510000 [Train]: loss=41.82, epe=2.58, lr=0.000006, samples/sec=38.4, sec/step=0.833, eta=20:49:06\n",
      "2018-09-27 11:21:48 Iter 510000 [Val]: loss=47.03, epe=3.02\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.02) doesn't outperform other checkpoints\n",
      "2018-09-27 11:37:35 Iter 511000 [Train]: loss=42.32, epe=2.61, lr=0.000006, samples/sec=38.5, sec/step=0.831, eta=20:32:49\n",
      "2018-09-27 11:37:52 Iter 511000 [Val]: loss=46.57, epe=3.00\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.00) doesn't outperform other checkpoints\n",
      "2018-09-27 11:53:43 Iter 512000 [Train]: loss=42.02, epe=2.59, lr=0.000006, samples/sec=38.3, sec/step=0.835, eta=20:25:11\n",
      "2018-09-27 11:54:00 Iter 512000 [Val]: loss=44.31, epe=2.81\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-512000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-512000\n",
      "2018-09-27 12:10:12 Iter 513000 [Train]: loss=42.13, epe=2.60, lr=0.000006, samples/sec=37.6, sec/step=0.850, eta=20:33:06\n",
      "2018-09-27 12:10:30 Iter 513000 [Val]: loss=46.30, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-513000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-513000\n",
      "2018-09-27 12:26:45 Iter 514000 [Train]: loss=42.17, epe=nan, lr=0.000006, samples/sec=37.8, sec/step=0.847, eta=20:13:37\n",
      "2018-09-27 12:27:03 Iter 514000 [Val]: loss=47.14, epe=3.04\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.04) doesn't outperform other checkpoints\n",
      "2018-09-27 12:43:00 Iter 515000 [Train]: loss=42.04, epe=2.59, lr=0.000006, samples/sec=38.1, sec/step=0.840, eta=19:50:40\n",
      "2018-09-27 12:43:18 Iter 515000 [Val]: loss=45.64, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-515000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-515000\n",
      "2018-09-27 12:59:20 Iter 516000 [Train]: loss=42.07, epe=2.60, lr=0.000006, samples/sec=38.1, sec/step=0.840, eta=19:36:19\n",
      "2018-09-27 12:59:38 Iter 516000 [Val]: loss=48.88, epe=nan\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-516000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-516000\n",
      "2018-09-27 13:15:48 Iter 517000 [Train]: loss=42.29, epe=2.61, lr=0.000006, samples/sec=37.7, sec/step=0.848, eta=19:33:10\n",
      "2018-09-27 13:16:06 Iter 517000 [Val]: loss=42.38, epe=2.70\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-517000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-517000\n",
      "2018-09-27 13:32:18 Iter 518000 [Train]: loss=41.79, epe=2.57, lr=0.000006, samples/sec=37.6, sec/step=0.851, eta=19:22:37\n",
      "2018-09-27 13:32:36 Iter 518000 [Val]: loss=48.69, epe=3.13\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.13) doesn't outperform other checkpoints\n",
      "2018-09-27 13:48:39 Iter 519000 [Train]: loss=42.27, epe=2.61, lr=0.000006, samples/sec=37.8, sec/step=0.846, eta=19:02:45\n",
      "2018-09-27 13:48:58 Iter 519000 [Val]: loss=45.27, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-519000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-519000\n",
      "2018-09-27 14:05:20 Iter 520000 [Train]: loss=41.90, epe=nan, lr=0.000006, samples/sec=37.2, sec/step=0.861, eta=19:07:58\n",
      "2018-09-27 14:05:38 Iter 520000 [Val]: loss=46.20, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-520000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-520000\n",
      "2018-09-27 14:22:01 Iter 521000 [Train]: loss=41.90, epe=2.58, lr=0.000006, samples/sec=37.2, sec/step=0.860, eta=18:52:33\n",
      "2018-09-27 14:22:19 Iter 521000 [Val]: loss=47.23, epe=3.06\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.06) doesn't outperform other checkpoints\n",
      "2018-09-27 14:38:36 Iter 522000 [Train]: loss=42.31, epe=2.61, lr=0.000006, samples/sec=37.2, sec/step=0.861, eta=18:38:58\n",
      "2018-09-27 14:38:55 Iter 522000 [Val]: loss=45.82, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-522000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-522000\n",
      "2018-09-27 14:55:38 Iter 523000 [Train]: loss=42.13, epe=2.60, lr=0.000006, samples/sec=36.4, sec/step=0.880, eta=18:49:03\n",
      "2018-09-27 14:55:57 Iter 523000 [Val]: loss=46.88, epe=3.02\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.02) doesn't outperform other checkpoints\n",
      "2018-09-27 15:12:42 Iter 524000 [Train]: loss=42.22, epe=2.61, lr=0.000006, samples/sec=36.1, sec/step=0.886, eta=18:42:05\n",
      "2018-09-27 15:13:01 Iter 524000 [Val]: loss=47.10, epe=3.05\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.05) doesn't outperform other checkpoints\n",
      "2018-09-27 15:29:38 Iter 525000 [Train]: loss=42.01, epe=2.59, lr=0.000006, samples/sec=36.5, sec/step=0.876, eta=18:15:12\n",
      "2018-09-27 15:29:57 Iter 525000 [Val]: loss=43.95, epe=2.81\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-525000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-525000\n",
      "2018-09-27 15:46:53 Iter 526000 [Train]: loss=42.14, epe=2.60, lr=0.000006, samples/sec=35.9, sec/step=0.891, eta=18:18:48\n",
      "2018-09-27 15:47:12 Iter 526000 [Val]: loss=45.42, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-526000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-526000\n",
      "2018-09-27 16:04:17 Iter 527000 [Train]: loss=41.38, epe=2.54, lr=0.000006, samples/sec=35.9, sec/step=0.891, eta=18:04:30\n",
      "2018-09-27 16:04:36 Iter 527000 [Val]: loss=47.17, epe=3.04\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.04) doesn't outperform other checkpoints\n",
      "2018-09-27 16:21:18 Iter 528000 [Train]: loss=42.52, epe=2.63, lr=0.000006, samples/sec=36.3, sec/step=0.882, eta=17:38:51\n",
      "2018-09-27 16:21:37 Iter 528000 [Val]: loss=48.26, epe=3.10\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.10) doesn't outperform other checkpoints\n",
      "2018-09-27 16:38:14 Iter 529000 [Train]: loss=42.24, epe=2.61, lr=0.000006, samples/sec=36.4, sec/step=0.879, eta=17:20:04\n",
      "2018-09-27 16:38:34 Iter 529000 [Val]: loss=45.45, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-529000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-529000\n",
      "2018-09-27 16:55:35 Iter 530000 [Train]: loss=42.26, epe=2.61, lr=0.000006, samples/sec=35.7, sec/step=0.896, eta=17:25:47\n",
      "2018-09-27 16:55:54 Iter 530000 [Val]: loss=44.40, epe=2.82\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-530000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-530000\n",
      "2018-09-27 17:12:55 Iter 531000 [Train]: loss=41.88, epe=2.58, lr=0.000006, samples/sec=35.7, sec/step=0.896, eta=17:09:56\n",
      "2018-09-27 17:13:14 Iter 531000 [Val]: loss=45.78, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-531000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-531000\n",
      "2018-09-27 17:30:15 Iter 532000 [Train]: loss=41.74, epe=2.57, lr=0.000006, samples/sec=35.7, sec/step=0.896, eta=16:55:16\n",
      "2018-09-27 17:30:35 Iter 532000 [Val]: loss=46.77, epe=3.02\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.02) doesn't outperform other checkpoints\n",
      "2018-09-27 17:47:26 Iter 533000 [Train]: loss=42.20, epe=2.60, lr=0.000006, samples/sec=35.9, sec/step=0.891, eta=16:35:09\n",
      "2018-09-27 17:47:45 Iter 533000 [Val]: loss=46.28, epe=2.99\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.99) doesn't outperform other checkpoints\n",
      "2018-09-27 18:04:34 Iter 534000 [Train]: loss=42.02, epe=2.59, lr=0.000006, samples/sec=36.0, sec/step=0.889, eta=16:18:08\n",
      "2018-09-27 18:04:53 Iter 534000 [Val]: loss=46.71, epe=2.99\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.99) doesn't outperform other checkpoints\n",
      "2018-09-27 18:21:37 Iter 535000 [Train]: loss=42.01, epe=nan, lr=0.000006, samples/sec=36.2, sec/step=0.885, eta=15:58:22\n",
      "2018-09-27 18:21:57 Iter 535000 [Val]: loss=43.53, epe=2.77\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-535000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-535000\n",
      "2018-09-27 18:39:17 Iter 536000 [Train]: loss=42.06, epe=2.60, lr=0.000006, samples/sec=34.9, sec/step=0.916, eta=16:16:57\n",
      "2018-09-27 18:39:36 Iter 536000 [Val]: loss=47.01, epe=3.01\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.01) doesn't outperform other checkpoints\n",
      "2018-09-27 18:56:43 Iter 537000 [Train]: loss=42.29, epe=2.61, lr=0.000006, samples/sec=35.3, sec/step=0.907, eta=15:52:51\n",
      "2018-09-27 18:57:02 Iter 537000 [Val]: loss=47.55, epe=3.08\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.08) doesn't outperform other checkpoints\n",
      "2018-09-27 19:14:05 Iter 538000 [Train]: loss=41.72, epe=2.57, lr=0.000006, samples/sec=35.4, sec/step=0.903, eta=15:33:21\n",
      "2018-09-27 19:14:25 Iter 538000 [Val]: loss=43.98, epe=2.80\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-538000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-538000\n",
      "2018-09-27 19:31:37 Iter 539000 [Train]: loss=42.10, epe=2.60, lr=0.000006, samples/sec=35.2, sec/step=0.909, eta=15:23:46\n",
      "2018-09-27 19:31:56 Iter 539000 [Val]: loss=45.80, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-539000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-539000\n",
      "2018-09-27 19:49:07 Iter 540000 [Train]: loss=42.07, epe=2.59, lr=0.000006, samples/sec=35.3, sec/step=0.905, eta=15:05:21\n",
      "2018-09-27 19:49:26 Iter 540000 [Val]: loss=46.93, epe=3.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-540000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-540000\n",
      "2018-09-27 20:06:26 Iter 541000 [Train]: loss=42.00, epe=2.59, lr=0.000006, samples/sec=35.7, sec/step=0.897, eta=14:41:35\n",
      "2018-09-27 20:06:45 Iter 541000 [Val]: loss=46.01, epe=2.93\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-541000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-541000\n",
      "2018-09-27 20:23:51 Iter 542000 [Train]: loss=41.97, epe=2.59, lr=0.000006, samples/sec=35.5, sec/step=0.901, eta=14:31:26\n",
      "2018-09-27 20:24:11 Iter 542000 [Val]: loss=46.33, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-542000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-542000\n",
      "2018-09-27 20:41:24 Iter 543000 [Train]: loss=42.21, epe=2.60, lr=0.000006, samples/sec=35.6, sec/step=0.899, eta=14:13:55\n",
      "2018-09-27 20:41:44 Iter 543000 [Val]: loss=46.00, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-543000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-543000\n",
      "2018-09-27 20:58:57 Iter 544000 [Train]: loss=42.00, epe=2.59, lr=0.000006, samples/sec=35.2, sec/step=0.908, eta=14:07:56\n",
      "2018-09-27 20:59:16 Iter 544000 [Val]: loss=45.23, epe=2.89\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-544000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-544000\n",
      "2018-09-27 21:16:29 Iter 545000 [Train]: loss=41.83, epe=2.58, lr=0.000006, samples/sec=35.2, sec/step=0.909, eta=13:53:31\n",
      "2018-09-27 21:16:49 Iter 545000 [Val]: loss=47.40, epe=3.06\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-545000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-545000\n",
      "2018-09-27 21:34:02 Iter 546000 [Train]: loss=42.12, epe=nan, lr=0.000006, samples/sec=35.2, sec/step=0.910, eta=13:38:35\n",
      "2018-09-27 21:34:22 Iter 546000 [Val]: loss=44.20, epe=2.82\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-546000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-546000\n",
      "2018-09-27 21:51:51 Iter 547000 [Train]: loss=41.80, epe=2.57, lr=0.000006, samples/sec=34.6, sec/step=0.924, eta=13:36:19\n",
      "2018-09-27 21:52:12 Iter 547000 [Val]: loss=48.10, epe=3.11\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-547000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-547000\n",
      "2018-09-27 22:09:38 Iter 548000 [Train]: loss=42.01, epe=2.59, lr=0.000006, samples/sec=34.8, sec/step=0.920, eta=13:17:34\n",
      "2018-09-27 22:09:58 Iter 548000 [Val]: loss=44.04, epe=2.80\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-548000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-548000\n",
      "2018-09-27 22:27:22 Iter 549000 [Train]: loss=41.80, epe=2.57, lr=0.000006, samples/sec=34.8, sec/step=0.921, eta=13:02:38\n",
      "2018-09-27 22:27:42 Iter 549000 [Val]: loss=46.49, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-549000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-549000\n",
      "2018-09-27 22:45:07 Iter 550000 [Train]: loss=42.14, epe=2.60, lr=0.000006, samples/sec=34.8, sec/step=0.921, eta=12:47:21\n",
      "2018-09-27 22:45:27 Iter 550000 [Val]: loss=45.56, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-550000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-550000\n",
      "2018-09-27 23:02:55 Iter 551000 [Train]: loss=42.17, epe=2.60, lr=0.000006, samples/sec=34.6, sec/step=0.924, eta=12:34:14\n",
      "2018-09-27 23:03:16 Iter 551000 [Val]: loss=46.45, epe=2.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-551000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-551000\n",
      "2018-09-27 23:20:47 Iter 552000 [Train]: loss=41.85, epe=2.58, lr=0.000006, samples/sec=34.5, sec/step=0.926, eta=12:20:59\n",
      "2018-09-27 23:21:08 Iter 552000 [Val]: loss=46.12, epe=2.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-552000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-552000\n",
      "2018-09-27 23:38:39 Iter 553000 [Train]: loss=41.78, epe=2.57, lr=0.000006, samples/sec=34.5, sec/step=0.927, eta=12:06:10\n",
      "2018-09-27 23:39:00 Iter 553000 [Val]: loss=46.14, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-553000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-553000\n",
      "2018-09-27 23:56:31 Iter 554000 [Train]: loss=42.15, epe=2.60, lr=0.000006, samples/sec=34.5, sec/step=0.927, eta=11:50:33\n",
      "2018-09-27 23:56:51 Iter 554000 [Val]: loss=46.32, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-554000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-554000\n",
      "2018-09-28 00:14:27 Iter 555000 [Train]: loss=41.92, epe=2.59, lr=0.000006, samples/sec=34.4, sec/step=0.930, eta=11:37:41\n",
      "2018-09-28 00:14:47 Iter 555000 [Val]: loss=46.29, epe=2.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-555000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-555000\n",
      "2018-09-28 00:32:22 Iter 556000 [Train]: loss=42.07, epe=2.59, lr=0.000006, samples/sec=34.4, sec/step=0.929, eta=11:21:27\n",
      "2018-09-28 00:32:43 Iter 556000 [Val]: loss=45.92, epe=2.94\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-556000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-556000\n",
      "2018-09-28 00:50:17 Iter 557000 [Train]: loss=41.54, epe=2.55, lr=0.000006, samples/sec=34.4, sec/step=0.930, eta=11:06:12\n",
      "2018-09-28 00:50:37 Iter 557000 [Val]: loss=46.64, epe=3.00\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.00) doesn't outperform other checkpoints\n",
      "2018-09-28 01:08:03 Iter 558000 [Train]: loss=42.26, epe=2.61, lr=0.000006, samples/sec=34.6, sec/step=0.925, eta=10:47:36\n",
      "2018-09-28 01:08:23 Iter 558000 [Val]: loss=46.23, epe=2.97\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-558000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-558000\n",
      "2018-09-28 01:26:08 Iter 559000 [Train]: loss=41.95, epe=2.59, lr=0.000006, samples/sec=34.0, sec/step=0.942, eta=10:43:43\n",
      "2018-09-28 01:26:29 Iter 559000 [Val]: loss=45.26, epe=2.89\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-559000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-559000\n",
      "2018-09-28 01:44:25 Iter 560000 [Train]: loss=41.86, epe=2.57, lr=0.000006, samples/sec=34.1, sec/step=0.939, eta=10:26:15\n",
      "2018-09-28 01:44:46 Iter 560000 [Val]: loss=45.81, epe=2.95\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-560000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-560000\n",
      "2018-09-28 02:02:28 Iter 561000 [Train]: loss=42.10, epe=2.59, lr=0.000006, samples/sec=34.1, sec/step=0.938, eta=10:09:34\n",
      "2018-09-28 02:02:49 Iter 561000 [Val]: loss=45.41, epe=2.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-561000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-561000\n",
      "2018-09-28 02:20:32 Iter 562000 [Train]: loss=42.27, epe=2.61, lr=0.000006, samples/sec=34.1, sec/step=0.939, eta=9:54:31\n",
      "2018-09-28 02:20:53 Iter 562000 [Val]: loss=46.40, epe=3.01\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.01) doesn't outperform other checkpoints\n",
      "2018-09-28 02:38:23 Iter 563000 [Train]: loss=41.36, epe=2.54, lr=0.000006, samples/sec=34.4, sec/step=0.930, eta=9:33:41\n",
      "2018-09-28 02:38:44 Iter 563000 [Val]: loss=47.06, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "2018-09-28 02:56:17 Iter 564000 [Train]: loss=42.23, epe=2.60, lr=0.000006, samples/sec=34.3, sec/step=0.933, eta=9:19:45\n",
      "2018-09-28 02:56:38 Iter 564000 [Val]: loss=44.76, epe=2.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-564000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-564000\n",
      "2018-09-28 03:14:28 Iter 565000 [Train]: loss=41.88, epe=2.58, lr=0.000006, samples/sec=33.9, sec/step=0.945, eta=9:11:05\n",
      "2018-09-28 03:14:49 Iter 565000 [Val]: loss=46.75, epe=3.02\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.02) doesn't outperform other checkpoints\n",
      "2018-09-28 03:32:29 Iter 566000 [Train]: loss=41.93, epe=2.59, lr=0.000006, samples/sec=34.1, sec/step=0.939, eta=8:51:54\n",
      "2018-09-28 03:32:50 Iter 566000 [Val]: loss=45.87, epe=2.96\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.96) doesn't outperform other checkpoints\n",
      "2018-09-28 03:50:26 Iter 567000 [Train]: loss=41.86, epe=2.58, lr=0.000006, samples/sec=34.2, sec/step=0.935, eta=8:34:20\n",
      "2018-09-28 03:50:47 Iter 567000 [Val]: loss=47.09, epe=3.04\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.04) doesn't outperform other checkpoints\n",
      "2018-09-28 04:08:18 Iter 568000 [Train]: loss=41.93, epe=2.58, lr=0.000006, samples/sec=34.4, sec/step=0.930, eta=8:16:10\n",
      "2018-09-28 04:08:38 Iter 568000 [Val]: loss=44.75, epe=2.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-568000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-568000\n",
      "2018-09-28 04:26:30 Iter 569000 [Train]: loss=41.95, epe=2.58, lr=0.000006, samples/sec=33.8, sec/step=0.947, eta=8:09:21\n",
      "2018-09-28 04:26:51 Iter 569000 [Val]: loss=46.27, epe=2.97\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.97) doesn't outperform other checkpoints\n",
      "2018-09-28 04:44:40 Iter 570000 [Train]: loss=41.92, epe=2.58, lr=0.000006, samples/sec=33.8, sec/step=0.948, eta=7:53:47\n",
      "2018-09-28 04:45:01 Iter 570000 [Val]: loss=45.54, epe=2.93\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-570000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-570000\n",
      "2018-09-28 05:02:55 Iter 571000 [Train]: loss=41.95, epe=2.59, lr=0.000006, samples/sec=33.8, sec/step=0.948, eta=7:37:58\n",
      "2018-09-28 05:03:16 Iter 571000 [Val]: loss=43.86, epe=2.79\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-571000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-571000\n",
      "2018-09-28 05:21:01 Iter 572000 [Train]: loss=41.96, epe=2.58, lr=0.000006, samples/sec=34.0, sec/step=0.940, eta=7:18:40\n",
      "2018-09-28 05:21:22 Iter 572000 [Val]: loss=48.94, epe=3.18\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.18) doesn't outperform other checkpoints\n",
      "2018-09-28 05:38:59 Iter 573000 [Train]: loss=41.81, epe=2.57, lr=0.000006, samples/sec=34.1, sec/step=0.938, eta=7:02:01\n",
      "2018-09-28 05:39:20 Iter 573000 [Val]: loss=46.43, epe=2.98\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.98) doesn't outperform other checkpoints\n",
      "2018-09-28 05:56:50 Iter 574000 [Train]: loss=41.97, epe=2.59, lr=0.000006, samples/sec=34.4, sec/step=0.929, eta=6:42:46\n",
      "2018-09-28 05:57:11 Iter 574000 [Val]: loss=46.22, epe=2.97\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.97) doesn't outperform other checkpoints\n",
      "2018-09-28 06:14:37 Iter 575000 [Train]: loss=41.57, epe=nan, lr=0.000006, samples/sec=34.5, sec/step=0.928, eta=6:26:28\n",
      "2018-09-28 06:14:57 Iter 575000 [Val]: loss=46.29, epe=2.98\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.98) doesn't outperform other checkpoints\n",
      "2018-09-28 06:32:20 Iter 576000 [Train]: loss=42.13, epe=2.60, lr=0.000006, samples/sec=34.6, sec/step=0.924, eta=6:09:36\n",
      "2018-09-28 06:32:40 Iter 576000 [Val]: loss=45.61, epe=2.93\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-576000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-576000\n",
      "2018-09-28 06:50:35 Iter 577000 [Train]: loss=41.79, epe=2.57, lr=0.000006, samples/sec=33.6, sec/step=0.951, eta=6:04:33\n",
      "2018-09-28 06:50:57 Iter 577000 [Val]: loss=45.29, epe=2.89\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-577000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-577000\n",
      "2018-09-28 07:08:43 Iter 578000 [Train]: loss=42.07, epe=2.59, lr=0.000006, samples/sec=34.0, sec/step=0.942, eta=5:45:29\n",
      "2018-09-28 07:09:04 Iter 578000 [Val]: loss=45.83, epe=2.94\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.94) doesn't outperform other checkpoints\n",
      "2018-09-28 07:26:39 Iter 579000 [Train]: loss=41.66, epe=2.56, lr=0.000006, samples/sec=34.2, sec/step=0.934, eta=5:27:03\n",
      "2018-09-28 07:27:00 Iter 579000 [Val]: loss=46.06, epe=2.96\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.96) doesn't outperform other checkpoints\n",
      "2018-09-28 07:44:29 Iter 580000 [Train]: loss=42.41, epe=2.62, lr=0.000006, samples/sec=34.4, sec/step=0.930, eta=5:09:51\n",
      "2018-09-28 07:44:50 Iter 580000 [Val]: loss=46.48, epe=2.98\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.98) doesn't outperform other checkpoints\n",
      "2018-09-28 08:02:15 Iter 581000 [Train]: loss=41.33, epe=2.54, lr=0.000006, samples/sec=34.6, sec/step=0.924, eta=4:52:45\n",
      "2018-09-28 08:02:36 Iter 581000 [Val]: loss=46.09, epe=2.96\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.96) doesn't outperform other checkpoints\n",
      "2018-09-28 08:19:57 Iter 582000 [Train]: loss=41.97, epe=2.58, lr=0.000006, samples/sec=34.7, sec/step=0.921, eta=4:36:27\n",
      "2018-09-28 08:20:17 Iter 582000 [Val]: loss=45.11, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-582000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-582000\n",
      "2018-09-28 08:38:09 Iter 583000 [Train]: loss=41.93, epe=2.58, lr=0.000006, samples/sec=33.8, sec/step=0.948, eta=4:28:32\n",
      "2018-09-28 08:38:30 Iter 583000 [Val]: loss=46.65, epe=2.98\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.98) doesn't outperform other checkpoints\n",
      "2018-09-28 08:56:50 Iter 584000 [Train]: loss=41.88, epe=2.58, lr=0.000006, samples/sec=34.0, sec/step=0.941, eta=4:10:53\n",
      "2018-09-28 08:57:11 Iter 584000 [Val]: loss=46.30, epe=2.99\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.99) doesn't outperform other checkpoints\n",
      "2018-09-28 09:14:52 Iter 585000 [Train]: loss=41.85, epe=2.58, lr=0.000006, samples/sec=34.0, sec/step=0.941, eta=3:55:09\n",
      "2018-09-28 09:15:13 Iter 585000 [Val]: loss=45.28, epe=2.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-585000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-585000\n",
      "2018-09-28 09:33:11 Iter 586000 [Train]: loss=41.98, epe=2.59, lr=0.000006, samples/sec=33.6, sec/step=0.952, eta=3:42:05\n",
      "2018-09-28 09:33:32 Iter 586000 [Val]: loss=42.90, epe=2.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-586000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-586000\n",
      "2018-09-28 09:51:15 Iter 587000 [Train]: loss=41.73, epe=2.57, lr=0.000006, samples/sec=34.1, sec/step=0.938, eta=3:23:16\n",
      "2018-09-28 09:51:36 Iter 587000 [Val]: loss=49.83, epe=3.22\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.22) doesn't outperform other checkpoints\n",
      "2018-09-28 10:09:07 Iter 588000 [Train]: loss=42.06, epe=2.59, lr=0.000006, samples/sec=34.4, sec/step=0.931, eta=3:06:09\n",
      "2018-09-28 10:09:27 Iter 588000 [Val]: loss=44.60, epe=2.88\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-588000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-588000\n",
      "2018-09-28 10:27:21 Iter 589000 [Train]: loss=41.62, epe=2.57, lr=0.000006, samples/sec=33.7, sec/step=0.949, eta=2:53:57\n",
      "2018-09-28 10:27:42 Iter 589000 [Val]: loss=46.78, epe=3.00\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.00) doesn't outperform other checkpoints\n",
      "2018-09-28 10:45:18 Iter 590000 [Train]: loss=42.02, epe=2.59, lr=0.000006, samples/sec=34.2, sec/step=0.935, eta=2:35:55\n",
      "2018-09-28 10:45:38 Iter 590000 [Val]: loss=42.79, epe=2.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-590000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-590000\n",
      "2018-09-28 11:03:30 Iter 591000 [Train]: loss=41.79, epe=2.57, lr=0.000006, samples/sec=33.8, sec/step=0.946, eta=2:21:52\n",
      "2018-09-28 11:03:51 Iter 591000 [Val]: loss=47.55, epe=3.06\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.06) doesn't outperform other checkpoints\n",
      "2018-09-28 11:21:31 Iter 592000 [Train]: loss=41.67, epe=2.56, lr=0.000006, samples/sec=34.1, sec/step=0.938, eta=2:05:07\n",
      "2018-09-28 11:21:52 Iter 592000 [Val]: loss=45.29, epe=2.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-592000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-592000\n",
      "2018-09-28 11:39:41 Iter 593000 [Train]: loss=41.91, epe=2.58, lr=0.000006, samples/sec=33.9, sec/step=0.943, eta=1:50:02\n",
      "2018-09-28 11:40:02 Iter 593000 [Val]: loss=47.59, epe=3.08\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.08) doesn't outperform other checkpoints\n",
      "2018-09-28 11:57:39 Iter 594000 [Train]: loss=41.83, epe=2.58, lr=0.000006, samples/sec=34.2, sec/step=0.937, eta=1:33:41\n",
      "2018-09-28 11:58:00 Iter 594000 [Val]: loss=43.70, epe=2.78\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-594000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-594000\n",
      "2018-09-28 12:15:56 Iter 595000 [Train]: loss=41.63, epe=2.56, lr=0.000006, samples/sec=33.7, sec/step=0.950, eta=1:19:10\n",
      "2018-09-28 12:16:17 Iter 595000 [Val]: loss=46.02, epe=2.95\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.95) doesn't outperform other checkpoints\n",
      "2018-09-28 12:33:54 Iter 596000 [Train]: loss=41.95, epe=2.59, lr=0.000006, samples/sec=34.2, sec/step=0.936, eta=1:02:22\n",
      "2018-09-28 12:34:15 Iter 596000 [Val]: loss=46.52, epe=3.01\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.01) doesn't outperform other checkpoints\n",
      "2018-09-28 12:51:47 Iter 597000 [Train]: loss=41.71, epe=2.57, lr=0.000006, samples/sec=34.4, sec/step=0.931, eta=0:46:33\n",
      "2018-09-28 12:52:08 Iter 597000 [Val]: loss=46.71, epe=3.00\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.00) doesn't outperform other checkpoints\n",
      "2018-09-28 13:09:39 Iter 598000 [Train]: loss=42.06, epe=2.59, lr=0.000006, samples/sec=34.4, sec/step=0.931, eta=0:31:01\n",
      "2018-09-28 13:10:00 Iter 598000 [Val]: loss=45.47, epe=2.94\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (2.94) doesn't outperform other checkpoints\n",
      "2018-09-28 13:27:25 Iter 599000 [Train]: loss=41.73, epe=2.57, lr=0.000006, samples/sec=34.7, sec/step=0.923, eta=0:15:23\n",
      "2018-09-28 13:27:45 Iter 599000 [Val]: loss=47.16, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "2018-09-28 13:45:05 Iter 600000 [Train]: loss=41.90, epe=2.58, lr=0.000006, samples/sec=34.8, sec/step=0.918, eta=0:00:00\n",
      "2018-09-28 13:45:26 Iter 600000 [Val]: loss=46.94, epe=3.03\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (3.03) doesn't outperform other checkpoints\n",
      "... done training.\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "nn.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note that the log above wasn't generated over a single run. Whenever we would hit numerical instability (i.e., NaNs) we had to restart the training process. This log is a concatenation of multiple runs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the training curves for the run above:\n",
    "\n",
    "![](img/pwcnet-sm-6-2-multisteps-chairsthingsmix/loss.png)\n",
    "![](img/pwcnet-sm-6-2-multisteps-chairsthingsmix/epe.png)\n",
    "![](img/pwcnet-sm-6-2-multisteps-chairsthingsmix/lr.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the predictions issued by the model for a few validation samples:\n",
    "\n",
    "![](img/pwcnet-sm-6-2-multisteps-chairsthingsmix/val1.png)\n",
    "![](img/pwcnet-sm-6-2-multisteps-chairsthingsmix/val2.png)\n",
    "![](img/pwcnet-sm-6-2-multisteps-chairsthingsmix/val3.png)\n",
    "![](img/pwcnet-sm-6-2-multisteps-chairsthingsmix/val4.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}