3. Training Siamese Network on Conv Features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build and Train SiameseNet with Triplet Loss\n",
    "\n",
    "Now when we have the encodings of all faces, we can train small siamese model (even on our laptop) to distinguish whether 2 images show the same person."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "celeb_data = pd.read_csv('identity_CelebA.txt', sep=\" \", header=None)\n",
    "celeb_data.columns = [\"image\", \"label\"]\n",
    "partition = pd.read_csv('list_eval_partition.csv')\n",
    "df_train = celeb_data[partition.partition==0]\n",
    "df_valid = celeb_data[partition.partition==1]\n",
    "df_test = celeb_data[partition.partition==2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "convfeats = np.load('conv_feats.npy')\n",
    "labels = celeb_data['label'].values\n",
    "\n",
    "X_train = convfeats[partition.partition==0]\n",
    "X_valid = convfeats[partition.partition==1]\n",
    "y_train = labels[partition.partition==0]\n",
    "y_valid = labels[partition.partition==1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Siamese Model\n",
    "Siamese model will encode the conv features to a 256 dim vector that will represent the image. \n",
    "Kind of a signature of the images. \n",
    "\n",
    "The fundamental assumption of this model is that same person should have similar \"signatures\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Input, Dense, LeakyReLU, Concatenate, Lambda, BatchNormalization\n",
    "from keras import backend as K\n",
    "from keras.models import Model, load_model\n",
    "\n",
    "def embedder(conv_feat_size):\n",
    "    input = Input((conv_feat_size,), name = 'input')\n",
    "    normalize = Lambda(lambda x: K.l2_normalize(x, axis=-1), name='normalize')\n",
    "    x = Dense(512)(input)\n",
    "    x = LeakyReLU(alpha=0.1)(x)\n",
    "    x = Dense(128)(x)\n",
    "    x = normalize(x)\n",
    "    model = Model(input, x)\n",
    "    return model\n",
    "    \n",
    "def get_siamese_model(conv_feat_size=2048):\n",
    "    \n",
    "    input_a = Input( (conv_feat_size,),  name='anchor')\n",
    "    input_p = Input( (conv_feat_size,),  name='positive')\n",
    "    input_n = Input( (conv_feat_size,),  name='negative')\n",
    "    \n",
    "    emb_model = embedder(conv_feat_size)\n",
    "    output_a = emb_model(input_a)\n",
    "    output_p = emb_model(input_p)\n",
    "    output_n = emb_model(input_n)\n",
    "    \n",
    "    merged_vector = Concatenate(axis=-1)([output_a, output_p, output_n])\n",
    "    model = Model(inputs=[input_a, input_p, input_n],\n",
    "                  outputs=merged_vector)\n",
    "\n",
    "    return model\n",
    "\n",
    "model = get_siamese_model(conv_feat_size = convfeats.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Siamese Model Loss - Triplet Loss\n",
    "\n",
    "Same person should have 'similar' signatures between his images, whilst new person should have differnet signature.\n",
    "One way to compare the \"similarity\" between this signatures (vectors) is to use euclidean distance metric or cosine distance.\n",
    "\n",
    "I chose to use cosine distance: https://en.wikipedia.org/wiki/Cosine_similarity\n",
    "But you can easily change it and check if you're getting better results. If you do, let me know :)\n",
    "\n",
    "Let's define 3 variables:\n",
    "1. Anchor - The image against which comparisons will be made\n",
    "2. Positive - Different image of the person in the anchor image\n",
    "3. Negative - image of a different person\n",
    "\n",
    "Our loss is essentially: \n",
    "\n",
    "###### Loss = Cos_dist(Anchor, Positive) - Cos_dist(Anchor, Negative) + alpha\n",
    "\n",
    "for more information visit here: https://towardsdatascience.com/siamese-network-triplet-loss-b4ca82c1aec8\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def triplet_loss(y_true, y_pred, cosine = True, alpha = 0.2, embedding_size = 128):\n",
    "    \n",
    "    ind = int(embedding_size * 2)\n",
    "    a_pred = y_pred[:, :embedding_size]\n",
    "    p_pred = y_pred[:, embedding_size:ind]\n",
    "    n_pred = y_pred[:, ind:]\n",
    "    if cosine:\n",
    "        positive_distance = 1 - K.sum((a_pred * p_pred), axis=-1)\n",
    "        negative_distance = 1 - K.sum((a_pred * n_pred), axis=-1)\n",
    "    else:\n",
    "        positive_distance = K.sqrt(K.sum(K.square(a_pred - p_pred), axis=-1))\n",
    "        negative_distance = K.sqrt(K.sum(K.square(a_pred - n_pred), axis=-1))\n",
    "    loss = K.maximum(0.0, positive_distance - negative_distance + alpha)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create image generator for siamese network\n",
    "The input of the model will be mini-batches of [Anchors, Positives, Negatives] conv features of the images. \n",
    "\n",
    "Don't forget - we train on conv features and not on the origianl images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import Sequence\n",
    "\n",
    "class EmbLoader(Sequence):\n",
    "    def __init__(self, convfeats, labels, batchSize = 16):\n",
    "        self.X = convfeats\n",
    "        self.batchSize = batchSize\n",
    "        self.y = labels\n",
    "        self.POS = np.zeros((batchSize, convfeats.shape[1]))\n",
    "        self.NEG = np.zeros((batchSize, convfeats.shape[1]))\n",
    "    #gets the number of batches this generator returns\n",
    "    def __len__(self):\n",
    "        l,rem = divmod(len(self.y), self.batchSize)\n",
    "        return (l + (1 if rem > 0 else 0))\n",
    "    #shuffles data on epoch end\n",
    "    def on_epoch_end(self):\n",
    "        a = np.arange(len(self.y))\n",
    "        np.random.shuffle(a)\n",
    "        self.X = self.X[a]\n",
    "        self.y = self.y[a]\n",
    "    #gets a batch with index = i\n",
    "    def __getitem__(self, i):\n",
    "        start = i*self.batchSize\n",
    "        stop  = (i+1)*self.batchSize\n",
    "        ancor_labels = self.y[start:stop]\n",
    "        ancors = self.X[start:stop]\n",
    "        for k, label in enumerate(ancor_labels):\n",
    "            pos_idx = np.where(self.y==label)[0]\n",
    "            neg_idx = np.where(self.y!=label)[0]\n",
    "            self.NEG[k] = self.X[np.random.choice(neg_idx)]\n",
    "            pos_idx_hat = pos_idx[(pos_idx<start) | (pos_idx>stop)]\n",
    "            if len(pos_idx_hat):\n",
    "                self.POS[k] = self.X[np.random.choice(pos_idx_hat)]\n",
    "            else:\n",
    "                # positive examples are within the batch or just 1 example in dataset\n",
    "                self.POS[k] = self.X[np.random.choice(pos_idx)]\n",
    "        return [ancors, self.POS[:k+1], self.NEG[:k+1]], np.empty(k+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0403 - val_loss: 0.0399\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 0.03989, saving model to siamese.h5\n",
      "Epoch 2/150\n",
      "2544/2544 [==============================] - 27s 11ms/step - loss: 0.0391 - val_loss: 0.0395\n",
      "\n",
      "Epoch 00002: val_loss improved from 0.03989 to 0.03951, saving model to siamese.h5\n",
      "Epoch 3/150\n",
      "2539/2544 [============================>.] - ETA: 0s - loss: 0.0379Epoch 3/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0379 - val_loss: 0.0395\n",
      "\n",
      "Epoch 00003: val_loss improved from 0.03951 to 0.03950, saving model to siamese.h5\n",
      "Epoch 4/150\n",
      "2544/2544 [==============================] - 27s 11ms/step - loss: 0.0378 - val_loss: 0.0389\n",
      "\b\n",
      "Epoch 00004: val_loss improved from 0.03950 to 0.03894, saving model to siamese.h5\n",
      "Epoch 5/150\n",
      "2538/2544 [============================>.] - ETA: 0s - loss: 0.0378\n",
      "2544/2544 [==============================] - 27s 11ms/step - loss: 0.0378 - val_loss: 0.0401\n",
      "\n",
      "Epoch 00005: val_loss did not improve from 0.03894\n",
      "Epoch 6/150\n",
      "2543/2544 [============================>.] - ETA: 0s - loss: 0.0380\n",
      "Epoch 00005: val_loss did not improve from 0.03894\n",
      "Epoch 6/150\n",
      "2544/2544 [==============================] - 29s 12ms/step - loss: 0.0380 - val_loss: 0.0378\n",
      "\n",
      "Epoch 00006: val_loss improved from 0.03894 to 0.03779, saving model to siamese.h5\n",
      "Epoch 7/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0374 - val_loss: 0.0384\n",
      "\b\n",
      "Epoch 00007: val_loss did not improve from 0.03779\n",
      "Epoch 8/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0373 - val_loss: 0.0395\n",
      "\n",
      "Epoch 00008: val_loss did not improve from 0.03779\n",
      "Epoch 9/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0371 - val_loss: 0.0386\n",
      "\n",
      "Epoch 00009: val_loss did not improve from 0.03779\n",
      "Epoch 10/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0366 - val_loss: 0.0381\n",
      "\n",
      "Epoch 00010: val_loss did not improve from 0.03779\n",
      "Epoch 11/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0364 - val_loss: 0.0382\n",
      "\n",
      "Epoch 00011: val_loss did not improve from 0.03779\n",
      "Epoch 12/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0367 - val_loss: 0.0377\n",
      "\n",
      "Epoch 00012: val_loss improved from 0.03779 to 0.03771, saving model to siamese.h5\n",
      "Epoch 13/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0364 - val_loss: 0.0382\n",
      "\n",
      "Epoch 00013: val_loss did not improve from 0.03771\n",
      "Epoch 14/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0366 - val_loss: 0.0390\n",
      "\n",
      "Epoch 00014: val_loss did not improve from 0.03771\n",
      "Epoch 15/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0361 - val_loss: 0.0388\n",
      "\n",
      "Epoch 00015: val_loss did not improve from 0.03771\n",
      "Epoch 16/150\n",
      "2544/2544 [==============================] - 29s 11ms/step - loss: 0.0367 - val_loss: 0.0372\n",
      "\n",
      "Epoch 00016: val_loss improved from 0.03771 to 0.03723, saving model to siamese.h5\n",
      "Epoch 17/150\n",
      "2544/2544 [==============================] - 27s 11ms/step - loss: 0.0360 - val_loss: 0.0373\n",
      "\n",
      "Epoch 00017: val_loss did not improve from 0.03723\n",
      "Epoch 18/150\n",
      "2543/2544 [============================>.] - ETA: 0s - loss: 0.0365\n",
      "Epoch 00017: val_loss did not improve from 0.03723\n",
      "Epoch 18/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0365 - val_loss: 0.0368\n",
      "\n",
      "Epoch 00018: val_loss improved from 0.03723 to 0.03682, saving model to siamese.h5\n",
      "Epoch 19/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0364 - val_loss: 0.0383\n",
      "\n",
      "Epoch 00019: val_loss did not improve from 0.03682\n",
      "Epoch 20/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0357 - val_loss: 0.0391\n",
      "\n",
      "Epoch 00020: val_loss did not improve from 0.03682\n",
      "Epoch 21/150\n",
      "2544/2544 [==============================] - 30s 12ms/step - loss: 0.0361 - val_loss: 0.0373\n",
      "\n",
      "Epoch 00021: val_loss did not improve from 0.03682\n",
      "Epoch 22/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0359 - val_loss: 0.0367\n",
      "\n",
      "Epoch 00022: val_loss improved from 0.03682 to 0.03666, saving model to siamese.h5\n",
      "Epoch 23/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0358 - val_loss: 0.0372\n",
      "\n",
      "Epoch 00023: val_loss did not improve from 0.03666\n",
      "Epoch 24/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0360 - val_loss: 0.0390\n",
      "\n",
      "Epoch 00024: val_loss did not improve from 0.03666\n",
      "Epoch 25/150\n",
      "2541/2544 [============================>.] - ETA: 0s - loss: 0.0355\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0355 - val_loss: 0.0367\n",
      "\n",
      "Epoch 00025: val_loss did not improve from 0.03666\n",
      "Epoch 26/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0354 - val_loss: 0.0366\n",
      "   5/2544 [..............................] - ETA: 33:11 - loss: 0.0403\n",
      "Epoch 00026: val_loss improved from 0.03666 to 0.03662, saving model to siamese.h5\n",
      "Epoch 27/150\n",
      "2541/2544 [============================>.] - ETA: 0s - loss: 0.0356Epoch 27/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0356 - val_loss: 0.0369\n",
      "\n",
      "Epoch 00027: val_loss did not improve from 0.03662\n",
      "Epoch 28/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0356 - val_loss: 0.0361\n",
      "\n",
      "Epoch 00028: val_loss improved from 0.03662 to 0.03609, saving model to siamese.h5\n",
      "Epoch 29/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0357 - val_loss: 0.0377\n",
      "\n",
      "Epoch 00029: val_loss did not improve from 0.03609\n",
      "Epoch 30/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0350 - val_loss: 0.0371\n",
      "\n",
      "Epoch 00030: val_loss did not improve from 0.03609\n",
      "Epoch 31/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0353 - val_loss: 0.0357\n",
      "\n",
      "Epoch 00031: val_loss improved from 0.03609 to 0.03572, saving model to siamese.h5\n",
      "Epoch 32/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0350 - val_loss: 0.0380\n",
      "\n",
      "Epoch 00032: val_loss did not improve from 0.03572\n",
      "Epoch 33/150\n",
      "2544/2544 [==============================] - 29s 11ms/step - loss: 0.0351 - val_loss: 0.0368\n",
      "\n",
      "Epoch 00033: val_loss did not improve from 0.035\n",
      "Epoch 34/150\n",
      "2538/2544 [============================>.] - ETA: 0s - loss: 0.0344\n",
      "Epoch 34/150\n",
      "2544/2544 [==============================] - 30s 12ms/step - loss: 0.0344 - val_loss: 0.0360\n",
      "\n",
      "Epoch 00034: val_loss did not improve from 0.03572\n",
      "Epoch 35/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0349 - val_loss: 0.0366\n",
      "\n",
      "Epoch 00035: val_loss did not improve from 0.03572\n",
      "Epoch 36/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0349 - val_loss: 0.0365\n",
      "\n",
      "Epoch 00036: val_loss did not improve from 0.03572\n",
      "Epoch 37/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0348 - val_loss: 0.0380\n",
      "\n",
      "Epoch 00037: val_loss did not improve from 0.03572\n",
      "Epoch 38/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0349 - val_loss: 0.0365\n",
      "\n",
      "Epoch 00038: val_loss did not improve from 0.03572\n",
      "Epoch 39/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0349 - val_loss: 0.0362\n",
      "\n",
      "Epoch 00039: val_loss did not improve from 0.03572\n",
      "Epoch 40/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0346 - val_loss: 0.0364\n",
      "\n",
      "Epoch 00040: val_loss did not improve from 0.03572\n",
      "Epoch 41/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0349 - val_loss: 0.0355\n",
      "\n",
      "Epoch 00041: val_loss improved from 0.03572 to 0.03546, saving model to siamese.h5\n",
      "Epoch 42/150\n",
      "2542/2544 [============================>.] - ETA: 0s - loss: 0.0350\n",
      "Epoch 00041: val_loss improved from 0.03572 to 0.03546, saving model to siamese.h5\n",
      "Epoch 42/150\n",
      "2544/2544 [==============================] - 27s 11ms/step - loss: 0.0350 - val_loss: 0.0369\n",
      "\n",
      "Epoch 00042: val_loss did not improve from 0.03546\n",
      "Epoch 43/150\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0346 - val_loss: 0.0361\n",
      "\n",
      "Epoch 00043: val_loss did not improve from 0.03546\n",
      "Epoch 44/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0345 - val_loss: 0.0359\n",
      "\b\n",
      "Epoch 00044: val_loss did not improve from 0.03546\n",
      "Epoch 45/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0338 - val_loss: 0.0352\n",
      "\n",
      "Epoch 00045: val_loss improved from 0.03546 to 0.03516, saving model to siamese.h5\n",
      "Epoch 46/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0347 - val_loss: 0.0355\n",
      "\n",
      "Epoch 00046: val_loss did not improve from 0.03516\n",
      "Epoch 47/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0347 - val_loss: 0.0366\n",
      "\n",
      "Epoch 00047: val_loss did not improve from 0.03516\n",
      "Epoch 48/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0344 - val_loss: 0.0372\n",
      "\n",
      "Epoch 00048: val_loss did not improve from 0.03516\n",
      "Epoch 49/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0341 - val_loss: 0.0380\n",
      "\n",
      "Epoch 00049: val_loss did not improve from 0.03516\n",
      "Epoch 50/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0344 - val_loss: 0.0382\n",
      "\n",
      "Epoch 00050: val_loss did not improve from 0.03516\n",
      "Epoch 51/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0344 - val_loss: 0.0366\n",
      "\n",
      "Epoch 00051: val_loss did not improve from 0.03516\n",
      "Epoch 52/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0343 - val_loss: 0.0352\n",
      "\b\n",
      "Epoch 00052: val_loss did not improve from 0.03516\n",
      "Epoch 53/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0340 - val_loss: 0.0369\n",
      "\n",
      "Epoch 00053: val_loss did not improve from 0.03516\n",
      "Epoch 54/150\n",
      "2544/2544 [==============================] - 27s 11ms/step - loss: 0.0348 - val_loss: 0.0356\n",
      "\n",
      "Epoch 00054: val_loss did not improve from 0.03516\n",
      "Epoch 55/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0340 - val_loss: 0.0364\n",
      "\n",
      "Epoch 00055: val_loss did not improve from 0.03516\n",
      "Epoch 56/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0342 - val_loss: 0.0352\n",
      "\n",
      "Epoch 00056: val_loss did not improve from 0.03516\n",
      "Epoch 57/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0341 - val_loss: 0.0358\n",
      "\n",
      "Epoch 00057: val_loss did not improve from 0.03516\n",
      "Epoch 58/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0341 - val_loss: 0.0352\n",
      "\n",
      "Epoch 00058: val_loss did not improve from 0.03516\n",
      "Epoch 59/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0340 - val_loss: 0.0371\n",
      "\n",
      "Epoch 00059: val_loss did not improve from 0.03516\n",
      "Epoch 60/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0342 - val_loss: 0.0356\n",
      "\n",
      "Epoch 00060: val_loss did not improve from 0.03516\n",
      "Epoch 61/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0338 - val_loss: 0.0360\n",
      "\n",
      "Epoch 00061: val_loss did not improve from 0.03516\n",
      "Epoch 62/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0338 - val_loss: 0.0355\n",
      "\n",
      "Epoch 00062: val_loss did not improve from 0.03516\n",
      "Epoch 63/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0338 - val_loss: 0.0355\n",
      "\n",
      "Epoch 00063: val_loss did not improve from 0.03516\n",
      "Epoch 64/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0339 - val_loss: 0.0346\n",
      "\n",
      "Epoch 00064: val_loss improved from 0.03516 to 0.03456, saving model to siamese.h5\n",
      "Epoch 65/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0343 - val_loss: 0.0358\n",
      "\n",
      "Epoch 00065: val_loss did not improve from 0.03456\n",
      "Epoch 66/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0336 - val_loss: 0.0371\n",
      "\n",
      "Epoch 00066: val_loss did not improve from 0.03456\n",
      "Epoch 67/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0338 - val_loss: 0.0358\n",
      "\n",
      "Epoch 00067: val_loss did not improve from 0.03456\n",
      "Epoch 68/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0336 - val_loss: 0.0355\n",
      "\n",
      "Epoch 00071: val_loss did not improve from 0.03391\n",
      "Epoch 72/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0338 - val_loss: 0.0354\n",
      "  13/2544 [..............................] - ETA: 14:15 - loss: 0.0309\n",
      "Epoch 00072: val_loss did not improve from 0.03391\n",
      "Epoch 73/150\n",
      "2544/2544 [==============================] - 26s 10ms/step - loss: 0.0338 - val_loss: 0.0351\n",
      "\n",
      "Epoch 00073: val_loss did not improve from 0.03391\n",
      "Epoch 74/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0335 - val_loss: 0.0352\n",
      "\n",
      "Epoch 00078: val_loss did not improve from 0.03391\n",
      "Epoch 79/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0340 - val_loss: 0.0362\n",
      "\n",
      "Epoch 00079: val_loss did not improve from 0.03391\n",
      "Epoch 80/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0336 - val_loss: 0.0350\n",
      "\n",
      "Epoch 00080: val_loss did not improve from 0.03391\n",
      "Epoch 81/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0334 - val_loss: 0.0351\n",
      "\n",
      "Epoch 00081: val_loss did not improve from 0.03391\n",
      "Epoch 82/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0339 - val_loss: 0.0345\n",
      "\n",
      "Epoch 00082: val_loss did not improve from 0.03391\n",
      "Epoch 83/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0330 - val_loss: 0.0367\n",
      "\n",
      "Epoch 00083: val_loss did not improve from 0.03391\n",
      "Epoch 84/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0333 - val_loss: 0.0343\n",
      "\n",
      "Epoch 00084: val_loss did not improve from 0.03391\n",
      "Epoch 85/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0335 - val_loss: 0.0360\n",
      "\n",
      "Epoch 00085: val_loss did not improve from 0.03391\n",
      "Epoch 86/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0333 - val_loss: 0.0360\n",
      "\n",
      "Epoch 00086: val_loss did not improve from 0.03391\n",
      "Epoch 87/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0336 - val_loss: 0.0355\n",
      "\n",
      "Epoch 00087: val_loss did not improve from 0.03391\n",
      "Epoch 88/150\n",
      "2544/2544 [==============================] - 29s 11ms/step - loss: 0.0334 - val_loss: 0.0352\n",
      "\n",
      "Epoch 00088: val_loss did not improve from 0.03391\n",
      "Epoch 89/150\n",
      "2544/2544 [==============================] - 29s 12ms/step - loss: 0.0336 - val_loss: 0.0354\n",
      "\n",
      "Epoch 00089: val_loss did not improve from 0.03391\n",
      "Epoch 90/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0332 - val_loss: 0.0347\n",
      "\n",
      "Epoch 00093: val_loss did not improve from 0.03391\n",
      "Epoch 94/150\n",
      "2544/2544 [==============================] - 30s 12ms/step - loss: 0.0331 - val_loss: 0.0353\n",
      "\n",
      "Epoch 00094: val_loss did not improve from 0.03391\n",
      "Epoch 95/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0334 - val_loss: 0.0349\n",
      "\n",
      "Epoch 00095: val_loss did not improve from 0.03391\n",
      "Epoch 96/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0330 - val_loss: 0.0344\n",
      "\n",
      "Epoch 00096: val_loss did not improve from 0.03391\n",
      "Epoch 97/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0338 - val_loss: 0.0343\n",
      "\n",
      "Epoch 00097: val_loss did not improve from 0.03391\n",
      "Epoch 98/150\n",
      "2544/2544 [==============================] - 31s 12ms/step - loss: 0.0334 - val_loss: 0.0349\n",
      "\n",
      "Epoch 00098: val_loss did not improve from 0.03391\n",
      "Epoch 99/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0331 - val_loss: 0.0353\n",
      "\n",
      "Epoch 00099: val_loss did not improve from 0.03391\n",
      "Epoch 100/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0331 - val_loss: 0.0338\n",
      "\n",
      "Epoch 00100: val_loss improved from 0.03391 to 0.03377, saving model to siamese.h5\n",
      "Epoch 101/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0335 - val_loss: 0.0353\n",
      "\n",
      "Epoch 00101: val_loss did not improve from 0.03377\n",
      "Epoch 102/150\n",
      "2544/2544 [==============================] - 25s 10ms/step - loss: 0.0330 - val_loss: 0.0350\n",
      "\n",
      "Epoch 00102: val_loss did not improve from 0.03377\n",
      "Epoch 103/150\n",
      "2544/2544 [==============================] - 28s 11ms/step - loss: 0.0328 - val_loss: 0.0338\n",
      "\n",
      "Epoch 00103: val_loss did not improve from 0.03377\n",
      "Epoch 104/150\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 230/2544 [=>............................] - ETA: 1:21 - loss: 0.0331"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Process ForkPoolWorker-2478:\n",
      "Process ForkPoolWorker-2485:\n",
      "Process ForkPoolWorker-2492:\n",
      "Process ForkPoolWorker-2479:\n",
      "Process ForkPoolWorker-2484:\n",
      "Process ForkPoolWorker-2490:\n",
      "Process ForkPoolWorker-2474:\n",
      "Process ForkPoolWorker-2482:\n",
      "Process ForkPoolWorker-2473:\n",
      "Process ForkPoolWorker-2495:\n",
      "Process ForkPoolWorker-2494:\n",
      "Process ForkPoolWorker-2489:\n",
      "Process ForkPoolWorker-2481:\n",
      "Process ForkPoolWorker-2477:\n",
      "Process ForkPoolWorker-2487:\n",
      "Process ForkPoolWorker-2476:\n",
      "Process ForkPoolWorker-2480:\n",
      "Process ForkPoolWorker-2483:\n",
      "Process ForkPoolWorker-2486:\n",
      "Process ForkPoolWorker-2488:\n",
      "Process ForkPoolWorker-2493:\n",
      "Process ForkPoolWorker-2496:\n",
      "Process ForkPoolWorker-2491:\n",
      "Process ForkPoolWorker-2475:\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 119, in worker\n",
      "    result = (True, func(*args, **kwds))\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 119, in worker\n",
      "    result = (True, func(*args, **kwds))\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 119, in worker\n",
      "    result = (True, func(*args, **kwds))\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py\", line 401, in get_index\n",
      "    return _SHARED_SEQUENCES[uid][i]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py\", line 401, in get_index\n",
      "    return _SHARED_SEQUENCES[uid][i]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py\", line 401, in get_index\n",
      "    return _SHARED_SEQUENCES[uid][i]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/process.py\", line 93, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"<ipython-input-5-b97b66e36043>\", line 27, in __getitem__\n",
      "    pos_idx = np.where(self.y==label)[0]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"<ipython-input-5-b97b66e36043>\", line 28, in __getitem__\n",
      "    neg_idx = np.where(self.y!=label)[0]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 119, in worker\n",
      "    result = (True, func(*args, **kwds))\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/pool.py\", line 108, in worker\n",
      "    task = get()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 343, in get\n",
      "    res = self._reader.recv_bytes()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"<ipython-input-5-b97b66e36043>\", line 29, in __getitem__\n",
      "    self.NEG[k] = self.X[np.random.choice(neg_idx)]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py\", line 401, in get_index\n",
      "    return _SHARED_SEQUENCES[uid][i]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 342, in get\n",
      "    with self._rlock:\n",
      "  File \"/usr/lib/python3.5/multiprocessing/queues.py\", line 343, in get\n",
      "    res = self._reader.recv_bytes()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/connection.py\", line 216, in recv_bytes\n",
      "    buf = self._recv_bytes(maxlength)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "  File \"<ipython-input-5-b97b66e36043>\", line 28, in __getitem__\n",
      "    neg_idx = np.where(self.y!=label)[0]\n",
      "  File \"/usr/lib/python3.5/multiprocessing/connection.py\", line 216, in recv_bytes\n",
      "    buf = self._recv_bytes(maxlength)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/connection.py\", line 407, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "  File \"/usr/lib/python3.5/multiprocessing/connection.py\", line 379, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n",
      "  File \"/usr/lib/python3.5/multiprocessing/synchronize.py\", line 96, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n",
      "KeyboardInterrupt\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-8c0ea4948f00>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     12\u001b[0m model.fit_generator(train_gen, steps_per_epoch=len(train_gen), epochs=150, \n\u001b[1;32m     13\u001b[0m                     \u001b[0mvalidation_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalid_gen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidation_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalid_gen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m                     workers=12, use_multiprocessing=True, callbacks=[checkpoint])\n\u001b[0m\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;31m# train for 50 more epochs with validation data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     89\u001b[0m                 warnings.warn('Update your `' + object_name + '` call to the ' +\n\u001b[1;32m     90\u001b[0m                               'Keras 2 API: ' + signature, stacklevel=2)\n\u001b[0;32m---> 91\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     92\u001b[0m         \u001b[0mwrapper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_original_function\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     93\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit_generator\u001b[0;34m(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\u001b[0m\n\u001b[1;32m   1416\u001b[0m             \u001b[0muse_multiprocessing\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_multiprocessing\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1417\u001b[0m             \u001b[0mshuffle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mshuffle\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1418\u001b[0;31m             initial_epoch=initial_epoch)\n\u001b[0m\u001b[1;32m   1419\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1420\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0minterfaces\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlegacy_generator_methods_support\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training_generator.py\u001b[0m in \u001b[0;36mfit_generator\u001b[0;34m(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\u001b[0m\n\u001b[1;32m    215\u001b[0m                 outs = model.train_on_batch(x, y,\n\u001b[1;32m    216\u001b[0m                                             \u001b[0msample_weight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msample_weight\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 217\u001b[0;31m                                             class_weight=class_weight)\n\u001b[0m\u001b[1;32m    218\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    219\u001b[0m                 \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mto_list\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mtrain_on_batch\u001b[0;34m(self, x, y, sample_weight, class_weight)\u001b[0m\n\u001b[1;32m   1215\u001b[0m             \u001b[0mins\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0msample_weights\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1216\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_train_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1217\u001b[0;31m         \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1218\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0munpack_singleton\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1219\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2713\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_legacy_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2714\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2715\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2716\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2717\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mpy_any\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mis_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2673\u001b[0m             \u001b[0mfetched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_metadata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2674\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2675\u001b[0;31m             \u001b[0mfetched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2676\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfetched\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1437\u001b[0m           ret = tf_session.TF_SessionRunCallable(\n\u001b[1;32m   1438\u001b[0m               \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_handle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1439\u001b[0;31m               run_metadata_ptr)\n\u001b[0m\u001b[1;32m   1440\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1441\u001b[0m           \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.optimizers import Adam\n",
    "# Compile the model\n",
    "model.compile(Adam(lr = 0.00005), loss = triplet_loss)\n",
    "\n",
    "# create generators\n",
    "train_gen = EmbLoader(X_train, y_train, batchSize = 64)\n",
    "valid_gen = EmbLoader(X_valid, y_valid, batchSize = 64)\n",
    "all_gen = EmbLoader(convfeats, labels, batchSize = 64)\n",
    "\n",
    "checkpoint = ModelCheckpoint('siamese.h5', monitor='val_loss', verbose=1, save_best_only=True, save_weights_only=True)\n",
    "model.fit_generator(train_gen, steps_per_epoch=len(train_gen), epochs=150, \n",
    "                    validation_data=valid_gen, validation_steps=len(valid_gen),\n",
    "                    workers=12, use_multiprocessing=True, callbacks=[checkpoint])\n",
    "\n",
    "# train for 50 more epochs with validation data\n",
    "model.load_weights('siamese.h5')\n",
    "model.fit_generator(all_gen, steps_per_epoch=len(all_gen), epochs=200, initial_epoch = 150,\n",
    "                    workers=8, use_multiprocessing=True)\n",
    "model.save('siamese.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_model = load_model('siamese_xception.h5', compile=False)\n",
    "# Modify fully-connected layers model with our new one\n",
    "\n",
    "full_model.layers[3].name = 'conv_model'\n",
    "model.layers[3].name = 'embedding_model'\n",
    "\n",
    "conv_model = full_model.layers[3]\n",
    "fc_model = model.layers[3]\n",
    "\n",
    "inp_a, inp_p, inp_n = full_model.input\n",
    "conv_a = conv_model(inp_a)\n",
    "conv_p = conv_model(inp_p)\n",
    "conv_n = conv_model(inp_n)\n",
    "out_a = fc_model(conv_a)\n",
    "out_p = fc_model(conv_p)\n",
    "out_n = fc_model(conv_n)\n",
    "\n",
    "merged_vector = Concatenate(axis=-1, name = 'Concat_3_images')([out_a, out_p, out_n])\n",
    "final_model = Model(inputs=[inp_a, inp_p, inp_n], outputs=merged_vector)\n",
    "\n",
    "final_model.save('siamese_xception.h5', include_optimizer=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_model = load_model('siamese_xception.h5', compile=False)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}