BiLSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOME NOTES\n",
    "\n",
    "- To improve performance, check https://svail.github.io/rnn_perf/\n",
    "\n",
    "    - For readers in first group (users of deep learning frameworks) - the main takeaway is to make layer sizes and mini-batch sizes multiples of 32, and if you're using cuBLAS, then make them multiples of 64 for best performance. If you're writing recurrent layers yourself, make sure to write them in such a way that you combine across time when possible. \n",
    "    \n",
    "- Try to hyperparameter optimization: http://maxpumperla.com/hyperas/\n",
    "\n",
    "- SincNet can be helpful. https://github.com/mravanelli/SincNet/\n",
    "\n",
    "- Check the transformer. https://jalammar.github.io/illustrated-transformer/ https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html http://nlp.seas.harvard.edu/2018/04/03/attention.html\n",
    "\n",
    "- Autokeras. https://github.com/jhfjhfj1/autokeras\n",
    "\n",
    "- Should we use FeedForward? https://www.offconvex.org/2018/07/27/approximating-recurrent/\n",
    "\n",
    "- TCN https://github.com/philipperemy/keras-tcn\n",
    "\n",
    "- Another method for speaker change detection https://github.com/philipperemy/speaker-change-detection\n",
    "\n",
    "- Rare sound event detection: https://www.cs.tut.fi/sgn/arg/dcase2017/documents/challenge_technical_reports/DCASE2017_Lim_204.pdf\n",
    "\n",
    "- We can augment data via [Pydub](https://github.com/jiaaro/pydub). https://towardsdatascience.com/tensorflow-speech-recognition-challenge-solution-outline-9c42dbd219c9\n",
    "\n",
    "- Convolutional Recurrent Neural Networks for Small Footprint Keyword Spotting https://arxiv.org/pdf/1703.05390.pdf\n",
    "\n",
    "- CNN and Speaker Change Detection http://mirlab.org/conference_papers/International_Conference/ICASSP%202017/pdfs/0004945.pdf\n",
    "\n",
    "- Different models for speech recognition https://github.com/lucko515/speech-recognition-neural-network/blob/master/vui_notebook.ipynb\n",
    "-----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TO-DO\n",
    "\n",
    "- Generator modelini implement et.\n",
    "\n",
    "- Visualize the system. https://musicinformationretrieval.com \n",
    "    - We can use https://github.com/stevetjoa/musicinformationretrieval.com/blob/gh-pages/onset_segmentation.ipynb\n",
    "    \n",
    "\n",
    "- Augment the input data.\n",
    "\n",
    "- Try CNN, CapsuleNet and Neural Turing Machine.\n",
    "\n",
    "- Fuzzy Labeling \n",
    "    - [We](http://mirlab.org/conference_papers/International_Conference/ICASSP%202017/pdfs/0004945.pdf) developed a new labeling strategy to cope with the uncertainty introduced by the human annotators.  Instead of the\n",
    "previously used binary labeling we use a fuzzy labeling. The\n",
    "value one is assigned when the speaker change is located in\n",
    "the middle of the analyzed window and linearly decreases to\n",
    "zero as the change moves away from the middle.\n",
    "\n",
    "-------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we are trying to reproduce [the paper](https://pdfs.semanticscholar.org/edff/b62b32ffcc2b5cc846e26375cb300fac9ecc.pdf) for speaker change detection\n",
    "\n",
    "**TODO**\n",
    "\n",
    "- Use AMI Corpus with feature extraction with pyannote.\n",
    "\n",
    "## Review\n",
    "\n",
    "**Sequence Labelling** \n",
    "\n",
    "They think this task as a binary classification. Thus, they label changing frame as a **1** and non-changing frame as a **0**. So that, they use the _binary cross-entropy loss function_.\n",
    "\n",
    "**Network Architecture**\n",
    "- 2 Bi-LSTM\n",
    "    - 64 and 32 outputs respectively.\n",
    "- Multi Layer Perceptron\n",
    "    - 3 Fully Connected Feedforward Layers\n",
    "        - 40, 20, 1 dimensional respectively.\n",
    "    - Tanh activation for first 2 layer\n",
    "    - Sigmoid activation for last layer\n",
    "    \n",
    "**Feature Extraction**\n",
    "- \"35-dimensional acoustic features are extracted every 16ms on a 32ms window using [Yaafe toolkit](http://yaafe.sourceforge.net).\"\n",
    "    - 11 Mel-Frequency Cepstral Coefficients (MFCC), \n",
    "    - Their first and second derivatives,\n",
    "    - First and second derivatives of the energy.\n",
    "\n",
    "**Class Imbalance**\n",
    "\n",
    "- _\"The number of positive labels isincreased artificially by labeling as positive every frame in the direct neighborhood of the manually annotated change point.\"_\n",
    "- A positive neighborhood of 100ms (50ms on both sides) is used around each change point, to partially solve the class imbalance problem.\n",
    "\n",
    "**Subsequences**\n",
    "\n",
    "- _\"The long audio sequences are split into short fixed-length overlapping sequences.\"_\n",
    "\n",
    "**Prediction**\n",
    "\n",
    "- _\"Finally, local score maxima exceeding a pre-determined threshold θ are marked as speaker change points.\"_\n",
    "\n",
    "**Training**\n",
    "\n",
    "- Subsequences for training are 3.2s long with a step of 800ms (i.e. two adjacent sequences overlap by 75%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**From Codebase**\n",
    "\n",
    "- For Optimizer: _SGD, momentum=0.9, nesterov=True_\n",
    "\n",
    "- Architecture:\n",
    "       name: StackedRNN\n",
    "       params:\n",
    "         rnn: LSTM\n",
    "         recurrent: [32, 20]\n",
    "         bidirectional: True\n",
    "         linear: [40, 10]\n",
    "         \n",
    "-  create final classification layer (with log-softmax activation)\n",
    "    - NLLLoss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code\n",
    "\n",
    "### Feature Extraction\n",
    "\n",
    "We will use Yaafe Toolkit. (To see all available features, you can use _!yaafe -l_) To learn how we can do that, start with http://yaafe.github.io/Yaafe/manual/quickstart.html#quick-start-using-yaafe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can view a description of each feature (or output format) with the -d option:\n",
    "\n",
    "# !yaafe -d MFCC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !yaafe -d Energy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's determine blockSize and stepSize. \n",
    "\n",
    "If we have 16kHz audio signal(in AMI, we have 16kHz), for 32 ms block, we need 16x32, For the stepsize as 16 ms, we need 16x16 size.\n",
    "\n",
    "We need these features:\n",
    "\n",
    "- mfcc: MFCC blockSize=512 stepSize=256 CepsNbCoeffs=11\n",
    "- mfcc_d1: MFCC blockSize=512 stepSize=256 CepsNbCoeffs=11 > Derivate DOrder=1\n",
    "- mfcc_d2: MFCC blockSize=512 stepSize=256 CepsNbCoeffs=11 > Derivate DOrder=2\n",
    "- energy_d1: Energy blockSize=512 stepSize=256  > Derivate DOrder=1\n",
    "- energy_d2: Energy blockSize=512 stepSize=256  > Derivate DOrder=2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To extract all of these, we will use [this technique](http://yaafe.github.io/Yaafe/manual/quickstart.html#extract-several-features). Shortly, we will write all these features into single text file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want to use 32 ms window length with 16 ms hop\n",
    "# This feature extraction is based on paper\n",
    "\n",
    "f = open(\"featureplan.txt\", \"w\")\n",
    "f.write(\"mfcc: MFCC blockSize=512 stepSize=256 CepsNbCoeffs=11 \\n\"\n",
    "        \"mfcc_d1: MFCC blockSize=512 stepSize=256 CepsNbCoeffs=11 > Derivate DOrder=1 \\n\"\n",
    "        \"mfcc_d2: MFCC blockSize=512 stepSize=256 CepsNbCoeffs=11 > Derivate DOrder=2 \\n\"\n",
    "        \"energy_d1: Energy blockSize=512 stepSize=256  > Derivate DOrder=1 \\n\"\n",
    "        \"energy_d2: Energy blockSize=512 stepSize=256  > Derivate DOrder=2\")\n",
    "f.close() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want to use 25 ms window length with 10 ms hop\n",
    "# This feature extraction is based on pyannote library\n",
    "\n",
    "f = open(\"featureplan_new.txt\", \"w\")\n",
    "f.write(\"mfcc: MFCC blockSize=400 stepSize=160 CepsNbCoeffs=19 \\n\"\n",
    "        \"mfcc_d1: MFCC blockSize=400 stepSize=160 CepsNbCoeffs=19 > Derivate DOrder=1 \\n\"\n",
    "        \"mfcc_d2: MFCC blockSize=400 stepSize=160 CepsNbCoeffs=19 > Derivate DOrder=2 \\n\"\n",
    "        \"energy_d1: Energy blockSize=400 stepSize=160 > Derivate DOrder=1 \\n\"\n",
    "        \"energy_d2: Energy blockSize=400 stepSize=160 > Derivate DOrder=2\")\n",
    "f.close() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, you can test the yaafe with following command\n",
    "\n",
    "# !yaafe -c \"featureplan_new.txt\" -r 16000 \"ES2009a.Mix-Headset.wav\" -p Precision=6 -p Metadata=False -n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", message=\"numpy.dtype size changed\")\n",
    "\n",
    "from numpy import genfromtxt\n",
    "import numpy as np\n",
    "import os\n",
    "import glob\n",
    "\n",
    "import librosa\n",
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "import math\n",
    "import sys\n",
    "\n",
    "def create_numpy_for_audio(audio_file, featureplan = \"\", feature_extractor=\"yaafe\", hop=10, win_len=25, sr=16000):\n",
    "    \"\"\"This function is based on YAAFE or Librosa. It will return 2D Array which is features of audio file. \n",
    "    Also it will save the numpy array.\n",
    "    \n",
    "    Its arguments:\n",
    "    audio_file: Path of audio file, it can be wav, mp3, ogg etc.\n",
    "    featureplan: Text file which introduce which features will be extracted.(for Yaafe)\n",
    "    feature_extractor: Which library will be used to extract features.\n",
    "    hop: Hop length (we need it for Librosa)\n",
    "    win_len: Window length (we need it for Librosa)\n",
    "    sr: Sample rate for audio file. Default as 16000.\"\"\"\n",
    "    \n",
    "    if (feature_extractor==\"yaafe\"):\n",
    "        !yaafe -c $featureplan -r $sr $audio_file -p Precision=6 -p Metadata=False -n\n",
    "        filename = (audio_file.split(\"/\")[-1]).split(\".\")[0]\n",
    "\n",
    "        my_data = genfromtxt(audio_file + \".mfcc.csv\", delimiter=',')\n",
    "        my_data = np.append(my_data, genfromtxt(audio_file + \".mfcc_d1.csv\", delimiter=','), axis=1)\n",
    "        my_data = np.append(my_data, genfromtxt(audio_file + \".mfcc_d2.csv\", delimiter=','), axis=1)\n",
    "\n",
    "        my_data = np.append(my_data, np.expand_dims(genfromtxt(audio_file + \".energy_d1.csv\", delimiter=','), axis=1), axis=1)\n",
    "        my_data = np.append(my_data, np.expand_dims(genfromtxt(audio_file + \".energy_d2.csv\", delimiter=','), axis=1), axis=1)\n",
    "\n",
    "        # Previous codes creates csv file for features to load numpy array. After that, we can \n",
    "        # remove them.\n",
    "        os.remove(audio_file + \".mfcc.csv\")\n",
    "        os.remove(audio_file + \".mfcc_d1.csv\")\n",
    "        os.remove(audio_file + \".mfcc_d2.csv\")\n",
    "        os.remove(audio_file + \".energy_d1.csv\")\n",
    "        os.remove(audio_file + \".energy_d2.csv\")\n",
    "        \n",
    "        np.save(\"./yaafe_ami_storage/\" + filename, my_data)\n",
    "\n",
    "        return my_data\n",
    "    \n",
    "    if (feature_extractor==\"librosa\"):\n",
    "        audio, sr = librosa.load(filename)\n",
    "        \n",
    "        # https://github.com/librosa/librosa/issues/584\n",
    "        mfccs = librosa.feature.mfcc(audio, sr, n_mfcc=11, hop_length=int(float(hop/1000)*sr), n_fft=int(float(win_len/1000)*sr))\n",
    "        mfccs_d1 = librosa.feature.delta(mfccs)\n",
    "        mfccs_d2 = librosa.feature.delta(mfccs, order=2)\n",
    "        energy = librosa.feature.rmse(y=audio, hop_length=int(float(hop/1000)*sr), frame_length=int(float(win_len/1000)*sr))\n",
    "        energy_d1 = librosa.feature.delta(energy)\n",
    "        energy_d2 = librosa.feature.delta(energy, order=2)\n",
    "\n",
    "        my_data = np.vstack((mfccs, mfccs_d1, mfccs_d2, energy_d1, energy_d2))\n",
    "        \n",
    "        np.save(\"./librosa_ami_storage/\" + filename, my_data)\n",
    "        \n",
    "        return my_data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand, how normalization and delta features affect performance https://ieeexplore.ieee.org/document/5711789/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "import glob\n",
    "import os\n",
    "import matplotlib.pyplot as pp\n",
    "import more_itertools as mit\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "def create_data_for_supervised(root_dir, hop, win_len, from_ep=0, to_ep=0, boost_for_imbalance=False, \n",
    "                              how_much_boost=6, balance=False, \n",
    "                              feature_extractor=\"pyannote\",\n",
    "                              fuzzy_label=False):\n",
    "    \n",
    "    \"\"\"If we have numpy array in the folder, it will create input and output array via this function.\n",
    "    \n",
    "    Arguments:\n",
    "    root_dir: The folder which stores numpy array.\n",
    "    hop: Hop length (we need it for Librosa).\n",
    "    win_len: Window length (we need it for Librosa).\n",
    "    from_ep: Location of first file which will be loaded into array.\n",
    "    to_ep: Location of last file which will be loaded into array.\n",
    "    boost_for_imbalance: If it is true, the number of positive labels is\n",
    "        increased artificially by labeling as positive every frame in the\n",
    "        direct  neighborhood  of  the  manually  annotated  change  point\n",
    "        and this number will be determined by how_much_boost parameter.\n",
    "    balance: if it is true, we will discard some frame which are at the \n",
    "        middle of single speaker segment.\n",
    "    feature_extractor: Which library will be used. (now, we can use Pyannote or Yaafe.)\n",
    "    fuzzy_label: If it is true, we will label the neighboorhood frames\n",
    "        with decreasing number. (for instance: if we will use 5 frame as a boost\n",
    "        middle frame will be 1, next frame will be 0.8, next one will be 0.6)\n",
    "    \"\"\"\n",
    "    \n",
    "    audios_path = glob.glob(os.path.join(root_dir, '*wav'))\n",
    "    input_array = []\n",
    "    \n",
    "    output= []\n",
    "    \n",
    "    num = 0\n",
    "    \n",
    "    for single_audio_path in audios_path:\n",
    "        \n",
    "        num += 1\n",
    "        \n",
    "        if ((num >= from_ep) and (num < to_ep)):\n",
    "            \n",
    "            change_seconds = []\n",
    "\n",
    "            filename = (single_audio_path.split(\"/\")[-1]).split(\".\")[0]\n",
    "            print (filename)\n",
    "            \n",
    "            \n",
    "            if (feature_extractor == \"pyannote\"):\n",
    "                feature_vector = np.load(\"./pyannote-audio/tutorials/feature-extraction/AMI/\" + filename + \".Mix-Headset.npy\")\n",
    "\n",
    "            if (feature_extractor == \"yaafe\"):\n",
    "                feature_vector = np.load(\"./yaafe_ami_storage/\" + filename + \".Mix-Headset.npy\")\n",
    "\n",
    "                print (feature_vector.shape)\n",
    "\n",
    "            feature_array = np.ravel(feature_vector)\n",
    "\n",
    "            if (feature_vector is not None):\n",
    "\n",
    "                input_array.extend(feature_array)\n",
    "\n",
    "                print (single_audio_path + \" is done.\")\n",
    "\n",
    "                path_for_output = \"./txt_ami_full/\" + filename + \"_full_time.txt\"\n",
    "\n",
    "                with open(path_for_output) as f:\n",
    "                    content = f.readlines()\n",
    "\n",
    "\n",
    "                # need to open text file\n",
    "                # after that, point the end point of speaker\n",
    "                # add 1 to point of speaker, add 0 otherwise\n",
    "                # time is in second format at the txt file\n",
    "\n",
    "                content = [x.strip() for x in content] \n",
    "\n",
    "                for single_change in content:\n",
    "\n",
    "                    change_seconds.append(single_change)\n",
    "\n",
    "                output_array = np.zeros(feature_vector.shape[0])\n",
    "\n",
    "                for single_change in change_seconds:\n",
    "\n",
    "                    single_change_ms = float(single_change)*1000\n",
    "\n",
    "                    which_start_hop = (single_change_ms-win_len)/hop # now we know, milisecond version of change\n",
    "                                                # which is located after which_hop paramater\n",
    "                                                # add 2 and round to up\n",
    "\n",
    "                    which_end_hop = single_change_ms/hop # round to up\n",
    "\n",
    "                    start_location = math.ceil(which_start_hop + 1)\n",
    "                    end_location = math.ceil(which_end_hop)\n",
    "\n",
    "                    if (boost_for_imbalance==False):\n",
    "\n",
    "                        output_array[start_location:end_location+1] = 1.0\n",
    "\n",
    "                    else:\n",
    "\n",
    "                        if (fuzzy_label==False):\n",
    "                            output_array[start_location-how_much_boost:end_location+1+how_much_boost] = 1.0\n",
    "                        else:\n",
    "                            output_array[start_location:end_location+1] = 1.0\n",
    "                            for ix_label in range(1, how_much_boost):\n",
    "                                output_array[start_location-ix_label] = 1.0 - (float(ix_label)/how_much_boost)\n",
    "                                output_array[end_location+1+ix_label] = 1.0 - (float(ix_label)/how_much_boost)\n",
    "\n",
    "                output.extend(output_array)\n",
    "\n",
    "    input_array = np.reshape(input_array, (-1, 59))\n",
    "        \n",
    "    output_array = np.asarray(output)\n",
    "    output_array = np.expand_dims(output_array, axis=1)\n",
    "    \n",
    "    if (balance == True):\n",
    "            loc_zeros = np.where(output_array == 0)[0]\n",
    "            list_cons = [list(group) for group in mit.consecutive_groups(loc_zeros)]\n",
    "            \n",
    "            for single_list_con in list_cons:\n",
    "                if (len(single_list_con) > 80):\n",
    "                    first_zero_loc = single_list_con[0]\n",
    "                    last_zero_loc = single_list_con[-1]\n",
    "                    output_all_array[first_zero_loc+20:last_zero_loc-20] = 2\n",
    "                    \n",
    "            loc_twos = np.where(output_array == 2)[0]\n",
    "            list_cons = [list(group) for group in mit.consecutive_groups(loc_twos)]\n",
    "            \n",
    "            output_array = output_array.squeeze(axis=1)\n",
    "            input_array = input_array[output_array != 2, :]\n",
    "            \n",
    "            output_array = output_array[output_all_array != 2]   \n",
    "            output_array = np.expand_dims(output_array, axis=1)\n",
    "            \n",
    "    \n",
    "    print(\"inputs shape: \", input_array.shape)\n",
    "\n",
    "    print(\"outputs shape: \", output_array.shape)\n",
    "\n",
    "    return (input_array, output_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------\n",
    "\n",
    "## Build Architectures and Train Them\n",
    "\n",
    "**2D Convolution and BiLSTM Combination**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", message=\"numpy.dtype size changed\")\n",
    "import keras\n",
    "\n",
    "from keras import layers\n",
    "from keras import models\n",
    "from keras import optimizers\n",
    "from keras.models import Model\n",
    "import tensorflow as tf\n",
    "from keras.utils.generic_utils import get_custom_objects\n",
    "from keras.models import load_model\n",
    "import time\n",
    "\n",
    "frame_shape = (800, 59, 1)\n",
    "\n",
    "## Network Architecture\n",
    "# https://keras.io/getting-started/sequential-model-guide/\n",
    "\n",
    "input_frame = keras.Input(frame_shape, name='main_input')\n",
    "\n",
    "conv1 = layers.Conv2D(800, 9, strides=(1,1), padding=\"same\")(input_frame)\n",
    "conv1 = layers.LeakyReLU()(conv1)\n",
    "conv1_BN = layers.BatchNormalization()(conv1)\n",
    "conv1_drop = layers.Dropout(0.5)(conv1_BN)\n",
    "\n",
    "xx = layers.TimeDistributed(layers.Flatten())(conv1_drop)\n",
    "\n",
    "bidirectional_1 = layers.Bidirectional(layers.LSTM(128, return_sequences=True, kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(xx)\n",
    "bidirectional_1 = layers.LeakyReLU()(bidirectional_1)\n",
    "bidirectional_1_BN = layers.BatchNormalization()(bidirectional_1)\n",
    "bidirectional_1_drop = layers.Dropout(0.5)(bidirectional_1_BN)\n",
    "\n",
    "bidirectional_2 = layers.Bidirectional(layers.LSTM(64, return_sequences=True,  kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(bidirectional_1_drop)\n",
    "bidirectional_2 = layers.LeakyReLU()(bidirectional_2)\n",
    "bidirectional_2_BN = layers.BatchNormalization()(bidirectional_2)\n",
    "bidirectional_2_drop = layers.Dropout(0.5)(bidirectional_2_BN)\n",
    "\n",
    "bidirectional_3 = layers.Bidirectional(layers.LSTM(36, return_sequences=True,  kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(bidirectional_2_drop)\n",
    "bidirectional_3 = layers.LeakyReLU()(bidirectional_3)\n",
    "bidirectional_3_BN = layers.BatchNormalization()(bidirectional_3)\n",
    "bidirectional_3_drop = layers.Dropout(0.5)(bidirectional_3_BN)\n",
    "\n",
    "\n",
    "tdistributed_1 = layers.TimeDistributed(layers.Dense(40, kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(bidirectional_3_drop)\n",
    "tdistributed_1 = layers.LeakyReLU()(tdistributed_1)\n",
    "tdistributed_1_BN = layers.BatchNormalization()(tdistributed_1)\n",
    "tdistributed_1_drop = layers.Dropout(0.5)(tdistributed_1_BN)\n",
    "\n",
    "tdistributed_2 = layers.TimeDistributed(layers.Dense(10, kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(tdistributed_1_drop)\n",
    "tdistributed_2 = layers.LeakyReLU()(tdistributed_2)\n",
    "tdistributed_2_BN = layers.BatchNormalization()(tdistributed_2)\n",
    "tdistributed_2_drop = layers.Dropout(0.5)(tdistributed_2_BN)\n",
    "\n",
    "\n",
    "tdistributed_3 = layers.TimeDistributed(layers.Dense(1, activation='sigmoid', kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(tdistributed_2_drop)\n",
    "\n",
    "\n",
    "# WE DO NOT NEED IT FOR TRAINING. SO DISCARD.\n",
    "## Source: https://stackoverflow.com/questions/37743574/hard-limiting-threshold-activation-function-in-tensorflow\n",
    "def step_activation(x):\n",
    "    threshold = 0.4\n",
    "    cond = tf.less(x, tf.fill(value=threshold, dims=tf.shape(x)))\n",
    "    out = tf.where(cond, tf.zeros(tf.shape(x)), tf.ones(tf.shape(x)))\n",
    "\n",
    "    return out\n",
    "\n",
    "# https://stackoverflow.com/questions/47034692/keras-set-output-of-intermediate-layer-to-0-or-1-based-on-threshold\n",
    "\n",
    "step_activation = layers.Dense(1, activation=step_activation, name='threshold_activation')(tdistributed_3)\n",
    "\n",
    "\n",
    "\n",
    "model = Model(input_frame, tdistributed_3)\n",
    "\n",
    "rmsprop = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=0.0001, decay=0.00001)\n",
    "\n",
    "Nadam = keras.optimizers.Nadam(lr=0.005, beta_1=0.9, beta_2=0.999, epsilon=0.00001, schedule_decay=0.0004)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer=\"Nadam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "how_many_step = 40\n",
    "how_many_repeat = 30\n",
    "\n",
    "ix_repeat = 0\n",
    "\n",
    "\n",
    "while (ix_repeat < how_many_repeat):\n",
    "    ix_repeat += 0\n",
    "    \n",
    "    print (\"REPEAT:\", ix_repeat)\n",
    "    ix_step = 0\n",
    "    from_epi = 1\n",
    "    \n",
    "    while (ix_step < how_many_step):\n",
    "        ix_step += 1\n",
    "\n",
    "        print (\"STEP:\", ix_step)\n",
    "        \n",
    "        #print(\"relax\")\n",
    "        #time.sleep(2.5) \n",
    "\n",
    "        input_array, output_array = create_data_for_supervised (\"./amicorpus/*/audio/\", 10, 25, from_epi, \n",
    "                                                                from_epi+3, True, 25, False, \"yaafe\" , True)\n",
    "        \n",
    "        print (np.mean(output_array))\n",
    "        max_len = 800 # how many frame will be taken\n",
    "        step = 800 # step size.\n",
    "\n",
    "        input_array_specified = []\n",
    "        output_array_specified = []\n",
    "\n",
    "        for i in range (0, input_array.shape[0]-max_len, step):\n",
    "            single_input_specified = (input_array[i:i+max_len,:])\n",
    "            single_output_specified = (output_array[i:i+max_len,:])\n",
    "\n",
    "            input_array_specified.append(single_input_specified)\n",
    "            output_array_specified.append(single_output_specified)\n",
    "\n",
    "        output_array_specified = np.asarray(output_array_specified)\n",
    "        input_array_specified = np.asarray(input_array_specified)\n",
    "\n",
    "        try:\n",
    "            input_array_specified = np.expand_dims(input_array_specified, axis=4)\n",
    "            model.fit(input_array_specified, output_array_specified,\n",
    "                   epochs=2,\n",
    "                   batch_size=16,\n",
    "                   shuffle=True)\n",
    "\n",
    "        except:\n",
    "            print (\"Pass this epoch.\")\n",
    "            pass\n",
    "\n",
    "\n",
    "        model.save_weights('bilstm_weights_2DCNN.h5')    \n",
    "\n",
    "        input_array = []\n",
    "        output_array = []\n",
    "\n",
    "        from_epi += 3\n",
    "    model.save_weights(\"bilstm_weights_2DCNN\" + str(ix_repeat) + \".h5\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BiLSTM**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", message=\"numpy.dtype size changed\")\n",
    "\n",
    "from keras import layers\n",
    "from keras import models\n",
    "from keras import optimizers\n",
    "import keras\n",
    "from keras.models import load_model\n",
    "import time\n",
    "from keras.models import Model\n",
    "import tensorflow as tf\n",
    "from keras.utils.generic_utils import get_custom_objects\n",
    "\n",
    "\n",
    "frame_shape = (800, 59)\n",
    "\n",
    "## Network Architecture\n",
    "\n",
    "input_frame = keras.Input(frame_shape, name='main_input')\n",
    "\n",
    "bidirectional_1 = layers.Bidirectional(layers.LSTM(64, activation=\"tanh\", return_sequences=True))(input_frame)\n",
    "bidirectional_2 = layers.Bidirectional(layers.LSTM(32, activation='tanh', return_sequences=True))(bidirectional_1)\n",
    "\n",
    "tdistributed_1 = layers.TimeDistributed(layers.Dense(40, activation='tanh'))(bidirectional_2)\n",
    "tdistributed_2 = layers.TimeDistributed(layers.Dense(10, activation='tanh'))(tdistributed_1)\n",
    "tdistributed_3 = layers.TimeDistributed(layers.Dense(1, activation='sigmoid'))(tdistributed_2)\n",
    "\n",
    "\n",
    "# WE DO NOT NEED IT FOR TRAINING. SO DISCARD.\n",
    "## Source: https://stackoverflow.com/questions/37743574/hard-limiting-threshold-activation-function-in-tensorflow\n",
    "def step_activation(x):\n",
    "    threshold = 0.4\n",
    "    cond = tf.less(x, tf.fill(value=threshold, dims=tf.shape(x)))\n",
    "    out = tf.where(cond, tf.zeros(tf.shape(x)), tf.ones(tf.shape(x)))\n",
    "\n",
    "    return out\n",
    "\n",
    "# https://stackoverflow.com/questions/47034692/keras-set-output-of-intermediate-layer-to-0-or-1-based-on-threshold\n",
    "\n",
    "step_activation = layers.Dense(1, activation=step_activation, name='threshold_activation')(tdistributed_3)\n",
    "\n",
    "\n",
    "\n",
    "model = Model(input_frame, tdistributed_3)\n",
    "\n",
    "rmsprop = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=0.0001, decay=0.00001)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer=\"rmsprop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "how_many_step = 40\n",
    "how_many_repeat = 30\n",
    "\n",
    "ix_repeat = 0\n",
    "\n",
    "\n",
    "while (ix_repeat < how_many_repeat):\n",
    "    ix_repeat += 0\n",
    "    \n",
    "    print (\"REPEAT:\", ix_repeat)\n",
    "    ix_step = 0\n",
    "    from_epi = 1\n",
    "    \n",
    "    while (ix_step < how_many_step):\n",
    "        ix_step += 1\n",
    "\n",
    "        print (\"STEP:\", ix_step)\n",
    "        \n",
    "        #print(\"relax\")\n",
    "        #time.sleep(2.5) \n",
    "\n",
    "        input_array, output_array = create_data_for_supervised (\"./amicorpus/*/audio/\", 10, 25, from_epi, \n",
    "                                                                from_epi+3, True, 40, False, \"yaafe\" , True)\n",
    "        \n",
    "        print (np.mean(output_array))\n",
    "        max_len = 800 # how many frame will be taken\n",
    "        step = 800 # step size.\n",
    "\n",
    "        input_array_specified = []\n",
    "        output_array_specified = []\n",
    "\n",
    "        for i in range (0, input_array.shape[0]-max_len, step):\n",
    "            single_input_specified = (input_array[i:i+max_len,:])\n",
    "            single_output_specified = (output_array[i:i+max_len,:])\n",
    "\n",
    "            input_array_specified.append(single_input_specified)\n",
    "            output_array_specified.append(single_output_specified)\n",
    "\n",
    "        output_array_specified = np.asarray(output_array_specified)\n",
    "        input_array_specified = np.asarray(input_array_specified)\n",
    "\n",
    "        try:\n",
    "\n",
    "            model.fit(input_array_specified, output_array_specified,\n",
    "                   epochs=2,\n",
    "                   batch_size=16,\n",
    "                   shuffle=True)\n",
    "\n",
    "        except:\n",
    "            print (\"Pass this epoch.\")\n",
    "            pass\n",
    "\n",
    "        # https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model\n",
    "\n",
    "        model.save_weights('bilstm_weights_yaafe.h5')    \n",
    "\n",
    "        input_array = []\n",
    "        output_array = []\n",
    "\n",
    "        from_epi += 3\n",
    "    model.save_weights(\"bilstm_weights_yaafe\" + str(ix_repeat) + \".h5\")    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1D Convolution combined with BiLSTM**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "import keras\n",
    "warnings.filterwarnings(\"ignore\", message=\"numpy.dtype size changed\")\n",
    "\n",
    "from keras import layers\n",
    "from keras import models\n",
    "from keras import optimizers\n",
    "from keras.models import Model\n",
    "import tensorflow as tf\n",
    "from keras.utils.generic_utils import get_custom_objects\n",
    "from keras.models import load_model\n",
    "import time\n",
    "\n",
    "frame_shape = (800, 59)\n",
    "\n",
    "## Network Architecture\n",
    "# https://keras.io/getting-started/sequential-model-guide/\n",
    "\n",
    "input_frame = keras.Input(frame_shape, name='main_input')\n",
    "\n",
    "conv1 = layers.Conv1D(800, 9, padding=\"same\")(input_frame)\n",
    "conv1 = layers.LeakyReLU()(conv1)\n",
    "conv1_BN = layers.BatchNormalization()(conv1)\n",
    "conv1_drop = layers.Dropout(0.5)(conv1_BN)\n",
    "\n",
    "#conv2 = layers.Conv1D(800, 9, padding=\"same\")(conv1_drop)\n",
    "#conv2 = layers.LeakyReLU()(conv2)\n",
    "#conv2_BN = layers.BatchNormalization()(conv2)\n",
    "#conv2_drop = layers.Dropout(0.3)(conv2_BN)\n",
    "\n",
    "\n",
    "bidirectional_1 = layers.Bidirectional(layers.LSTM(128, return_sequences=True, kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(conv1_drop)\n",
    "bidirectional_1 = layers.LeakyReLU()(bidirectional_1)\n",
    "bidirectional_1_BN = layers.BatchNormalization()(bidirectional_1)\n",
    "bidirectional_1_drop = layers.Dropout(0.5)(bidirectional_1_BN)\n",
    "\n",
    "bidirectional_2 = layers.Bidirectional(layers.LSTM(64, return_sequences=True,  kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(bidirectional_1_drop)\n",
    "bidirectional_2 = layers.LeakyReLU()(bidirectional_2)\n",
    "bidirectional_2_BN = layers.BatchNormalization()(bidirectional_2)\n",
    "bidirectional_2_drop = layers.Dropout(0.5)(bidirectional_2_BN)\n",
    "\n",
    "bidirectional_3 = layers.Bidirectional(layers.LSTM(36, return_sequences=True,  kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(bidirectional_2_drop)\n",
    "bidirectional_3 = layers.LeakyReLU()(bidirectional_3)\n",
    "bidirectional_3_BN = layers.BatchNormalization()(bidirectional_3)\n",
    "bidirectional_3_drop = layers.Dropout(0.5)(bidirectional_3_BN)\n",
    "\n",
    "\n",
    "tdistributed_1 = layers.TimeDistributed(layers.Dense(40, kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(bidirectional_3_drop)\n",
    "tdistributed_1 = layers.LeakyReLU()(tdistributed_1)\n",
    "tdistributed_1_BN = layers.BatchNormalization()(tdistributed_1)\n",
    "tdistributed_1_drop = layers.Dropout(0.5)(tdistributed_1_BN)\n",
    "\n",
    "tdistributed_2 = layers.TimeDistributed(layers.Dense(10, kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(tdistributed_1_drop)\n",
    "tdistributed_2 = layers.LeakyReLU()(tdistributed_2)\n",
    "tdistributed_2_BN = layers.BatchNormalization()(tdistributed_2)\n",
    "tdistributed_2_drop = layers.Dropout(0.5)(tdistributed_2_BN)\n",
    "\n",
    "\n",
    "tdistributed_3 = layers.TimeDistributed(layers.Dense(1, activation='sigmoid', kernel_initializer='random_normal',\n",
    "                bias_initializer='zeros'))(tdistributed_2_drop)\n",
    "\n",
    "\n",
    "# WE DO NOT NEED IT FOR TRAINING. SO DISCARD.\n",
    "## Source: https://stackoverflow.com/questions/37743574/hard-limiting-threshold-activation-function-in-tensorflow\n",
    "def step_activation(x):\n",
    "    threshold = 0.4\n",
    "    cond = tf.less(x, tf.fill(value=threshold, dims=tf.shape(x)))\n",
    "    out = tf.where(cond, tf.zeros(tf.shape(x)), tf.ones(tf.shape(x)))\n",
    "\n",
    "    return out\n",
    "\n",
    "# https://stackoverflow.com/questions/47034692/keras-set-output-of-intermediate-layer-to-0-or-1-based-on-threshold\n",
    "\n",
    "step_activation = layers.Dense(1, activation=step_activation, name='threshold_activation')(tdistributed_3)\n",
    "\n",
    "\n",
    "\n",
    "model = Model(input_frame, tdistributed_3)\n",
    "\n",
    "rmsprop = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=0.0001, decay=0.00001)\n",
    "\n",
    "Nadam = keras.optimizers.Nadam(lr=0.005, beta_1=0.9, beta_2=0.999, epsilon=0.00001, schedule_decay=0.0004)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer=\"Nadam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_weights(\"bilstm_cnn_yaafe8.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "how_many_step = 60\n",
    "how_many_repeat = 15\n",
    "\n",
    "ix_repeat = 0\n",
    "\n",
    "\n",
    "while (ix_repeat < how_many_repeat):\n",
    "    ix_repeat += 1\n",
    "    \n",
    "    print (\"REPEAT:\", ix_repeat)\n",
    "    ix_step = 0\n",
    "    from_epi = 2\n",
    "    \n",
    "    while (ix_step < how_many_step):\n",
    "        ix_step += 1\n",
    "\n",
    "        print (\"STEP:\", ix_step)\n",
    "        \n",
    "        #print(\"relax\")\n",
    "        #time.sleep(2.5) \n",
    "        try:\n",
    "            input_array, output_array = create_data_for_supervised (\"./amicorpus/*/audio/\", 10, 25, \n",
    "                                                                    from_epi, from_epi+5, True, 12, True, \"yaafe\" , True)\n",
    "\n",
    "            print (np.mean(output_array))\n",
    "\n",
    "            max_len = 800 # how many frame will be taken\n",
    "            step = 800 # step size.\n",
    "\n",
    "            input_array_specified = []\n",
    "            output_array_specified = []\n",
    "\n",
    "            for i in range (0, input_array.shape[0]-max_len, step):\n",
    "                single_input_specified = (input_array[i:i+max_len,:])\n",
    "                single_output_specified = (output_array[i:i+max_len,:])\n",
    "\n",
    "                input_array_specified.append(single_input_specified)\n",
    "                output_array_specified.append(single_output_specified)\n",
    "\n",
    "            output_array_specified = np.asarray(output_array_specified)\n",
    "            input_array_specified = np.asarray(input_array_specified)\n",
    "\n",
    "\n",
    "            try:\n",
    "                model.fit(input_array_specified, output_array_specified,\n",
    "                   epochs=5,\n",
    "                   batch_size=8,\n",
    "                   validation_split = 0.2,\n",
    "                   shuffle=False)\n",
    "\n",
    "            except ValueError :\n",
    "                print (\"Pass this epoch.\")\n",
    "                pass\n",
    "            \n",
    "            model.save_weights('bilstm_cnn_yaafe.h5')    \n",
    "\n",
    "            input_array = []\n",
    "            output_array = []\n",
    "        except IndexError:\n",
    "            print (\"Index Error. Pass this.\")\n",
    "            pass\n",
    "\n",
    "        from_epi += 5\n",
    "    model.save_weights(\"bilstm_cnn_yaafe\" + str(ix_repeat) + \".h5\")    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------\n",
    "\n",
    "**With trained models, we will create prediction.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To get prediction, we need to give k, 800, 59 array to system.\n",
    "# Our output is like k, 320, 1\n",
    "# We need to convert it into milisecond version\n",
    "\n",
    "import more_itertools as mit\n",
    "\n",
    "def create_prediction(filename, hop, win_len, threshold, lstm_system, featureplan, sr, overlapping = False, feature_extractor=\"yaafe\"):\n",
    "    \n",
    "    \"\"\"\"It takes audio file and create prediction via lstm system. If output exceeds\n",
    "    threshold, we will say there is speaker change.\n",
    "    \n",
    "    Arguments:\n",
    "    filename= Which file will be considered.\n",
    "    hop: Hop length (we need it for Librosa)\n",
    "    win_len: Window length (we need it for Librosa)\n",
    "    threshold: If prediction exceed this value, we will say there is speaker change\n",
    "    lstm_system: System will create prediction\n",
    "    featureplan: Which txt will be used for yaafe feature extraction.\n",
    "    sr: Sample rate of audio input.\n",
    "    \n",
    "    Outputs:\n",
    "    prediction_array: It stores prediction value for each frame\n",
    "    prediction_array_rav: Ravel version of prediction array. We will use it.\n",
    "    prediction_array_ms = It stores which milisecond we have speaker change point.\n",
    "    \"\"\"\n",
    "    \n",
    "    prediction_vector = []\n",
    "    \n",
    "    if (feature_extractor==\"pyannote\"):\n",
    "        feature_vector = np.load(\"./pyannote-audio/tutorials/feature-extraction/AMI/\" + filename + \".npy\")\n",
    "\n",
    "    if (feature_extractor==\"yaafe\"):\n",
    "        \n",
    "        try:\n",
    "            feature_vector = np.load(\"./yaafe_ami_storage/\" + filename + \".npy\")\n",
    "        except: \n",
    "            feature_vector = create_numpy_for_audio(audio_file=\"rad_bremen_media/\" + filename + \".wav\", \n",
    "                                                            feature_extractor=\"yaafe\", \n",
    "                                                            hop=hop, win_len=win_len, \n",
    "                                                            featureplan=featureplan, sr=sr)\n",
    "    ix_frame = 0\n",
    "    \n",
    "    if (overlapping):\n",
    "        \n",
    "        while ((ix_frame+799)<feature_vector.shape[0]):        \n",
    "           \n",
    "            prediction = lstm_system.predict(np.expand_dims(feature_vector[ix_frame:ix_frame+800], axis=0))\n",
    "            prediction = prediction.squeeze(axis=2)\n",
    "            prediction = prediction.squeeze(axis=0)\n",
    "\n",
    "            prediction_vector.append(prediction)\n",
    "            \n",
    "            ix_frame += 200\n",
    "            \n",
    "        prediction_vector = np.asarray(prediction_vector)\n",
    "        prediction_array = np.ravel(prediction_vector)\n",
    "\n",
    "\n",
    "        prediction_array_sec = []\n",
    "        prediction_array_msec = []\n",
    "        prediction_array_average = []\n",
    "        \n",
    "        ix_frame_pred = 0\n",
    "\n",
    "        total_prediction = len(prediction_array)\n",
    "        \n",
    "        print (total_prediction)\n",
    "\n",
    "        prediction_array_average[0:200] = prediction_array[0:200]\n",
    "        prediction_array_average[200:400] = (prediction_array[200:400]+prediction_array[800:1000]) * 0.5\n",
    "        prediction_array_average[400:600] = (prediction_array[400:600]+prediction_array[1000:1200]+\n",
    "                                             prediction_array[1600:1800]) * 0.33\n",
    "        \n",
    "        ix_frame = 600\n",
    "        count = 0\n",
    "        \n",
    "        while ((ix_frame+798)<total_prediction):        \n",
    "\n",
    "            next_frame = ix_frame + (count * 600) \n",
    "            try:\n",
    "                prediction_array_average[ix_frame:ix_frame+200] = (prediction_array[next_frame:next_frame+200]+\n",
    "                                                                   prediction_array[next_frame+600:next_frame+800]+\n",
    "                                                                   prediction_array[next_frame+1200:next_frame+1400]+\n",
    "                                                                   prediction_array[next_frame+1800:next_frame+2000]) * 0.25\n",
    "            except:\n",
    "                pass\n",
    "            ix_frame += 200\n",
    "            count += 1\n",
    "\n",
    "\n",
    "        prediction_array = np.asarray(prediction_array_average)\n",
    "\n",
    "        for pred in prediction_array:\n",
    "\n",
    "            if (pred > threshold):\n",
    "                ms_version = float(win_len + (ix_frame_pred * hop)) # milisecond version to represent end point of first embed            \n",
    "                prediction_array_msec.append(int(ms_version))\n",
    "                prediction_array_sec.append(ms_version/1000)\n",
    "\n",
    "            ix_frame_pred += 1\n",
    "\n",
    "\n",
    "        prediction_array_smooth = []\n",
    "        for pred in prediction_array_msec:\n",
    "            if (pred-hop not in prediction_array_msec):\n",
    "                prediction_array_smooth.append(pred*0.001)\n",
    "\n",
    "\n",
    "        prediction_array_tenth_ms = np.asarray(prediction_array_msec)/10\n",
    "\n",
    "        list_cons = [list(group) for group in mit.consecutive_groups(prediction_array_tenth_ms)]\n",
    "\n",
    "        prediction_array_msec_smooth = []\n",
    "\n",
    "        for single_list_cons in list_cons:\n",
    "            prediction_array_msec_smooth.append(np.mean(prediction_array_msec_smooth)*0.01)\n",
    "\n",
    "        prediction_array_msec_smooth = np.asarray(prediction_array_msec_smooth)\n",
    "\n",
    "        which_turn = 0\n",
    "\n",
    "        for single_mean_s in prediction_array_msec_smooth:\n",
    "            which_turn += 1\n",
    "\n",
    "            try:\n",
    "                start_time = float(prediction_array_msec_smooth[which_turn-1])\n",
    "                end_time = float(prediction_array_msec_smooth[which_turn])\n",
    "\n",
    "                if ((start_time+0.5) > end_time):\n",
    "                    prediction_array_msec_smooth[which_turn] = ((start_time+end_time) / 2)\n",
    "                    prediction_array_msec_smooth = np.delete(mean_s, which_turn-1)\n",
    "                    which_turn -= 1\n",
    "            except:\n",
    "                pass\n",
    "        \n",
    "    else:\n",
    "        while (ix_frame+799<matrix_of_single_audio.shape[0]):        \n",
    "        \n",
    "            prediction = lstm_system.predict(np.expand_dims(prediction_vector[ix_frame:ix_frame+800], axis=0))\n",
    "            prediction = prediction.squeeze(axis=2)\n",
    "            prediction = prediction.squeeze(axis=0)\n",
    "\n",
    "            prediction_vector.append(prediction)\n",
    "            ix_frame += 800\n",
    "        \n",
    "        prediction_vector = np.asarray(prediction_vector)\n",
    "        print (prediction_vector.shape)\n",
    "\n",
    "        prediction_array = np.ravel(prediction_vector)\n",
    "\n",
    "\n",
    "        prediction_array_sec = []\n",
    "        prediction_array_msec = []\n",
    "        ix_frame_pred = 0\n",
    "\n",
    "        for pred in prediction_array:\n",
    "            if (pred > threshold):\n",
    "                ms_version = float(win_len + (ix_frame_pred * hop)) # milisecond version to represent end point of first embed            \n",
    "                prediction_array_msec.append(int(ms_version))\n",
    "                prediction_array_sec.append(ms_version/1000)\n",
    "\n",
    "            ix_frame_pred += 1\n",
    "\n",
    "        prediction_array_tenth_ms = np.asarray(prediction_array_msec)/10\n",
    "\n",
    "        list_cons = [list(group) for group in mit.consecutive_groups(prediction_array_tenth_ms)]\n",
    "\n",
    "        prediction_array_msec_smooth = []\n",
    "\n",
    "        for single_list_cons in list_cons:\n",
    "            prediction_array_msec_smooth.append(np.mean(single_list_cons)*0.01)\n",
    "\n",
    "        prediction_array_msec_smooth = np.asarray(prediction_array_msec_smooth)\n",
    "\n",
    "        which_turn = 0\n",
    "\n",
    "        for single_mean_s in prediction_array_msec_smooth:\n",
    "            which_turn += 1\n",
    "\n",
    "            try:\n",
    "                start_time = float(prediction_array_msec_smooth[which_turn-1])\n",
    "                end_time = float(prediction_array_msec_smooth[which_turn])\n",
    "\n",
    "                if ((start_time+0.5) > end_time):\n",
    "                    mean_s[which_turn] = ((start_time+end_time) / 2)\n",
    "                    mean_s = np.delete(mean_s, which_turn-1)\n",
    "                    which_turn -= 1\n",
    "                    \n",
    "            except:\n",
    "                pass\n",
    "\n",
    "                \n",
    "    # https://codereview.stackexchange.com/questions/5196/grouping-consecutive-numbers-into-ranges-in-python-3-2\n",
    "\n",
    "    np.savetxt(fname=\"./prediction_txt/\" + filename + \"_prediction.txt\", \n",
    "               X=prediction_array_msec_smooth, \n",
    "               delimiter=' ', fmt='%1.3f')\n",
    "\n",
    "    return (prediction_array, prediction_array_msec_smooth)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def txt_file_to_matrix (filename, type_of_text, sr, featureplan, hop=10, win_len=25, feature_extractor=\"yaafe\"):\n",
    "    \n",
    "    \n",
    "    \"\"\"It takes the reference(ground truth) text file or prediction text file (they are in second version) and \n",
    "    return output array which represent the which frames has a speaker change point.\n",
    "    \n",
    "    Arguments:\n",
    "    filename: Which file will be considered.\n",
    "    type_of_text: Is it prediction or reference file.\n",
    "    sr: Sample rate.\n",
    "    featureplan: Which txt will be used for yaafe feature extraction.\n",
    "    hop: Hop length. (for Librosa.)\n",
    "    win_len: Window length. (for Librosa)\n",
    "    feature_extractor: Which feature extractor will be used. (Now, we have 2 option as\n",
    "        Pyannote or Yaafe.)\"\"\"\n",
    "    \n",
    "    \n",
    "    if (feature_extractor==\"pyannote\"):\n",
    "        feature_vector = np.load(\"./pyannote-audio/tutorials/feature-extraction/AMI/\" + filename + \".npy\")\n",
    "        \n",
    "    if (feature_extractor==\"yaafe\"):\n",
    "        try:\n",
    "            feature_vector = np.load(\"./yaafe_ami_storage/\" + filename + \".npy\")\n",
    "        except: \n",
    "            feature_vector = create_numpy_for_audio(audio_file=\"rad_bremen_media/\" + filename + \".wav\", \n",
    "                                                            feature_extractor=\"yaafe\", hop=hop, win_len=win_len, \n",
    "                                                            featureplan=featureplan, sr=sr)\n",
    "    \n",
    "    \n",
    "    if (type_of_text == \"reference\"):\n",
    "        path_for_txt = \"./txt_files/\" + filename.split(\".\")[0] + \"_full_time.txt\" \n",
    "    \n",
    "    \n",
    "    if (type_of_text == \"prediction\"):\n",
    "        path_for_txt = \"./prediction_txt/\" + filename + \"_prediction.txt\"\n",
    "        \n",
    "    change_seconds = []\n",
    "\n",
    "\n",
    "    with open(path_for_txt) as f:\n",
    "        content = f.readlines()\n",
    "        \n",
    "    content = [x.strip() for x in content] \n",
    "\n",
    "    for single_change in change_seconds:\n",
    "\n",
    "        change_seconds.append(single_change)\n",
    "\n",
    "    output_array = np.zeros(feature_vector.shape[0])\n",
    "\n",
    "    for single_change in change_seconds:\n",
    "        single_change_ms = float(single_change)*1000\n",
    "        which_start_hop = (single_change_ms-win_len)/hop # now we know, milisecond version of change\n",
    "                                    # which is located after which_hop paramater\n",
    "                                    # add 2 and round to up\n",
    "        which_end_hop = end_time_ms/hop # round to up\n",
    "\n",
    "        start_location = math.ceil(which_start_hop + 1)\n",
    "        end_location = math.ceil(which_end_hop)\n",
    "\n",
    "\n",
    "        output_array[start_location:end_location+1] = 1.0\n",
    "\n",
    "    return (output_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "**We need to evaluate our results**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as pp\n",
    "pp.rcParams['figure.figsize'] = (50.8, 10.0)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluation Metric\n",
    "from pyannote.metrics.diarization import DiarizationPurityCoverageFMeasure\n",
    "from pyannote.metrics.segmentation import SegmentationPurity\n",
    "from pyannote.metrics.segmentation import SegmentationCoverage\n",
    "from pyannote.metrics.diarization import DiarizationErrorRate\n",
    "from pyannote.database import get_protocol\n",
    "from pyannote.core import Segment, Timeline, Annotation\n",
    "\n",
    "\n",
    "metric = DiarizationPurityCoverageFMeasure()\n",
    "seg_purity = SegmentationPurity()\n",
    "seg_coverage = SegmentationCoverage()\n",
    "\n",
    "def complete_evaluation_single_file(filename, hop, win_len, thres, model, sr, featureplan, overlapping = False, show_graph_frame=True, \n",
    "                                    feature_extractor=\"yaafe\"):\n",
    "    \n",
    "    \"\"\"With this function, we can get different metrics to evaluate the system's performance.\n",
    "    \n",
    "    Arguments:\n",
    "    filename: Which file will be considered.\n",
    "    hop: Hop length.\n",
    "    win_len: Window length.\n",
    "    thres:\n",
    "    model: Which Deep Learning model will be used to predict.\n",
    "    sr: Sample rate of input.\n",
    "    featureplan:\n",
    "    overlapping: Will we use overlapping-averaging to predict output.\n",
    "    show_graph_frame: If it is true, we can see the prediction\n",
    "        and ground truth at the figure.\n",
    "    feature_extractor: Which feature extractor will be used. (currently, \n",
    "        Pyannote and Yaafe are available.)\"\"\"\n",
    "    \n",
    "    prediction_array, prediction_array_msec = create_prediction(filename, hop=hop, win_len=win_len, \n",
    "                                                                threshold = thres, lstm_system=model,\n",
    "                                                                feature_extractor=feature_extractor, sr=sr,\n",
    "                                                                featureplan=featureplan,\n",
    "                                                                overlapping=overlapping)\n",
    "    \n",
    "    ground_truth = txt_file_to_matrix(filename, \"reference\", sr, featureplan, \n",
    "                                      hop=hop, win_len=win_len, feature_extractor=feature_extractor)\n",
    "    \n",
    "    prediction_output_array = txt_file_to_matrix(filename, \"prediction\", sr, featureplan, \n",
    "                                                 hop=hop, win_len=win_len, feature_extractor=feature_extractor)\n",
    "    \n",
    "    path_for_reference = \"./txt_files/\" + filename.split(\".\")[0] + \"_full_time.txt\"\n",
    "   \n",
    "        \n",
    "    path_for_prediction = \"./prediction_txt/\" + filename + \"_prediction.txt\"\n",
    "   \n",
    "        \n",
    "    with open(path_for_reference) as f:\n",
    "        change_seconds_ref = f.readlines()\n",
    "        \n",
    "    change_seconds_ref = [x.strip() for x in change_seconds_ref] \n",
    "    \n",
    "    change_miliseconds_ref = [int(1000*float(i)) for i in change_seconds_ref]\n",
    "    \n",
    "    x_axis = np.arange(1, len(ground_truth)+1)\n",
    "    \n",
    "    if (show_graph_frame):\n",
    "    \n",
    "        pp.rcParams['figure.figsize'] = (50.8, 10.0)\n",
    "        pp.plot(prediction_array[0:20000])\n",
    "        pp.plot(x_axis[0:20000], ground_truth[0:20000], 'x', color='black');\n",
    "        pp.plot(x_axis[0:20000], 0.95*prediction_output_array[0:20000], '.', color=\"pink\");\n",
    "\n",
    "        pp.axhline(y=thres, color='r', linestyle='-')\n",
    "        pp.show()\n",
    "\n",
    "    hypothesis = Annotation()\n",
    "\n",
    "    which_turn = 0\n",
    "    \n",
    "    with open(path_for_prediction) as f:\n",
    "        change_seconds_pred = f.readlines()\n",
    "        \n",
    "    change_seconds_pred = [x.strip() for x in change_seconds_pred] \n",
    "        \n",
    "    for single_change_pred in change_seconds_pred:\n",
    "        try:\n",
    "            which_turn += 1\n",
    "            start_time = change_seconds_pred[which_turn-1]\n",
    "            end_time = change_seconds_pred[which_turn]\n",
    "            hypothesis[Segment(float(start_time), float(end_time))]=\"a\"\n",
    "        except:\n",
    "            pass\n",
    "   \n",
    "    change_seconds_pred = [x.strip() for x in change_seconds_pred] \n",
    "    change_miliseconds_pred = [int(1000*float(i)) for i in change_seconds_pred]\n",
    "    \n",
    "    \n",
    "    total_change_ref = len(change_miliseconds_ref)\n",
    "    \n",
    "    total_change_pred = len(change_miliseconds_pred)\n",
    "    \n",
    "    correct_guess = 0\n",
    "\n",
    "    for single_change_ref in change_miliseconds_ref:\n",
    "        single_guess_mem = False \n",
    "        for single_change_pred in change_miliseconds_pred:\n",
    "            if (single_change_pred in range(single_change_ref-500, single_change_ref+500)):\n",
    "                correct_guess += 1\n",
    "                break\n",
    "            \n",
    "    print (\"Correst Guess: \", correct_guess)    \n",
    "    print (\"Total End Time: \", total_end_time_num)    \n",
    "    print (\"Total Guess: \", total_guess_num)   \n",
    "\n",
    "    # Calculate precision, recall and f1 score.\n",
    "    precision = correct_guess / float(total_guess_num)\n",
    "    recall = correct_guess / float(total_end_time_num)\n",
    "    \n",
    "    try:\n",
    "        f1 =  2 * (precision*recall) / (precision+recall)\n",
    "    except ZeroDivisionError:\n",
    "        f1 = 0\n",
    "    \n",
    "    print (\"precison: \", precision)\n",
    "    print (\"recall: \", recall)\n",
    "    print (\"f1: \", f1)\n",
    "    \n",
    "    \n",
    "    purity = 0\n",
    "    coverage = 0\n",
    "    \n",
    "    try:\n",
    "        # If file in the Ami Corpus, we can get the result from\n",
    "        # pyannote.metrics.\n",
    "        \n",
    "        protocol = get_protocol('AMI.SpeakerDiarization.MixHeadset')\n",
    "\n",
    "        for i in protocol.train():\n",
    "            if (i[\"uri\"] == filename):\n",
    "                 reference = i['annotation']\n",
    "\n",
    "        for i in protocol.test():\n",
    "            if (i[\"uri\"] == filename):             \n",
    "                reference = i['annotation']\n",
    "\n",
    "        for i in protocol.development():\n",
    "            if (i[\"uri\"] == filename):             \n",
    "                reference = i['annotation']\n",
    "\n",
    "        metric = SegmentationPurity()\n",
    "        purity = metric(reference, hypothesis)\n",
    "        print (\"Segmentation Purity: \", purity)\n",
    "\n",
    "        metric = SegmentationCoverage()\n",
    "        coverage = metric(reference, hypothesis)\n",
    "        print (\"Segmentation Coverage: \", coverage)\n",
    "\n",
    "\n",
    "    except:\n",
    "        pass\n",
    "    \n",
    "    return (precision, recall, f1, purity, coverage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_evaluation_single_file(filename=\"TV_bubi6_2017-04-11_gek\", hop=10, win_len=25, thres = 0.25, \n",
    "                               model=model, feature_extractor=\"yaafe\",\n",
    "                               sr=16000, featureplan = \"featureplan_new.txt\",\n",
    "                               overlapping=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_evaluation_single_file(filename=\"TV_3nach9\", hop=10, win_len=25, thres = 0.3, \n",
    "                               model=model, feature_extractor=\"yaafe\",\n",
    "                               sr=16000, featureplan = \"featureplan_new.txt\",\n",
    "                               overlapping=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_evaluation_single_file(filename=\"TV_Wetter_2017-04-10\", hop=10, win_len=25, thres = 0.15, \n",
    "                               model=model, feature_extractor=\"yaafe\",\n",
    "                               sr=16000, featureplan = \"featureplan_new.txt\",\n",
    "                               overlapping=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_evaluation_single_file(filename=\"TS3003a.Mix-Headset\", hop=10, win_len=25, thres = 0.18, \n",
    "                               model=model, feature_extractor=\"yaafe\",\n",
    "                               sr=16000, featureplan = \"featureplan_new.txt\",\n",
    "                               overlapping=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complete_evaluation_single_file(filename=\"Wie_ist_nachhaltiger_Tourismus_moeglich_2017-09-16\", hop=10, win_len=25, \n",
    "                               thres = 0.1, model=model, feature_extractor=\"yaafe\",\n",
    "                               sr=16000, featureplan = \"featureplan_new.txt\",\n",
    "                               overlapping=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TO GET ALL RESULT FROM ALL CORPUS\n",
    "\n",
    "precision_array = []\n",
    "recall_array = []\n",
    "f1_array = []\n",
    "purity_array = []\n",
    "coverage_array = []\n",
    "\n",
    "corpus = os.listdir(\"/home/herdogan/Desktop/SpChangeDetect/df_media/\")\n",
    "\n",
    "for file in corpus:\n",
    "    try:\n",
    "        filename = file.split(\".\")[0]\n",
    "        print (filename)\n",
    "        \n",
    "        precision, recall, f1, _, _ = complete_evaluation_single_file(filename, hop=10, win_len=25, \n",
    "                                                                      feature_extractor=\"yaafe\", model=model, show_graph_frame=False, \n",
    "                                                                      thres=0.12, sr=16000, featureplan=\"featureplan_new.txt\", \n",
    "                                                                      overlapping=False)\n",
    "        precision_array.append(precision)\n",
    "        recall_array.append(recall)\n",
    "        f1_array.append(f1)\n",
    "        \n",
    "        \n",
    "        \n",
    "    except FileNotFoundError:\n",
    "        print (filename + \" is not in the directory\")\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TO GET ALL RESULT FROM ALL CORPUS\n",
    "\n",
    "precision_array = []\n",
    "recall_array = []\n",
    "f1_array = []\n",
    "purity_array = []\n",
    "coverage_array = []\n",
    "\n",
    "corpus = os.listdir(\"/home/herdogan/Desktop/SpChangeDetect/txt_ami_full/\")\n",
    "\n",
    "for file in corpus:\n",
    "    try:\n",
    "        filename = (file.split(\".\")[0]).split(\"_\")[0]\n",
    "        filename = filename + \".Mix-Headset\"\n",
    "        print (filename)\n",
    "        \n",
    "        precision, recall, f1, purity, coverage = complete_evaluation_single_file(filename, hop=10, win_len=25, \n",
    "                                                                                  feature_extractor=\"yaafe\", model=model, \n",
    "                                                                                  show_graph_frame=False, \n",
    "                                                                                  thres=0.168, sr=16000, \n",
    "                                                                                  featureplan=\"featureplan_new.txt\", overlapping=True)\n",
    "        precision_array.append(precision)\n",
    "        recall_array.append(recall)\n",
    "        f1_array.append(f1)\n",
    "        purity_array.append(purity)\n",
    "        coverage_array.append(coverage)\n",
    "        \n",
    "    except FileNotFoundError:\n",
    "        print (filename + \" is not in the directory\")\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = os.listdir(\"/home/herdogan/Desktop/SpChangeDetect/amicorpus/*/\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.mean(precision_array))\n",
    "print(np.mean(recall_array))\n",
    "print(np.mean(f1_array))\n",
    "print(np.mean(purity_array))\n",
    "print(np.mean(coverage_array))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd rad_bremen_media/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = os.listdir(\"/home/herdogan/Desktop/SpChangeDetect/rad_bremen_media/\")\n",
    "\n",
    "for file in corpus:\n",
    "    print ((file.split(\".\")[0]))\n",
    "    new_file = ((file.split(\".\")[0])) + \"RR.wav\" \n",
    "    !sox $file -r 16000 $new_file\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Subsequences with Label\n",
    "\n",
    "At that point, we should create training and test data with their label. Also, we can use directly [pyannote.metrics](https://github.com/pyannote/pyannote-metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep Learning Architecture\n",
    "\n",
    "We can directly upload the model's architecture from the .yml file which is provided by writer.\n",
    "\n",
    "However, I want to directly write all steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Author's .yml files\n",
    "\n",
    "!wget https://raw.githubusercontent.com/yinruiqing/change_detection/master/model/architecture.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load to model\n",
    "\n",
    "from keras.models import model_from_yaml\n",
    "yaml_file = open('architecture.yml', 'r')\n",
    "loaded_model_yaml = yaml_file.read()\n",
    "yaml_file.close()\n",
    "model = model_from_yaml(loaded_model_yaml)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rmsprop = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer=\"rmsprop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "print (keras.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import layers\n",
    "from keras import models\n",
    "from keras import optimizers\n",
    "import keras\n",
    "from keras.models import Model\n",
    "import tensorflow as tf\n",
    "from keras.layers.advanced_activations import *\n",
    "from keras.utils.generic_utils import get_custom_objects\n",
    "\n",
    "\n",
    "frame_shape = (320, 35)\n",
    "\n",
    "## Network Architecture\n",
    "\n",
    "input_frame = keras.Input(frame_shape, name='main_input')\n",
    "\n",
    "bidirectional_1 = layers.Bidirectional(layers.LSTM(32, return_sequences=True))(input_frame)\n",
    "bidirectional_2 = layers.Bidirectional(layers.LSTM(20, activation='tanh', return_sequences=True))(bidirectional_1)\n",
    "\n",
    "tdistributed_1 = layers.TimeDistributed(layers.Dense(40, activation='tanh'))(bidirectional_2)\n",
    "tdistributed_2 = layers.TimeDistributed(layers.Dense(10, activation='tanh'))(tdistributed_1)\n",
    "tdistributed_3 = layers.TimeDistributed(layers.Dense(1, activation='sigmoid'))(tdistributed_2)\n",
    "\n",
    "\n",
    "# WE DO NOT NEED IT FOR TRAINING. SO DISCARD.\n",
    "## Source: https://stackoverflow.com/questions/37743574/hard-limiting-threshold-activation-function-in-tensorflow\n",
    "def step_activation(x):\n",
    "    threshold = 0.4\n",
    "    cond = tf.less(x, tf.fill(value=threshold, dims=tf.shape(x)))\n",
    "    out = tf.where(cond, tf.zeros(tf.shape(x)), tf.ones(tf.shape(x)))\n",
    "\n",
    "    return out\n",
    "\n",
    "# https://stackoverflow.com/questions/47034692/keras-set-output-of-intermediate-layer-to-0-or-1-based-on-threshold\n",
    "\n",
    "step_activation = layers.Dense(1, activation=step_activation, name='threshold_activation')(tdistributed_3)\n",
    "\n",
    "\n",
    "\n",
    "model = Model(input_frame, tdistributed_3)\n",
    "\n",
    "rmsprop = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer=\"rmsprop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To save our model\n",
    "\n",
    "model_yaml = model.to_yaml()\n",
    "with open(\"model.yaml\", \"w\") as yaml_file:\n",
    "    yaml_file.write(model_yaml)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To look our model\n",
    "\n",
    "!cat model.yaml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}