Speaker2Vec_dev.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DRAFT VERSION.\n",
    "\n",
    "In this notebook, we are trying to reproduce [the paper](http://scuba.usc.edu/pdf/jati2017_Speaker2Vec.pdf).\n",
    "\n",
    "For review of the paper, you can look [here](https://hedonistrh.github.io/2018-07-09-Literature-Review-for-Speaker-Change-Detection/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction\n",
    "\n",
    "_\"We have adopted 40 dimensional high definition MFCC features extracted from 40 mel-spaced filters over a 25ms hamming window with a shift of 10ms using Kaldi toolkit\"_ However, I will use [Librosa](https://librosa.github.io).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data\n",
    "\n",
    "_\"We have used d = 100 frames (1s) for all training scenarios. This makes the size of input and output layers of the DNN models to be 4000.\"_\n",
    "\n",
    "_\"40 mel-spaced filters over a 25ms hamming window with a shift of 10ms\"_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir audio_files # this folder store audio_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use some data from Youtube. To download these files, [youtube-dl](https://rg3.github.io/youtube-dl/index.html) is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://askubuntu.com/questions/564567/how-to-download-playlist-from-youtube-dl\n",
    "# https://www.slashgeek.net/2016/06/24/5-youtube-dl-tips-might-not-know/\n",
    "!youtube-dl --extract-audio --audio-format m4a -o \"./audio_files/%(title)s.%(ext)s\" https://www.youtube.com/playlist?list="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** YAAFE Implementation **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open(\"featureplan_speaker2vec.txt\", \"w\")\n",
    "f.write(\"mfcc: MFCC blockSize=400 stepSize=160 CepsNbCoeffs=40 CepsIgnoreFirstCoeff=0\")\n",
    "f.close() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import genfromtxt\n",
    "import numpy as np\n",
    "import os\n",
    "import glob\n",
    "\n",
    "import librosa\n",
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "import math\n",
    "import sys\n",
    "    \n",
    "\n",
    "def create_numpy_for_audio(audio_file, featureplan = \"\",  feature_extractor= \"yaafe\", hop=10, win_len=25):\n",
    "    ## This function is based on YAAFE and librosa. Its arguments:\n",
    "    # audio_file: Path of audio file, it can be wav, mp3, ogg etc.\n",
    "    # featureplan: Text file which introduce which features will be extracted. (we need it for YAAFE\n",
    "    # feature_extractor: Which library will be used to extract features\n",
    "    # hop: Hop length (we need it for Librosa)\n",
    "    # win_len: Window length (we need it for Librosa)\n",
    "    \n",
    "    ## It will return 2D Array which is features of audio file.\n",
    "    \n",
    "    if (feature_extractor==\"yaafe\"):\n",
    "        !yaafe -c $featureplan -r 16000 $audio_file -p Precision=6 -p Metadata=False -n\n",
    "        filename = (audio_file.split(\"/\")[-1]).split(\".\")[0]\n",
    "\n",
    "        my_data = genfromtxt(audio_file + \".mfcc.csv\", delimiter=',')\n",
    "\n",
    "\n",
    "        # Previous codes creates csv file for features to load numpy array. After that, we can \n",
    "        # remove them.\n",
    "        os.remove(audio_file + \".mfcc.csv\")\n",
    "\n",
    "\n",
    "        return my_data\n",
    "    \n",
    "    if (feature_extractor==\"librosa\"):\n",
    "        audio, sr = librosa.load(filename)\n",
    "        # print (audio.shape, sr)\n",
    "        # https://github.com/librosa/librosa/issues/584\n",
    "        mfccs = librosa.feature.mfcc(audio, sr, n_mfcc=40, hop_length=int(float(hop/1000)*sr), n_fft=int(float(win_len/1000)*sr))\n",
    "        return mfccs\n",
    "\n",
    "\n",
    "def create_data_for_unsupervised(root_dir, featureplan, file_type=\"*mp3\", feature_extractor=\"yaafe\"):\n",
    "    ## This function is based YAAFE and Librosa. Its arguments:\n",
    "    # root_dir: Folder which stores audio files\n",
    "    # featureplan: Text file which introduce which features will be extracted. (we need it for YAAFE)\n",
    "    # file_type: Type of audio files. Default is mp3\n",
    "    # feature_extractor: Which library will be used to extract features\n",
    "    \n",
    "    all_audio_paths = glob.glob(os.path.join(root_dir, file_type))\n",
    "    matrix_of_audio_wav = []\n",
    "\n",
    "   \n",
    "    for single_audio_path in all_audio_paths:\n",
    "        print (single_audio_path)\n",
    "        matrix_of_single_audio = create_numpy_for_audio(featureplan, single_audio_path)\n",
    "        array_of_single_audio = np.ravel(matrix_of_single_audio)\n",
    "        \n",
    "        if (matrix_of_single_audio is not None):\n",
    "            print (matrix_of_single_audio.shape)\n",
    "            matrix_of_all_audio.extend(array_of_single_audio)\n",
    "            print (single_audio_path + \" is done.\")\n",
    "            \n",
    "    audio_array = np.asarray(matrix_of_all_audio)\n",
    "    audio_array = np.reshape(matrix_of_all_audio, (-1, 40))\n",
    "    input_array = []\n",
    "    output_array = []\n",
    "\n",
    "    print (audio_array.shape)\n",
    "    \n",
    "    start_point = 0\n",
    "    \n",
    "    while (start_point+200 < audio_array.shape[0]):\n",
    "        single_input = audio_array[start_point:start_point+100, 0:40]\n",
    "        single_output = audio_array[start_point+100:start_point+200, 0:40]\n",
    "        input_array.append(single_input)\n",
    "        output_array.append(single_output)\n",
    "        start_point += 100\n",
    "        \n",
    "        \n",
    "    input_array = np.asarray(input_array)\n",
    "    input_array = input_array.reshape((len(input_array), np.prod(input_array.shape[1:])))  \n",
    "    print(input_array.shape)\n",
    "\n",
    "    output_array = np.asarray(output_array)\n",
    "    output_array = output_array.reshape((len(output_array), np.prod(output_array.shape[1:])))  \n",
    "    print(output_array.shape)\n",
    "    \n",
    "    return (input_array, output_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = create_numpy_for_audio( \"./ES2009a.Mix-Headset.wav\", \"featureplan_speaker2vec.txt\",)\n",
    "import matplotlib.pyplot as pp\n",
    "%matplotlib inline\n",
    "\n",
    "pp.plot(a[200:600])\n",
    "pp.axhline(y=0.5, color='r', linestyle='-')\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_data_for_unsupervised(\"./audio_files/\", \"featureplan_speaker2vec.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "import sys\n",
    "\n",
    "def upload_npy_to_arrays(root_dir, from_ep = 0, to_ep=0):\n",
    "    ## If we have stored numpy arrays, we can just load into \n",
    "    # numpy array via this function. Its arguments:\n",
    "    # root_dir: Folder which stores numpy arrays\n",
    "    # from_ep: Location of first file which will be loaded into array\n",
    "    # to_ep: Location of last file which will be loaded into array\n",
    "    \n",
    "    all_npy_paths = glob.glob(os.path.join(root_dir, '*npy'))\n",
    "    matrix_of_all_audio = []\n",
    "\n",
    "    num = 0\n",
    "    for single_npy_path in all_npy_paths:\n",
    "        num += 1\n",
    "        if (num >= from_ep):\n",
    "            array_of_single_audio = np.load(single_npy_path)\n",
    "            \n",
    "            if (num > to_ep):\n",
    "                break\n",
    "                    \n",
    "            else:\n",
    "                if (array_of_single_audio is not None):\n",
    "                     matrix_of_all_audio.extend(array_of_single_audio)\n",
    "            \n",
    "    audio_array = np.asarray(matrix_of_all_audio)\n",
    "    audio_array = np.reshape(matrix_of_all_audio, (-1,40))\n",
    "    input_array = []\n",
    "    output_array = []\n",
    "\n",
    "    print (audio_array.shape)\n",
    "    \n",
    "    start_point = 0\n",
    "    \n",
    "    while (start_point+200 < audio_array.shape[0]):\n",
    "        single_input = audio_array[start_point:start_point+100, 0:40]\n",
    "        single_output = audio_array[start_point+100:start_point+200, 0:40]\n",
    "        input_array.append(single_input)\n",
    "        output_array.append(single_output)\n",
    "        start_point += 100\n",
    "        \n",
    "        \n",
    "        \n",
    "    input_array = np.asarray(input_array)\n",
    "    input_array = input_array.reshape((len(input_array), np.prod(input_array.shape[1:])))  \n",
    "    print(input_array.shape)\n",
    "\n",
    "    output_array = np.asarray(output_array)\n",
    "    output_array = output_array.reshape((len(output_array), np.prod(output_array.shape[1:])))  \n",
    "    print(output_array.shape)\n",
    "    \n",
    "    return (input_array, output_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_array, output_array = upload_npy_to_arrays('./yaafe_ami_storage/', 0, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's focus on AMI Corpus**\n",
    "\n",
    "In the [dev.mdtm](https://github.com/pyannote/pyannote-db-odessa-ami/tree/master/AMI/data/speaker_diarization), we can see the:\n",
    "    - filename, offset, duration, \"speaker NA unknown\", id of speaker\n",
    "\n",
    "We should focus on this. We can download the corresponding .wav files easily. \n",
    "\n",
    "Firsly, I will create database for speaker change detection. It will be not-optimized until the end of test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir txt_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "\n",
    "def create_ground_truth(main_set):\n",
    "    ## It takes mdtm file and creates text file\n",
    "    # which will store end time of each speak\n",
    "    \n",
    "    # start_time_array = []\n",
    "    end_time_array = []\n",
    "    output_frame_array = []\n",
    "    with open(main_set) as f:\n",
    "        content = f.readlines()\n",
    "    # you may also want to remove whitespace characters like `\\n` at the end of each line\n",
    "    content = [x.strip() for x in content] \n",
    "    tmp_filename = content[0].split(' ')[0]\n",
    "    # sys.exit(\"WAIT, TEST :)\")\n",
    "\n",
    "    for single_line in content:\n",
    "        filename = single_line.split(' ')[0]\n",
    "        # print (filename)\n",
    "        if (filename != tmp_filename):\n",
    "            # start_time_array = np.asarray(start_time_array)\n",
    "            # np.savetxt(fname=tmp_filename + \"_start_time.txt\", X=start_time_array, delimiter=' ', fmt='%1.3f')\n",
    "            # end_time_array = np.asarray(end_time_array)\n",
    "            np.savetxt(fname=tmp_filename + \"_end_time.txt\", X=end_time_array, delimiter=' ', fmt='%1.3f')\n",
    "            # start_time_array = []\n",
    "            end_time_array = []\n",
    "            \n",
    "        tmp_filename = single_line.split(' ')[0]\n",
    "        offset = float(single_line.split(' ')[2])\n",
    "        duration = float(single_line.split(' ')[3])\n",
    "        end_time = offset+duration\n",
    "        # start_time_array.append(offset)\n",
    "        end_time_array.append(end_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd txt_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/pyannote/pyannote-db-odessa-ami/master/AMI/data/speaker_diarization/dev.mdtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/pyannote/pyannote-db-odessa-ami/master/AMI/data/speaker_diarization/tst.mdtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/pyannote/pyannote-db-odessa-ami/master/AMI/data/speaker_diarization/trn.mdtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_ground_truth('dev.mdtm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_ground_truth('trn.mdtm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_ground_truth('tst.mdtm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning Architectures\n",
    "\n",
    "- They use 2 different autoencoder. \n",
    "    - Smaller is 4000 → 2000 → 40 → 2000 → 4000\n",
    "    - Larger is 4000 → 6000 → 2000 → 40 → 2000 → 6000 → 4000\n",
    "![Image of Autoencoder](https://docs.google.com/uc?id=1epse9ba1fRTdmyN3pF0XoECEHcaiLsa3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import layers\n",
    "from keras import models\n",
    "from keras import optimizers\n",
    "import keras\n",
    "from keras.models import Model\n",
    "import tensorflow as tf\n",
    "from keras.layers.advanced_activations import *\n",
    "\n",
    "\n",
    "input_frame = keras.Input(shape=(4000,), name='main_input')\n",
    "encoded_1 = layers.Dense(4000, activation='relu', name='encoded_1')(input_frame) # not quite sure about the activation\n",
    "encoded_2 = layers.Dense(2000, activation='relu', name='encoded_2')(encoded_1)\n",
    "\n",
    "embedding = layers.Dense(40, activation='relu', name='embedding')(encoded_2)\n",
    "\n",
    "decoded_1 = layers.Dense(2000, activation='relu', name='decoded_1')(embedding)\n",
    "decoded_2 = layers.Dense(4000, activation='linear', name='decoded_2')(decoded_1)\n",
    "\n",
    "autoencoder = Model(input_frame, decoded_2)\n",
    "\n",
    "rmsprop = keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)\n",
    "autoencoder.compile(optimizer='rmsprop', loss='mean_squared_logarithmic_error')\n",
    "\n",
    "autoencoder.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from keras.models import load_model\n",
    "autoencoder.load_weights('speaker2vec_weights.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "\n",
    "how_many_step = 30\n",
    "ix_step = 0\n",
    "from_epi = 0\n",
    "\n",
    "while (ix_step < how_many_step):\n",
    "    ix_step += 1\n",
    "    \n",
    "    input_array, output_array = upload_npy_to_arrays('./npy_files/', from_ep=from_epi, to_ep=from_epi+4)\n",
    "    \n",
    "    try:\n",
    "        autoencoder.fit(input_array, output_array,\n",
    "               epochs=2,\n",
    "               batch_size=16,\n",
    "               shuffle=True)\n",
    "        autoencoder.save_weights('speaker2vec_weights.h5')    \n",
    "\n",
    "    except:\n",
    "        print (\"pass this epoch\")\n",
    "        pass\n",
    "    \n",
    "    # https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model\n",
    "\n",
    "    \n",
    "    input_array = []\n",
    "    output_array = []\n",
    "    \n",
    "    from_epi += 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's try to detect speaker change points. We will use encoder part of autoencoder.**\n",
    "- Firstly, extract the embedding,\n",
    "- Compare neighboorhood embeddings via different parameters,\n",
    "- If result of comparision exceed determined threshold, it represent the speaker change point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_system = Model(input_frame, embedding) # create system to extraxt embed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have embedding system. We should feed this sytem via input frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.signal import savgol_filter\n",
    "import scipy \n",
    "import more_itertools as mit\n",
    "\n",
    "\n",
    "def KL(P,Q):\n",
    "    \"\"\" Epsilon is used here to avoid conditional code for\n",
    "    checking that neither P nor Q is equal to 0. \"\"\"\n",
    "    epsilon = 0.00001\n",
    "\n",
    "     # You may want to instead make copies to avoid changing the np arrays.\n",
    "    P = P+epsilon\n",
    "    Q = Q+epsilon\n",
    "\n",
    "    divergence = np.sum(P*np.log(P/Q))\n",
    "    return divergence\n",
    "\n",
    "def create_prediction(audio_file, hop, win_len, threshold, embedding_system, featureplan=\"\", feature_extractor=\"librosa\"):\n",
    "    ## This takes audio file and create prediction to understand speaker change points.\n",
    "    # Its arguments:\n",
    "    # audio_file: Path of audio file, it can be wav, mp3, ogg etc.\n",
    "    # hop: Hop length (we need it for Librosa)\n",
    "    # win_len: Window length (we need it for Librosa)\n",
    "    # threshold: Threshold value for speaker chanhe point. Apply into smoothed version of array\n",
    "    # featureplan: Text file which introduce which features will be extracted. (we need it for YAAFE)\n",
    "    # feature_extractor: Which library will be used to extract features\n",
    "    \n",
    "    ## It will return:\n",
    "    # prediction_array_sec = Array which stores which second we have speaker change points.\n",
    "    # KL_array_to_0_1: Min max normalized version of KL Divergence array between embeds.\n",
    "    # KL_array_smooth: Smoothed version of KL_array_to_0_1\n",
    "\n",
    "    \n",
    "    KL_array = []\n",
    "    prediction_array_sec = []\n",
    "    \n",
    "    # at that point, we should not use overlapping segment\n",
    "    # https://github.com/librosa/librosa/issues/584\n",
    "    if (feature_extractor==\"librosa\"):\n",
    "        audio, sr = librosa.load(audio_file)\n",
    "        mfccs = librosa.feature.mfcc(audio, sr, n_mfcc=40, hop_length=int(hop/1000*sr), n_fft=int(win_len/1000*sr))\n",
    "    if (feature_extractor==\"yaafe\"):\n",
    "        mfccs = create_numpy_for_audio(audio_file, featureplan)\n",
    "    # now we have mfcc of all audio file.\n",
    "    \n",
    "    offset=0\n",
    "    print (\"K\")\n",
    "    \n",
    "    while (offset+201 < mfccs.shape[1]):\n",
    "    # line_mfccs = np.ravel(mfccs, order='F')\n",
    "        first_frame = np.ravel(mfccs[0:40,offset:offset+100])\n",
    "        first_frame = np.expand_dims(first_frame, axis=0)\n",
    "        first_embed = embedding_system.predict(first_frame)\n",
    "        \n",
    "        second_frame = np.ravel(mfccs[0:40,offset+101:offset+201])\n",
    "        second_frame = np.expand_dims(second_frame, axis=0)\n",
    "        second_embed = embedding_system.predict(second_frame)\n",
    "        KL_array.append(KL(first_embed, second_embed))\n",
    "        offset += 1\n",
    "        \n",
    "        \n",
    "    print (\"total offset: \", offset)\n",
    "    \n",
    "    KL_array = np.asarray(KL_array)\n",
    "        \n",
    "    # min max normalization\n",
    "    KL_array_to_0_1 = (KL_array-min(KL_array))/(max(KL_array)-min(KL_array))\n",
    "    KL_array_to_0_1 = np.asarray(KL_array_to_0_1)\n",
    "    # print (KL_array_to_0_1)    \n",
    "    \n",
    "    # Now, we should apply lowpass filter to get smooth KL curve.\n",
    "    # https://stats.stackexchange.com/questions/323069/can-kl-divergence-ever-be-greater-than-1\n",
    "    # https://stackoverflow.com/questions/20618804/how-to-smooth-a-curve-in-the-right-way\n",
    "    KL_array_smooth = savgol_filter(KL_array_to_0_1, 21, 7)\n",
    "    KL_array_smooth = np.asarray(KL_array_smooth)    \n",
    "        \n",
    "    offset = 0\n",
    "    for KL_value in KL_array_smooth:\n",
    "        if (KL_value > threshold):\n",
    "            ms_version = float(win_len + ((offset+1) * hop)) # milisecond version to represent end point of first embed       \n",
    "            prediction_array_sec.append(ms_version/1000)\n",
    "            \n",
    "        offset += 1\n",
    "            \n",
    "    prediction_array_sec = np.asarray(prediction_array_sec)\n",
    "    \n",
    "    prediction_array_msec = prediction_array_sec*1000\n",
    "    \n",
    "    prediction_array_smooth = []\n",
    "    for pred in prediction_array_msec:\n",
    "        if (pred-hop not in prediction_array_msec):\n",
    "            prediction_array_smooth.append(pred*0.001)\n",
    "            \n",
    "            \n",
    "    prediction_array_tenth_ms = np.asarray(prediction_array_msec)/10\n",
    "\n",
    "    list_cons = [list(group) for group in mit.consecutive_groups(prediction_array_tenth_ms)]\n",
    "    \n",
    "    mean_s = []\n",
    "    \n",
    "    for single_list_cons in list_cons:\n",
    "        # print (np.mean(single_list_cons))\n",
    "        mean_s.append(np.mean(single_list_cons)*0.01)\n",
    "                \n",
    "    np.savetxt(fname=audio_file.split(\"/\")[-1] + \"_prediction.txt\", X=mean_s, \n",
    "               delimiter=' ', fmt='%1.3f')\n",
    "\n",
    "    return (prediction_array_sec, KL_array_to_0_1, KL_array_smooth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def txt_file_to_matrix(filename, type_of_text, hop=10, win_len=25):\n",
    "    ## It takes the reference end time text file or prediction text file and return output array which represent the which frames has a speaker change point.\n",
    "    # filename= Which file will be considered.\n",
    "    # hop: Hop length (we need it for Librosa)\n",
    "    # win_len: Window length (we need it for Librosa)\n",
    "    \n",
    "    \n",
    "    matrix_of_single_audio = np.load(\"/home/herdogan/Desktop/SpChangeDetect/pyannote-audio/tutorials/feature-extraction/AMI/\" + filename + \".Mix-Headset.npy\")\n",
    "    \n",
    "    if (type_of_text == \"reference\"):\n",
    "        main_set = \"./txt_files/\" + filename + \"_end_time.txt\"# FILENAME PATH for TXT\n",
    "    \n",
    "    \n",
    "    if (type_of_text == \"prediction\"):\n",
    "        main_set = \"./\" + filename + \"_prediction.txt\"# FILENAME PATH for TXT\n",
    "\n",
    "    end_time_array_second = []\n",
    "\n",
    "\n",
    "    with open(main_set) as f:\n",
    "        content = f.readlines()\n",
    "        \n",
    "    content = [x.strip() for x in content] \n",
    "\n",
    "    for single_line in content:\n",
    "\n",
    "        end_time_array_second.append(single_line)\n",
    "\n",
    "    output_array = np.zeros(matrix_of_single_audio.shape[0])\n",
    "\n",
    "    for end_time in end_time_array_second:\n",
    "        end_time_ms = float(end_time)*1000\n",
    "        which_start_hop = (end_time_ms-win_len)/hop # now we know, milisecond version of change\n",
    "                                    # which is located after which_hop paramater\n",
    "                                    # add 2 and round to up\n",
    "        which_end_hop = end_time_ms/hop # round to up\n",
    "\n",
    "        start_location = math.ceil(which_start_hop + 1)\n",
    "        end_location = math.ceil(which_end_hop)\n",
    "\n",
    "        # print (\"s:\", start_location)\n",
    "        # print (\"e:\", end_location)\n",
    "        output_array[start_location:end_location+1] = 1.0\n",
    "\n",
    "    return (output_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now, implement transfer learning part. We will do it because of unsupervised domain adaptation.**\n",
    "- Find the speaker change points by trained DNN,\n",
    "- Get all possible speaker homogeneous regions,\n",
    "- Retrain the same DNN again on these homogeneous segments of speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "change_points, arr, arr_s = create_prediction('./test_16k.wav', hop=10, win_len=25, threshold=0.72, embedding_system=embedding_system)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr_s.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "offset = 0\n",
    "hop = 10\n",
    "win_len = 25\n",
    "threshold = 0.72\n",
    "\n",
    "print (\"thres:\", threshold)\n",
    "for KL_value in arr_s:\n",
    "    # print (KL_value)\n",
    "    if (KL_value > threshold):\n",
    "        print (offset)\n",
    "        ms_version = float(win_len + ((offset+1) * hop)) # milisecond version to represent end point of first embed       \n",
    "        \n",
    "\n",
    "    offset += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(change_points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(arr_s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as pp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot(arr_s)\n",
    "pp.rcParams['figure.figsize'] = (19.8, 10.0)\n",
    "\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot(arr_s)\n",
    "pp.rcParams['figure.figsize'] = (19.8, 12.0)\n",
    "\n",
    "pp.axhline(y=0.42, color='r', linestyle='-')\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot(grounth_truth_matrix(\"EN2001a\", 10, 25))\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa = grounth_truth_matrix(\"EN2001a\", 10, 25)\n",
    "bb = pred_file_to_frame(\"EN2001a\", 10, 25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (aa.shape)\n",
    "print (bb.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(1, len(aa)+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as pp\n",
    "%matplotlib inline\n",
    "\n",
    "pp.rcParams['figure.figsize'] = (19.8, 10.0)\n",
    "\n",
    "bb = bb * 0.9\n",
    "\n",
    "pp.plot(x[0:20000], aa[10000:30000], 'x', color='black');\n",
    "pp.plot(x[0:20000], bb[10000:30000], '.', color='pink');\n",
    "\n",
    "# pp.axhline(y=thres, color='r', linestyle='-')\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot(arr)\n",
    "pp.plot(savgol_filter(arr, 51, 3))\n",
    "\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot(arr)\n",
    "pp.plot(savgol_filter(arr, 31, 3))\n",
    "\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot(arr)\n",
    "pp.plot(savgol_filter(arr, 11, 3))\n",
    "\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp.plot(arr)\n",
    "pp.plot(savgol_filter(arr, 11, 7))\n",
    "\n",
    "pp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "change_points = create_prediction('test.wav', hop=10, win_len=25, threshold=0.6, embedding_system=embedding_system)\n",
    "\n",
    "# This is for unsupervised adaptation.\n",
    "\n",
    "## Very bad according to optimization view. !!!\n",
    "def creata_data_for_unsuper_adaptation(root_dir, hop, win_len):\n",
    "    all_wav_paths = glob.glob(os.path.join(root_dir, '*m4a'))\n",
    "    matrix_of_all_wav = []\n",
    "    \n",
    "    input_array = []\n",
    "    output_array = []\n",
    "    \n",
    "    for single_wav_path in all_wav_paths:\n",
    "        prediction_array = create_prediction(wav_file, hop, win_len, threshold, embedding_system)\n",
    "        matrix_of_single_wav = wav_to_matrix(single_wav_path, hop, win_len)\n",
    "            \n",
    "        start_point = 0\n",
    "        while (start_point+200 < matrix_of_single_wav.shape[1]):\n",
    "            change = 0 # value to check any value in prediction array is in range of frame\n",
    "            for single_change in prediction_array:\n",
    "                if (single_change in range(float(win_len + ((offset) * hop)), \n",
    "                                               float(win_len + ((offset+200) * hop)))):\n",
    "                    change = 1\n",
    "            if (change == 0):\n",
    "                single_input = wav_array[0:40,start_point:start_point+100]\n",
    "                single_output = wav_array[:40, start_point+100:start_point+200]\n",
    "                input_array.append(single_input)\n",
    "                output_array.append(single_output)\n",
    "            start_point += 100\n",
    "\n",
    "    input_array = np.asarray(input_array)\n",
    "    input_array = input_array.reshape((len(input_array), np.prod(input_array.shape[1:])))\n",
    "    \n",
    "    output_array = np.asarray(output_array)\n",
    "    output_array = output_array.reshape((len(output_array)*output_array.shape[1], output_array.shape[2]*output_array.shape[3] ))\n",
    "\n",
    "    print(output_array.shape)\n",
    "    \n",
    "    return (input_array, output_array)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}