5-ffbird-cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Workbook 5 - Recognising birds from FreeField1010 recordings using a Convolutional Neural Network "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous 4 workbooks have been processing audio the UrbanSound8K data set, this was chosen because it provided a large and varied set of samples, belonging to just 10 classes, which made it an ideal basis for developing and testing a general audio classifier. \n",
    "\n",
    "As the goal of this project is to recognise birdsong, the next logical step is to try it with actual recordings of birds. Unfortunately it seems difficult to find an equivalent of the UrbanSound8k data for birds, i.e. one where the recordings are labelled with the bird species, but I did find some interesting data sets with binary labels at http://machine-listening.eecs.qmul.ac.uk/bird-audio-detection-challenge/\n",
    "\n",
    "The binary labels are indicate that a human listener has stated that either they believe a bird is present, or that no birds are present. No species information is included, so the goal here is bird detection rather than identification, i.e. to find segments of 24-hour automated field recordings that might have interesting content, which can be forwarded to human experts who will identify what might be present.\n",
    "\n",
    "So here I'm going to adapt the CNN produced in workbook 4, and train it on the FreeField1010 data set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "import os\n",
    "import librosa\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the feature extraction code, it uses a similar process to that described in workbook 4. You'll only need to run this once, to obtain the feature data from the raw recordings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def windows(data, window_size):\n",
    "    start = 0\n",
    "    while start < len(data):\n",
    "        yield start, start + window_size\n",
    "        start += (window_size / 2)\n",
    "\n",
    "def extract_features(parent_dir,sub_dirs,file_ext=\"*.wav\",bands = 60, frames = 41):\n",
    "    window_size = 512 * (frames - 1)\n",
    "    log_specgrams = []\n",
    "    labels = []\n",
    "    for l, sub_dir in enumerate(sub_dirs):\n",
    "        for fn in glob.glob(os.path.join(parent_dir, sub_dir, file_ext)):\n",
    "            sound_clip,s = librosa.load(fn)\n",
    "            \n",
    "            idstr = os.path.basename(fn).split('.wav')[0]\n",
    "            if idstr not in my_dict.keys():\n",
    "                print \"No such entry \", idstr, 'skipping'\n",
    "                continue\n",
    "                \n",
    "            label = my_dict[idstr]    \n",
    "            \n",
    "            for (start,end) in windows(sound_clip,window_size):\n",
    "                if(len(sound_clip[start:end]) == window_size):\n",
    "                    signal = sound_clip[start:end]\n",
    "                    melspec = librosa.feature.melspectrogram(signal, n_mels = bands)\n",
    "                    logspec = librosa.logamplitude(melspec)\n",
    "                    logspec = logspec.T.flatten()[:, np.newaxis].T\n",
    "                    log_specgrams.append(logspec)\n",
    "                    labels.append(label)\n",
    "            \n",
    "    log_specgrams = np.asarray(log_specgrams).reshape(len(log_specgrams),bands,frames,1)\n",
    "    features = np.concatenate((log_specgrams, np.zeros(np.shape(log_specgrams))), axis = 3)\n",
    "    for i in range(len(features)):\n",
    "        features[i, :, :, 1] = librosa.feature.delta(features[i, :, :, 0])\n",
    "    \n",
    "    return np.array(features), np.array(labels,dtype = np.int)\n",
    "\n",
    "def one_hot_encode(labels):\n",
    "    n_labels = len(labels)\n",
    "    n_unique_labels = len(np.unique(labels))\n",
    "    one_hot_encode = np.zeros((n_labels,n_unique_labels))\n",
    "    one_hot_encode[np.arange(n_labels), labels] = 1\n",
    "    return one_hot_encode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code extracts features from the raw FreeField1010 recordings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entries: 7690\n"
     ]
    }
   ],
   "source": [
    "# create a dictionary from the labels file\n",
    "from numpy import loadtxt\n",
    "data_dir = \"data/ffbird-np-cnn\"\n",
    "filename = data_dir + \"/ff1010bird_metadata.csv\"\n",
    "lines = np.genfromtxt(filename, delimiter=\",\", skip_header=1, dtype=None)\n",
    "my_dict = dict()\n",
    "for i in range(len(lines)):\n",
    "    my_dict[str(lines[i][0])] = lines[i][1]\n",
    "    \n",
    "print \"Entries:\", len(my_dict) \n",
    "\n",
    "def assure_path_exists(path):\n",
    "    mydir = os.path.join(os.getcwd(), path)\n",
    "    if not os.path.exists(mydir):\n",
    "        os.makedirs(mydir)\n",
    "        \n",
    "# use this to process the audio files into numpy arrays\n",
    "def save_folds(data_dir):\n",
    "    for k in range(1,9):\n",
    "        fold_name = 'fold' + str(k)\n",
    "        print \"\\nSaving \" + fold_name\n",
    "        features, labels = extract_features(parent_dir, [fold_name])\n",
    "        labels = one_hot_encode(labels)\n",
    "        \n",
    "        print \"Features of\", fold_name , \" = \", features.shape\n",
    "        print \"Labels of\", fold_name , \" = \", labels.shape\n",
    "        \n",
    "        feature_file = os.path.join(data_dir, fold_name + '_x.npy')\n",
    "        labels_file = os.path.join(data_dir, fold_name + '_y.npy')\n",
    "        np.save(feature_file, features)\n",
    "        print \"Saved \" + feature_file\n",
    "        np.save(labels_file, labels)\n",
    "        print \"Saved \" + labels_file\n",
    "\n",
    "# update and uncomment the following lines if you want to run feature extraction for yourself   \n",
    "#parent_dir = \"path-to-the-freefield1010-data-set\"\n",
    "#assure_path_exists(data_dir)\n",
    "#save_folds(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reload Extracted Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fold1 features:  (19400, 60, 41, 2)\n",
      "fold2 features:  (19400, 60, 41, 2)\n",
      "train_x:  (38800, 60, 41, 2)\n",
      "2 train_y:  (38800, 2)\n",
      "1 train_y:  (38800, 1)\n",
      "fold8 features:  (18000, 60, 41, 2)\n",
      "valid_x:  (18000, 60, 41, 2)\n",
      "valid_y:  (18000, 1)\n",
      "fold7 features:  (19400, 60, 41, 2)\n",
      "test_x: (19400, 60, 41, 2)\n",
      "test_y: (19400, 1)\n"
     ]
    }
   ],
   "source": [
    "import scipy\n",
    "# this is used to load the folds incrementally\n",
    "data_dir = \"data/ffbird-np-cnn/np\"\n",
    "\n",
    "def load_folds(folds):\n",
    "    subsequent_fold = False\n",
    "    for k in range(len(folds)):\n",
    "        fold_name = 'fold' + str(folds[k])\n",
    "        feature_file = os.path.join(data_dir, fold_name + '_x.npy')\n",
    "        labels_file = os.path.join(data_dir, fold_name + '_y.npy')\n",
    "        loaded_features = np.load(feature_file)\n",
    "        loaded_labels = np.load(labels_file)\n",
    "        print fold_name, \"features: \", loaded_features.shape\n",
    "\n",
    "        if subsequent_fold:\n",
    "            features = np.concatenate((features, loaded_features))\n",
    "            labels = np.concatenate((labels, loaded_labels))\n",
    "        else:\n",
    "            features = loaded_features\n",
    "            labels = loaded_labels\n",
    "            subsequent_fold = True\n",
    "        \n",
    "    return features, labels\n",
    "\n",
    "train_x, train_y = load_folds([1,2])\n",
    "print \"train_x: \", train_x.shape\n",
    "print \"2 train_y: \", train_y.shape\n",
    "\n",
    "train_y = scipy.delete(train_y, 1, 1) \n",
    "print \"1 train_y: \", train_y.shape\n",
    "\n",
    "valid_x, valid_y = load_folds([8])\n",
    "valid_y = scipy.delete(valid_y, 1, 1) \n",
    "print \"valid_x: \", valid_x.shape\n",
    "print \"valid_y: \", valid_y.shape\n",
    "\n",
    "test_x, test_y = load_folds([7])\n",
    "test_y = scipy.delete(test_y, 1, 1) \n",
    "print \"test_x:\", test_x.shape\n",
    "print \"test_y:\", test_y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a Convolutional Neural Network with Keras and TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels: 1\n",
      "Feature size: 2460\n"
     ]
    }
   ],
   "source": [
    "frames = 41\n",
    "bands = 60\n",
    "\n",
    "feature_size = bands * frames #60x41\n",
    "num_labels = test_y.shape[1]\n",
    "num_channels = 2\n",
    "\n",
    "print \"Labels:\", num_labels\n",
    "print \"Feature size:\", feature_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation, Flatten\n",
    "from keras.layers import Convolution2D, MaxPooling2D\n",
    "from keras.optimizers import Adam, Adagrad, SGD\n",
    "from keras.callbacks import EarlyStopping\n",
    "from keras.regularizers import l2, activity_l2\n",
    "from keras.utils import np_utils\n",
    "\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "def evaluate(model):\n",
    "    y_prob = model.predict_proba(test_x, verbose=0)\n",
    "    y_pred = np_utils.probas_to_classes(y_prob)\n",
    "    y_true = np.argmax(test_y, 1)\n",
    "\n",
    "    roc = roc_auc_score(test_y, y_prob)\n",
    "    print \"ROC:\",  round(roc,3)\n",
    "\n",
    "    # evaluate the model\n",
    "    score, accuracy = model.evaluate(test_x, test_y, batch_size=32)\n",
    "    print(\"Accuracy = {:.2f}\".format(accuracy))\n",
    "    \n",
    "    return roc, accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# implements CNN described in https://arxiv.org/pdf/1608.04363.pdf\n",
    "\n",
    "def build_model():\n",
    "    \n",
    "    model = Sequential()\n",
    "    # input: 60x41 data frames with 2 channels => (60,41,2) tensors\n",
    "\n",
    "    # filters of size 3x3 - paper describes using 5x5, but their input data is 128x128\n",
    "    f_size = 3\n",
    "\n",
    "    # Layer 1 - 24 filters with a receptive field of (f,f), i.e. W has the\n",
    "    # shape (24,1,f,f).  This is followed by (4,2) max-pooling over the last\n",
    "    # two dimensions and a ReLU activation function\n",
    "    model.add(Convolution2D(24, f_size, f_size, border_mode='same', input_shape=(bands, frames, num_channels)))\n",
    "    model.add(MaxPooling2D(pool_size=(4, 2)))\n",
    "    model.add(Activation('relu'))\n",
    "\n",
    "    # Layer 2 - 48 filters with a receptive field of (f,f), i.e. W has the \n",
    "    # shape (48, 24, f, f). Like L1 this is followed by (4,2) max-pooling \n",
    "    # and a ReLU activation function.\n",
    "    model.add(Convolution2D(48, f_size, f_size, border_mode='same'))\n",
    "    model.add(MaxPooling2D(pool_size=(4, 2)))\n",
    "    model.add(Activation('relu'))\n",
    "\n",
    "    # Layer 3 - 48 filters with a receptive field of (f,f), i.e. W has the\n",
    "    # shape (48, 48, f, f). This is followed by a ReLU but no pooling.\n",
    "    model.add(Convolution2D(48, f_size, f_size, border_mode='valid'))\n",
    "    model.add(Activation('relu'))\n",
    "\n",
    "    # flatten output into a single dimension, let Keras do shape inference\n",
    "    model.add(Flatten())\n",
    "\n",
    "    # Layer 4 - a fully connected NN layer of 64 hidden units, L2 penalty of 0.001\n",
    "    model.add(Dense(64, W_regularizer=l2(0.001)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "\n",
    "    # Layer 5 - an output layer with one output unit per class, with L2 penalty, \n",
    "    # followed by a softmax activation function\n",
    "    model.add(Dense(num_labels, W_regularizer=l2(0.001)))\n",
    "    model.add(Dropout(0.5))\n",
    "    # softmax on single output will always normalise the value to 1.0, so need sigmoid\n",
    "    model.add(Activation('sigmoid'))\n",
    "\n",
    "    # create an optimiser\n",
    "    # adagrad = Adagrad(lr=0.01, epsilon=1e-08, decay=0.0)\n",
    "    sgd = SGD(lr=0.001, momentum=0.9, decay=0.0, nesterov=True)\n",
    "\n",
    "    # compile and fit model, reduce epochs if you want a result faster\n",
    "    # the validation set is used to identify parameter settings (epoch) that achieves \n",
    "    # the highest classification accuracy (note binary rather than categorial crossentropy)\n",
    "    model.compile(loss='binary_crossentropy', metrics=['accuracy'], optimizer=sgd)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building model...\n",
      "Training model...\n",
      "Train on 38800 samples, validate on 18000 samples\n",
      "Epoch 1/15\n",
      "38800/38800 [==============================] - 158s - loss: 0.7319 - acc: 0.7634 - val_loss: 0.5546 - val_acc: 0.7633\n",
      "Epoch 2/15\n",
      "38800/38800 [==============================] - 156s - loss: 0.7204 - acc: 0.7646 - val_loss: 0.6075 - val_acc: 0.7633\n",
      "Epoch 3/15\n",
      "38800/38800 [==============================] - 151s - loss: 0.7144 - acc: 0.7646 - val_loss: 0.5822 - val_acc: 0.7665\n",
      "Evaluating model...\n",
      "ROC: 0.695\n",
      "19400/19400 [==============================] - 26s    \n",
      "Accuracy = 0.77\n",
      "R.O.C: 0.695\n",
      "Accuracy: 0.768\n"
     ]
    }
   ],
   "source": [
    "# the ModelCheckpoint callback automatically saves the current model's weights to a file whenever the validation accuracy improves\n",
    "earlystop = EarlyStopping(monitor='val_loss', patience=1, verbose=0, mode='auto')\n",
    "\n",
    "print(\"Building model...\")\n",
    "model = build_model()\n",
    "\n",
    "# now fit the model to the training data, evaluating loss against the validation data\n",
    "print(\"Training model...\")\n",
    "model.fit(train_x, train_y, validation_data=(valid_x, valid_y), callbacks=[earlystop], batch_size=128, nb_epoch=15)\n",
    "        \n",
    "# now evaluate the trained model against the unseen test data\n",
    "print(\"Evaluating model...\")\n",
    "roc, acc = evaluate(model)\n",
    "       \n",
    "print 'R.O.C:', round(roc, 3)\n",
    "print 'Accuracy:', round(acc, 3)\n",
    "\n",
    "# best ROC (2 folds FF) = 0.69, acc = 0.77\n",
    "# best ROC (all 6 folds FF set only) = 0.73\n",
    "# best ROC (2 sets) = 0.869, acc=0.79 (Adagrad batch=128, patience=2) 50 epochs\n",
    "# best ROC (2 sets) = 0.87 (SGD batch=128, nesterov, momentum=0.9, lr-0.001) 50 epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Getting even better results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "There is another set of field recordings available from http://machine-listening.eecs.qmul.ac.uk/bird-audio-detection-challenge/ - the Warblr data set. By combining both the FreeField1010 and Warblr data sets I was able to obtain even better results, an F-score accuracy of 0.87, this however took considerable processing on a powerful cloud computing instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}