5-data-augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Classifying audio data with convolutional neural networks\n",
    "\n",
    "<br/>\n",
    "by Eduardo Garcia Rajo<br/>\n",
    "<br/>\n",
    "\n",
    "This notebook is part of the project \"Urban sounds classification with Covnolutional Neural Networks\" on [my Github](https://github.com/GorillaBus/urban-audio-classifier).<br/>\n",
    "<br/>\n",
    "Licensed under the GNU LESSER GENERAL PUBLIC LICENSE Version 3, 29 June 2007<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UrbanSound8K Data Augmentation\n",
    "\n",
    "On this notebook I'm creating variations of the original files applying the following effects:\n",
    "\n",
    "1. Time stretch\n",
    "2. Pitch shift\n",
    "3. Noise\n",
    "\n",
    "By default the new files are being saved in **UrbanSound8K/audio/augmented** with the same fold schema. At the end of the notebook a new metadata file is created combining the original dataset files and the generated ones.<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import librosa\n",
    "import librosa.display\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib as plt\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import clear_output, display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### * Load metadata CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define general variables\n",
    "\n",
    "# Set your path to the dataset\n",
    "us8k_path = os.path.abspath('./UrbanSound8K')\n",
    "audio_path = os.path.join(us8k_path, 'audio')\n",
    "augmented_path = os.path.join(audio_path, 'augmented')\n",
    "\n",
    "# Metadata\n",
    "metadata_path = os.path.join(us8k_path, 'metadata/UrbanSound8K.csv')\n",
    "metadata_augmented_path = os.path.abspath('data/augmented-data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### * Load metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>slice_file_name</th>\n",
       "      <th>fsID</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "      <th>salience</th>\n",
       "      <th>fold</th>\n",
       "      <th>classID</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>100032-3-0-0.wav</td>\n",
       "      <td>100032</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.317551</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>dog_bark</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>100263-2-0-117.wav</td>\n",
       "      <td>100263</td>\n",
       "      <td>58.5</td>\n",
       "      <td>62.500000</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>children_playing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>100263-2-0-121.wav</td>\n",
       "      <td>100263</td>\n",
       "      <td>60.5</td>\n",
       "      <td>64.500000</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>children_playing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>100263-2-0-126.wav</td>\n",
       "      <td>100263</td>\n",
       "      <td>63.0</td>\n",
       "      <td>67.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>children_playing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>100263-2-0-137.wav</td>\n",
       "      <td>100263</td>\n",
       "      <td>68.5</td>\n",
       "      <td>72.500000</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>children_playing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      slice_file_name    fsID  start        end  salience  fold  classID  \\\n",
       "0    100032-3-0-0.wav  100032    0.0   0.317551         1     5        3   \n",
       "1  100263-2-0-117.wav  100263   58.5  62.500000         1     5        2   \n",
       "2  100263-2-0-121.wav  100263   60.5  64.500000         1     5        2   \n",
       "3  100263-2-0-126.wav  100263   63.0  67.000000         1     5        2   \n",
       "4  100263-2-0-137.wav  100263   68.5  72.500000         1     5        2   \n",
       "\n",
       "              class  \n",
       "0          dog_bark  \n",
       "1  children_playing  \n",
       "2  children_playing  \n",
       "3  children_playing  \n",
       "4  children_playing  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the metadata from the generated CSV\n",
    "metadata = pd.read_csv(metadata_path)\n",
    "\n",
    "# Examine dataframe\n",
    "metadata.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Time stretching (changing play time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Progress: 17464/17464\n",
      "Last file:  99812-1-6-0.wav\n"
     ]
    }
   ],
   "source": [
    "rates = [0.81, 1.07]\n",
    "total = len(metadata) * len(rates)\n",
    "count = 0\n",
    "for rate in rates: \n",
    "    # Generate new stretched audio file\n",
    "    for index, row in metadata.iterrows():        \n",
    "        curr_fold = str(row['fold'])\n",
    "        curr_file_path = audio_path + '/fold' + curr_fold + '/' + row['slice_file_name']\n",
    "        \n",
    "        # Speed sub-dir inside current fold dir\n",
    "        curr_rate_path = augmented_path + '/fold' + curr_fold + '/speed_' + str(int(rate*100))\n",
    "\n",
    "        \n",
    "        # Create sub-dir if it does not exist\n",
    "        if not os.path.exists(curr_rate_path):\n",
    "            os.makedirs(curr_rate_path)\n",
    "                    \n",
    "        output_path = curr_rate_path + '/' + row['slice_file_name']\n",
    "        \n",
    "        # Skip when file already exists\n",
    "        if (os.path.isfile(output_path)):\n",
    "            count += 1 \n",
    "            continue\n",
    "        \n",
    "        y, sr = librosa.load(curr_file_path)  \n",
    "        y_changed = librosa.effects.time_stretch(y, rate=rate)\n",
    "        librosa.output.write_wav(output_path, y_changed, sr)\n",
    "        \n",
    "        count += 1 \n",
    "        \n",
    "        clear_output(wait=True)\n",
    "        print(\"Progress: {}/{}\".format(count, total))\n",
    "        print(\"Last file: \", row['slice_file_name'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Pitch shifting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Progress: 34928/34928\n",
      "Last file:  99812-1-6-0.wav\n"
     ]
    }
   ],
   "source": [
    "tone_steps = [-1, -2, 1, 2]\n",
    "total = len(metadata) * len(tone_steps)\n",
    "count = 0\n",
    "for tone_step in tone_steps:\n",
    "    # Generate new pitched audio\n",
    "    for index, row in metadata.iterrows():        \n",
    "        curr_fold = str(row['fold'])\n",
    "        curr_file_path = audio_path + '/fold' + curr_fold + '/' + row['slice_file_name']\n",
    "\n",
    "        # Pitch Shift sub-dir inside current fold dir\n",
    "        curr_ps_path = augmented_path + '/fold' + curr_fold + '/pitch_' + str(tone_step)\n",
    "\n",
    "        # Create sub-dir if it does not exist\n",
    "        if not os.path.exists(curr_ps_path):\n",
    "            os.makedirs(curr_ps_path)\n",
    "        \n",
    "        output_path = curr_ps_path + '/' + row['slice_file_name']\n",
    "        \n",
    "        # Skip when file already exists\n",
    "        if (os.path.isfile(output_path)):\n",
    "            count += 1 \n",
    "            continue\n",
    "        \n",
    "        y, sr = librosa.load(curr_file_path)  \n",
    "        y_changed = librosa.effects.pitch_shift(y, sr, n_steps=tone_step)\n",
    "        librosa.output.write_wav(output_path, y_changed, sr)\n",
    "        \n",
    "        count += 1 \n",
    "        \n",
    "        clear_output(wait=True)\n",
    "        print(\"Progress: {}/{}\".format(count, total))\n",
    "        print(\"Last file: \", row['slice_file_name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Progress: 8732/8732\n",
      "Last file:  99812-1-6-0.wav\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "def add_noise(data):\n",
    "    noise = np.random.rand(len(data))\n",
    "    noise_amp = random.uniform(0.005, 0.008)\n",
    "    data_noise = data + (noise_amp * noise)\n",
    "    return data_noise\n",
    "\n",
    "total = len(metadata)\n",
    "count = 0\n",
    "\n",
    "# Generate new noised audio\n",
    "for index, row in metadata.iterrows():        \n",
    "    curr_fold = str(row['fold'])\n",
    "    curr_file_path = audio_path + '/fold' + curr_fold + '/' + row['slice_file_name']\n",
    "\n",
    "    # Noised sub-dir inside current fold dir\n",
    "    curr_noise_path = augmented_path + '/fold' + curr_fold + '/noise'\n",
    "\n",
    "    # Create sub-dir if it does not exist\n",
    "    if not os.path.exists(curr_noise_path):\n",
    "        os.makedirs(curr_noise_path)\n",
    "        \n",
    "    output_path = curr_noise_path + '/' + row['slice_file_name']\n",
    "        \n",
    "    # Skip when file already exists\n",
    "    if (os.path.isfile(output_path)):\n",
    "        count += 1 \n",
    "        continue\n",
    "        \n",
    "    y, sr = librosa.load(curr_file_path)  \n",
    "    y_changed = add_noise(y)\n",
    "    librosa.output.write_wav(output_path, y_changed, sr)\n",
    "    \n",
    "    count += 1 \n",
    "\n",
    "    clear_output(wait=True)\n",
    "    print(\"Progress: {}/{}\".format(count, total))\n",
    "    print(\"Last file: \", row['slice_file_name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Create metadata for the new files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_files_recursive(path):\n",
    "    # create a list of file and sub directories names in the given directory \n",
    "    file_list = os.listdir(path)\n",
    "    all_files = list()\n",
    "    # Iterate over all the entries\n",
    "    for entry in file_list:\n",
    "        # Create full path\n",
    "        full_path = os.path.join(path, entry)\n",
    "        # If entry is a directory then get the list of files in this directory \n",
    "        if os.path.isdir(full_path):\n",
    "            all_files = all_files + get_files_recursive(full_path)\n",
    "        else:\n",
    "            all_files.append(full_path)\n",
    "                \n",
    "    return all_files   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61124 new entries\n"
     ]
    }
   ],
   "source": [
    "# Get every single file within the tree\n",
    "files = get_files_recursive(augmented_path)\n",
    "\n",
    "# Define metadata columns\n",
    "names = []\n",
    "classes = []\n",
    "folds = []\n",
    "augmentations = []\n",
    "\n",
    "# Iterate and collect name, fold and class\n",
    "for file in files:\n",
    "    pieces = file.split(\"/\")\n",
    "    file = pieces[len(pieces) - 1]\n",
    "    fold = pieces[len(pieces) - 3] \n",
    "    augment = pieces[len(pieces) - 2] \n",
    "    fold_num = fold[4:len(fold)]\n",
    "    class_id = file.split(\"-\")[1]\n",
    "\n",
    "    # Push records\n",
    "    names.append(file)\n",
    "    folds.append(fold_num)\n",
    "    classes.append(class_id)\n",
    "    augmentations.append(augment)\n",
    "\n",
    "# Create a dataframe with the new augmented data\n",
    "new_meta = pd.DataFrame({'file': names, 'fold': folds, 'class_id': classes, 'augment': augmentations })\n",
    "\n",
    "# Make sure class_id is int\n",
    "new_meta['class_id'] = new_meta['class_id'].astype(np.int64)\n",
    "\n",
    "print(len(new_meta), \"new entries\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add class names to the new dataframe using merge\n",
    "classes = pd.DataFrame({\n",
    "    'class_id': range(0,10),\n",
    "    'class': [\n",
    "        'air_conditioner',\n",
    "        'car_horn',\n",
    "        'children_playing',\n",
    "        'dog_bark',\n",
    "        'drilling',\n",
    "        'engine_idling',\n",
    "        'gun_shot',\n",
    "        'jackhammer',\n",
    "        'siren',\n",
    "        'street_music'\n",
    "    ]\n",
    "})\n",
    "\n",
    "new_meta = pd.merge(new_meta, classes, on='class_id')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>fold</th>\n",
       "      <th>class_id</th>\n",
       "      <th>augment</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>61119</td>\n",
       "      <td>199769-1-0-6.wav</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>pitch_2</td>\n",
       "      <td>car_horn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>61120</td>\n",
       "      <td>18594-1-1-0.wav</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>pitch_2</td>\n",
       "      <td>car_horn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>61121</td>\n",
       "      <td>151359-1-0-0.wav</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>pitch_2</td>\n",
       "      <td>car_horn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>61122</td>\n",
       "      <td>18594-1-0-0.wav</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>pitch_2</td>\n",
       "      <td>car_horn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>61123</td>\n",
       "      <td>199769-1-0-12.wav</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>pitch_2</td>\n",
       "      <td>car_horn</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    file fold  class_id  augment     class\n",
       "61119   199769-1-0-6.wav    3         1  pitch_2  car_horn\n",
       "61120    18594-1-1-0.wav    3         1  pitch_2  car_horn\n",
       "61121   151359-1-0-0.wav    3         1  pitch_2  car_horn\n",
       "61122    18594-1-0-0.wav    3         1  pitch_2  car_horn\n",
       "61123  199769-1-0-12.wav    3         1  pitch_2  car_horn"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_meta.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Integrate metadata in a single file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify original data to fit the new structure\n",
    "del metadata['fsID'], metadata['start'], metadata['end'], metadata['salience']\n",
    "metadata.columns = ['file', 'fold', 'class_id', 'class']\n",
    "metadata['augment'] = 'none'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataframes merged correctly!\n",
      "Initial data: 8732\n",
      "New data: 61124\n",
      "Merged data: 69856\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/edu/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/ipykernel_launcher.py:2: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=False'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass 'sort=True'.\n",
      "\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# Concat the two dataframes\n",
    "full_meta = pd.concat([metadata, new_meta])\n",
    "\n",
    "# Verify lengths\n",
    "if (len(full_meta) == len(metadata) + len(new_meta)):\n",
    "    print(\"Dataframes merged correctly!\")\n",
    "else:\n",
    "    print(\"Error! Lengths do not match.\")\n",
    "\n",
    "print(\"Initial data:\", len(metadata))\n",
    "print(\"New data:\", len(new_meta))\n",
    "print(\"Merged data:\", len(full_meta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  6. Save the new dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the new metadata\n",
    "full_meta.to_csv(metadata_augmented_path, index=False, encoding=\"utf-8\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}