binaural-hearing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Binaural Hearing\n",
    "\n",
    "[return to main page](index.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import soundfile as sf\n",
    "import sounddevice as sd\n",
    "from scipy.signal import fftconvolve\n",
    "import tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interaural Correlation, Degree of Coherence\n",
    "\n",
    "This exercise examines the degree of coherence $k$ between right and left ear using noise signals.\n",
    "You can find more information in the (non-public) lecture notes on slides 4-29 to 4-31."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fs = 44100\n",
    "dur = 1  # seconds\n",
    "ks = np.linspace(0, 1, 11)\n",
    "stddev = 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "length = int(dur * fs)\n",
    "\n",
    "np.random.seed(7)\n",
    "\n",
    "for k in ks:\n",
    "    R = [[1, k],\n",
    "         [k, 1]]\n",
    "    L, V = np.linalg.eig(R)\n",
    "    L.shape = 1, -1  # turn L into row vector\n",
    "    W = V * np.sqrt(L)\n",
    "    # create a new random signal in each iteration:\n",
    "    stereo_noise = np.random.normal(scale=stddev, size=(length, 2))\n",
    "    outsig = np.dot(stereo_noise, W.T)  # matrix product\n",
    "    sf.write('outdata/noise_k{:.1f}.wav'.format(k), outsig, fs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates several WAV files containing noise.\n",
    "The degree of coherence $k$ between both channels is part of the\n",
    "respective filename.\n",
    "\n",
    "*Exercise:* Listen to the different files and note how you perceive different degrees of\n",
    "coherence.\n",
    "\n",
    "$k=1.0$<audio src=\"outdata/noise_k1.0.wav\" controls></audio>\n",
    "[outdata/noise_k1.0.wav](outdata/noise_k1.0.wav)\n",
    "\n",
    "$k=0.9$<audio src=\"outdata/noise_k0.9.wav\" controls></audio>\n",
    "[outdata/noise_k0.9.wav](outdata/noise_k0.9.wav)\n",
    "\n",
    "$k=0.8$<audio src=\"outdata/noise_k0.8.wav\" controls></audio>\n",
    "[outdata/noise_k0.8.wav](outdata/noise_k0.8.wav)\n",
    "\n",
    "$k=0.7$<audio src=\"outdata/noise_k0.7.wav\" controls></audio>\n",
    "[outdata/noise_k0.7.wav](outdata/noise_k0.7.wav)\n",
    "\n",
    "$k=0.6$<audio src=\"outdata/noise_k0.6.wav\" controls></audio>\n",
    "[outdata/noise_k0.6.wav](outdata/noise_k0.6.wav)\n",
    "\n",
    "$k=0.5$<audio src=\"outdata/noise_k0.5.wav\" controls></audio>\n",
    "[outdata/noise_k0.5.wav](outdata/noise_k0.5.wav)\n",
    "\n",
    "$k=0.4$<audio src=\"outdata/noise_k0.4.wav\" controls></audio>\n",
    "[outdata/noise_k0.4.wav](outdata/noise_k0.4.wav)\n",
    "\n",
    "$k=0.3$<audio src=\"outdata/noise_k0.3.wav\" controls></audio>\n",
    "[outdata/noise_k0.3.wav](outdata/noise_k0.3.wav)\n",
    "\n",
    "$k=0.2$<audio src=\"outdata/noise_k0.2.wav\" controls></audio>\n",
    "[outdata/noise_k0.2.wav](outdata/noise_k0.2.wav)\n",
    "\n",
    "$k=0.1$<audio src=\"outdata/noise_k0.1.wav\" controls></audio>\n",
    "[outdata/noise_k0.1.wav](outdata/noise_k0.1.wav)\n",
    "\n",
    "$k=0.0$<audio src=\"outdata/noise_k0.0.wav\" controls></audio>\n",
    "[outdata/noise_k0.0.wav](outdata/noise_k0.0.wav)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binaural Unmasking\n",
    "\n",
    "Create audio examples based on slide 4-33.\n",
    "\n",
    "To do that, convolve the given HRIRs (stored in the files `data/hrir00.wav`,\n",
    "`data/hrir45.wav` and `data/hrir90.wav`) on the one hand with an arbitrary\n",
    "speech signal (e.g. from the `data/` directory), on the other hand with a noise signal\n",
    "(use the function [numpy.random.normal()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.random.normal.html)).\n",
    "\n",
    "Add speech and noise in different combinations of angles of incidence.\n",
    "Amplify or attenuate speech and noise in a way that the speech is just barely\n",
    "understandable.\n",
    "\n",
    "Listen to the different combinations.\n",
    "In which combination can you understand the speech better or worse?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interaural Time Difference/Interaural Level Difference\n",
    "\n",
    "Two important *cues* for spatial hearing are ITD and ILD.\n",
    "\n",
    "*Exercise:* Click and learn:\n",
    "\n",
    "* Watch (and listen to!) the videos on this page: http://auditoryneuroscience.com/spatial-hearing/binaural-cues\n",
    "\n",
    "* Here is an interactive example: http://auditoryneuroscience.com/spatial-hearing/time-intensity-trading\n",
    "\n",
    "*Exercise (only if you happen to have access to Matlab®):* Do the listening test described on this page: http://auditoryneuroscience.com/spatial-hearing/ITD-ILD-practical\n",
    "\n",
    "*Exercise:* Try to calculate the ITD as function of the angle of incidence using a\n",
    "very much simplified head model. Assume a spherical head with ear holes on\n",
    "exactly opposite points on the sphere.\n",
    "Make the head radius an adjustable parameter in your calculations; use\n",
    "8.75 cm as default value.\n",
    "Assume further, that the sound source is sufficiently far away, so that the\n",
    "angle of incidence is constant on the whole sphere.\n",
    "\n",
    "What is the maximum distance in the path from the sound source to the two ear\n",
    "holes, respectively?\n",
    "\n",
    "What time difference does that correspond to, assuming the speed of sound\n",
    "$c = 343 \\text{ m/s}$?\n",
    "\n",
    "Plot the ITD as a function of angle of incidence for a given head radius."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Spatial Hearing of Multiple Sound Sources\n",
    "\n",
    "Simulate the ear signals of the listening experiments on slide 4-26 and 4.27.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise*: Create a function that takes a mono signal and returns two signals.\n",
    "The left signal is same as the input but the right signal is delayed and scaled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def delay_and_attenuate(x, delay, gain, fs=44100):\n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ----------\n",
    "    x : array_like\n",
    "        Mono signal\n",
    "    delay : float\n",
    "        Delay of the second channel (in milliseconds)\n",
    "    gain : float\n",
    "        Gain of the second channel (in dB)\n",
    "    \"\"\"\n",
    "    d = int(np.round(delay * fs * 0.001))\n",
    "    x = np.tile(np.concatenate((x, np.zeros(np.abs(d))), axis=-1), (2, 1)).T\n",
    "    x[:, 1] = 10**(gain / 20) * np.roll(x[:, 1], d, axis=0)\n",
    "        \n",
    "    return tools.normalize(x, maximum=0.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise*: Use two HRIR pairs to create the ear signals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def virtual_stereo(x, **kwargs):\n",
    "    \"\"\"\n",
    "    \n",
    "    \"\"\"\n",
    "    # load HRIRs\n",
    "    hright, _ = sf.read('data/hrir30.wav')\n",
    "    hleft = np.roll(hright, 1)\n",
    "\n",
    "    return np.column_stack([fftconvolve(x[:, 0], ir, **kwargs) for ir in hleft.T]) \\\n",
    "            + np.column_stack([fftconvolve(x[:, 1], ir, **kwargs) for ir in hright.T])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the ear signals of the 'Virtual Stereo System'.\n",
    "Try it with different types of signals in 'data/' ('castanets.wav', 'xmax.wav', 'singing.wav')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "delay = 0\n",
    "gain = 0\n",
    "# mono_signal, _ = sf.read('...')\n",
    "\n",
    "stereo_signal = delay_and_attenuate(mono_signal, delay, gain)\n",
    "y = tools.normalize(virtual_stereo(stereo_signal), maximum=0.6)\n",
    "\n",
    "sd.play(y)\n",
    "sd.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summing Localization\n",
    "Vary the relative gain between the left and right channels.\n",
    "Try to localize the sound source.\n",
    "Do you hear only one source?\n",
    "Compare your observations with the results on Slide 4-26."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Law of the First Wave Front\n",
    "\n",
    "Apply a delay to the second (right) signal, in the range of 0 -- 50 ms. Compare your observation with the results shown in Slide 4-27."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applications of Binaural Synthesis\n",
    "* Games https://www.youtube.com/watch?v=8e-O2V8QcUE\n",
    "* Virtual studio http://www.waves.com/plugins/nx#nx-head-tracker-for-headphones-quick-start\n",
    "* Demo with head-tracker https://www.waves.com/nx/player\n",
    "* Demo https://www.bbc.co.uk/programmes/p05189jv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solutions\n",
    "\n",
    "If you had problems solving some of the exercises, don't despair!\n",
    "Have a look at the [example solutions](binaural-hearing-solutions.ipynb).\n",
    "\n",
    "<p xmlns:dct=\"http://purl.org/dc/terms/\">\n",
    "  <a rel=\"license\"\n",
    "     href=\"http://creativecommons.org/publicdomain/zero/1.0/\">\n",
    "    <img src=\"http://i.creativecommons.org/p/zero/1.0/88x31.png\" style=\"border-style: none;\" alt=\"CC0\" />\n",
    "  </a>\n",
    "  <br />\n",
    "  To the extent possible under law,\n",
    "  <span rel=\"dct:publisher\" resource=\"[_:publisher]\">the person who associated CC0</span>\n",
    "  with this work has waived all copyright and related or neighboring\n",
    "  rights to this work.\n",
    "</p>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}