jupyter_examples/example1/example1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming that everything is running properly, [a live version of this notebook is available on epoxyhub](http://epoxyhub.org/?source=https://github.com/flatironinstitute/mountainsort_examples&path=jupyter_examples/example1/example1.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic spike sorting with MountainSort and Singularity\n",
    "\n",
    "This example shows how to use MountainSort within JupyterLab using code stored in Singularity containers\n",
    "\n",
    "This notebook accomplishes the following\n",
    "\n",
    "1. Specify which version of the processing to run by pointing to Singularity containers stored on Singularity Hub (Singularity container registry)\n",
    "1. User selects a (remote or local) processing resource (if local, Singularity must be installed on the computer running jupyterlab)\n",
    "1. Create a synthetic dataset\n",
    "1. Run the spike sorting\n",
    "1. Compare with ground truth\n",
    "1. Provide a simple output showing the results of the ground-truth comparison\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "1. Conda packages (see environment.yml) including\n",
    "    - mountainlab\n",
    "    - mountainlab_pytools\n",
    "    - spikeforestwidgets\n",
    "    - matplotlib\n",
    "1. *Singularity -- only needed if running on the local machine, i.e., the maching running jupyterlab*\n",
    "    - note that Singularity should not be installed using conda. Instead, install it on your system using admin priviliges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First import the required python modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#######################################\n",
    "# imports and initialization\n",
    "#######################################\n",
    "\n",
    "# For development purposes, reload imported modules when source changes\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def append_to_path(dir0): # A convenience function\n",
    "    if dir0 not in sys.path:\n",
    "        sys.path.append(dir0)\n",
    "\n",
    "# standard imports\n",
    "import os, sys, json\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# mountainlab imports\n",
    "from mountainlab_pytools import mlproc as mlp\n",
    "from mountainlab_pytools import mdaio\n",
    "import spikeforestwidgets as SFW\n",
    "\n",
    "# imports from this repo\n",
    "append_to_path(os.getcwd()+'/../../python')\n",
    "from mountainsort4_1_0 import sort_dataset as ms4_sort_dataset # MountainSort spike sorting\n",
    "from validate_sorting_results import validate_sorting_results # Validation processors\n",
    "from default_lari_servers import default_lari_servers # Choices for processing servers\n",
    "from synthesize_dataset import synthesize_dataset # Synthesize a test dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify the Singularity containers (on Singularity Hub) containing the required MountainLab processors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define which Singularity containers we will use for the processing\n",
    "# The name of the processor determines which container is used\n",
    "mlp.setContainerRules([])\n",
    "mlp.addContainerRule(pattern='ephys.*',container='shub://magland/ml_ephys:v0.2.5')\n",
    "mlp.addContainerRule(pattern='ms4alg.*',container='shub://magland/ml_ms4alg:v0.1.4')\n",
    "mlp.addContainerRule(pattern='pyms.*',container='shub://magland/ml_pyms:v0.0.1')\n",
    "mlp.addContainerRule(pattern='ms3.*',container='shub://magland/ml_ms3:v0.0.2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The user selects the processing resource\n",
    "\n",
    "If this is going to the be local computer (i.e., the computer running JupyterLab), then you must have singularity installed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#######################################\n",
    "# LARI login and initialize the pipeline object\n",
    "#######################################\n",
    "\n",
    "SFW.LariLoginWidget(default_lari_servers()).display()\n",
    "Pipeline=mlp.initPipeline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the synthetic dataset\n",
    "\n",
    "This will go into a new directory called `dataset/`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make synthetic ephys data and create output directory\n",
    "dsdir=os.getcwd()+'/dataset'\n",
    "with Pipeline:\n",
    "    synthesize_dataset(dsdir,M=4,duration=600,average_snr=8)\n",
    "    \n",
    "output_base_dir=os.getcwd()+'/output'\n",
    "if not os.path.exists(output_base_dir):\n",
    "    os.mkdir(output_base_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Note that the following does not work yet when using the local computer for computation\n",
    "## because I have not yet exposed the file system to the javascript widget\n",
    "\n",
    "#SFW.viewDataset(directory=dsdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the spike sorting and comparison with ground truth\n",
    "\n",
    "The output will go into a new directory called `output/`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#######################################\n",
    "# RUN THE PIPELINE\n",
    "#######################################\n",
    "output_dir=output_base_dir+'/ms4'\n",
    "with Pipeline:\n",
    "    ms4_sort_dataset(dataset_dir=dsdir,output_dir=output_dir,adjacency_radius=-1,detect_threshold=3)\n",
    "    A=validate_sorting_results(dataset_dir=dsdir,sorting_output_dir=output_dir,output_dir=output_dir)\n",
    "    amplitudes_true=A['amplitudes_true']\n",
    "    accuracies=A['accuracies']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the comparison with ground truth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the comparison with ground truth\n",
    "plt.plot(amplitudes_true,accuracies,'.')\n",
    "plt.xlabel('Amplitude')\n",
    "plt.ylabel('Accuracy');\n",
    "plt.title('Accuracy vs. amplitude for {}'.format('simulated data'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}