Add documentation and comments, improve tests and examples

README.md

@@ -2,10 +2,24 @@
 [U-net](https://arxiv.org/abs/1505.04597) based CNN for segmenting blood vessel and thereafter removal of vessels from fundus image to allow better diagnostic models using classifiers trained on top of this cleaned up version of fundus as well as classifiers trained to analyse the vessel maps to identify clinical features associated with vessel shapes, like vessel tortuosity.
 
 ## Training Data
-Training Data is obtained from [DRIVE](https://www.isi.uu.nl/Research/Databases/DRIVE/) and [STARE](http://cecas.clemson.edu/~ahoover/stare/) datasets.
+Training Data is obtained from [DRIVE](https://www.isi.uu.nl/Research/Databases/DRIVE/) and [STARE](http://cecas.clemson.edu/~ahoover/stare/) datasets. For STARE dataset the target vessel map annotated by Valentina Kouznetsova is used since it was more detailed.
 
 ## Data preprocessing and Dataset generation
 The notebook [generate_patches](generate_patches.ipynb) is used for generating of a vast dataset of `256 X 256` patches from the images available in DRIVE and STARE datasets. The patches are generated at random. For robust training patches involving flipping of image and addition of noise are also generated.
+In order to use the notebook without any changes ensure following tree structure for storing DRIVE and STARE datasets:
+```
+VesselExtract/
+├── DRIVE
+│   ├── test
+│   └── training
+├── STARE
+│   ├── labels-vk
+│   └── stare-images
+├── generate_patches.ipynb
+├── README.md
+├── research_model.ipynb
+├── run_tests.ipynb
+```
 
 ## Model
 The training of model for vessel segmentation is done in the notebook [research_model](research_model.ipynb). Results of vessel segmentation in presence of various clinical features is also documented.

generate_patches.ipynb

@@ -15,44 +15,11 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": 13,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "['29_training.tif',\n",
-       " '36_training.tif',\n",
-       " '35_training.tif',\n",
-       " '25_training.tif',\n",
-       " '34_training.tif',\n",
-       " '32_training.tif',\n",
-       " '22_training.tif',\n",
-       " '21_training.tif',\n",
-       " '38_training.tif',\n",
-       " '23_training.tif',\n",
-       " '26_training.tif',\n",
-       " '31_training.tif',\n",
-       " '33_training.tif',\n",
-       " '24_training.tif',\n",
-       " '40_training.tif',\n",
-       " '27_training.tif',\n",
-       " '39_training.tif',\n",
-       " '28_training.tif',\n",
-       " '37_training.tif',\n",
-       " '30_training.tif']"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
    "source": [
-    "image_dir = \"DRIVE/training/\"\n",
-    "image_list = os.listdir(\"DRIVE/training/images/\")\n",
-    "image_list"
+    "#### Data generation by splitting image \n",
+    "`random_patch` selects random 256 * 256 patches from the image and generates training examples from that. Moreover to increase the count of distinct examples image flipping and noise addition is used"
    ]
   },
   {
@@ -61,7 +28,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "def random_patch(image,manual,mask,count):\n",
+    "def random_patch(image, manual, mask, count):\n",
     "    # patch without any effects\n",
     "    x,y = np.random.randint(0,im.shape[0]-256),np.random.randint(0,im.shape[1]-256)\n",
     "    while (mask[x][y]<0.001 and mask[x][y+256]<0.001 and mask[x+256][y]<0.001 and mask[x+256][y+256]<0.001):\n",
@@ -93,28 +60,101 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Setting up the directory structures for storing training data\n",
     "!mkdir -p data/training\n",
     "!mkdir -p data/manual\n",
     "!rm -rf data/training/*\n",
-    "!rm -rf data/manual/*\n",
-    "\n",
+    "!rm -rf data/manual/*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# To ignore warnings when splitting images\n",
     "import warnings\n",
-    "warnings.simplefilter('ignore')\n",
-    "\n",
+    "warnings.simplefilter('ignore')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Patch generation for DRIVE"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['29_training.tif',\n",
+       " '36_training.tif',\n",
+       " '35_training.tif',\n",
+       " '25_training.tif',\n",
+       " '34_training.tif',\n",
+       " '32_training.tif',\n",
+       " '22_training.tif',\n",
+       " '21_training.tif',\n",
+       " '38_training.tif',\n",
+       " '23_training.tif',\n",
+       " '26_training.tif',\n",
+       " '31_training.tif',\n",
+       " '33_training.tif',\n",
+       " '24_training.tif',\n",
+       " '40_training.tif',\n",
+       " '27_training.tif',\n",
+       " '39_training.tif',\n",
+       " '28_training.tif',\n",
+       " '37_training.tif',\n",
+       " '30_training.tif']"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "image_dir = \"DRIVE/training/\"\n",
+    "image_list = os.listdir(\"DRIVE/training/images/\")\n",
+    "image_list"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "count=0\n",
     "for image in image_list:\n",
     "    im = io.imread(image_dir+\"images/\"+image)\n",
     "    mask = io.imread(image_dir+\"mask/\"+image[0:2]+\"_training_mask.gif\")\n",
     "    manual = io.imread(image_dir+\"1st_manual/\"+image[0:2]+\"_manual1.gif\")\n",
+    "    \n",
+    "    # For each image generate 32 * 3 patches\n",
     "    for i in range(32):\n",
     "        random_patch(im,manual,mask,count)\n",
     "        count+=1"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Patch generation for STARE"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 16,
@@ -167,11 +207,19 @@
     "    manual = io.imread(image_dir+\"labels-vk/\"+image)\n",
     "    mask = np.ones((im.shape[0],im.shape[1]))\n",
     "    \n",
+    "    # For each image generate 32 * 3 patches\n",
     "    for i in range(32):\n",
     "        random_patch(im,manual,mask,count)\n",
     "        count+=1"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Some examples of the patches generated"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 19,