vision/object-detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Object detection\n",
    "\n",
    "This notebook explores different methods of simple object detection in printed forms using OpenCV. Object detection serves as one of the methods of digitising printed forms, so the data can be used for further analysis. The accompanying blog post can be found [here](https://fuzzylabs.ai/blog/checkbox-detection/)\n",
    "\n",
    "As an example, we will take a section of a questionnaire and try to detect checkboxes present in it. The sample image can be found at `gs://fuzzylabs-jupyter-delicacies/form_segment.png`.\n",
    "\n",
    "This approach can be extended to account for other objects on forms (so these forms can be digitised for the purpose of further data analysis). Therefore, you are encouraged to try it on different images (local files and Google Cloud Storage are supported) and modify the code to make it work on your examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run this cell to install the necessary packages, if they are not already installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install opencv-python numpy matplotlib gcsfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cv2 import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import math\n",
    "import pandas as pd\n",
    "import gcsfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_image(img, colorspace=cv2.COLOR_GRAY2RGB):\n",
    "    plt.figure(figsize=(12,12))\n",
    "    plt.imshow(cv2.cvtColor(img, colorspace))\n",
    "    plt.show()\n",
    "    \n",
    "def imread_wrapper(uri):\n",
    "    if uri.startswith(\"gs://\"):\n",
    "        with gcsfs.GCSFileSystem().open(uri, \"rb\") as f:\n",
    "            arr = np.asarray(bytearray(f.read()), dtype=\"uint8\")\n",
    "            img = cv2.imdecode(arr, cv2.IMREAD_COLOR)\n",
    "            return img\n",
    "    else: # Assume local file\n",
    "        return cv2.imread(uri, cv2.IMREAD_COLOR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = imread_wrapper(\"gs://fuzzylabs-jupyter-delicacies/form_segment.png\")\n",
    "show_image(img, colorspace=cv2.COLOR_BGR2RGB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image preparation\n",
    "\n",
    "Three methods of preprocessing are investigated:\n",
    "* Thresholding (with Otsu's method to determine an optimal threshold)\n",
    "* Opening on the thresholded image. Modifiable parameter: `KERNEL_LENGTH`\n",
    "* Canny edge detector. Modifiable parameter: `CANNY_KERNEL_SIZE`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image thresholding\n",
    "_, img_bin = cv2.threshold(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)\n",
    "img_bin = 255 - img_bin\n",
    "print(\"Thresholded image\")\n",
    "show_image(img_bin)\n",
    "\n",
    "# Opening with vertical kernel\n",
    "KERNEL_LENGTH = 15\n",
    "vertical_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1,KERNEL_LENGTH))\n",
    "vertical = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, vertical_kernel, iterations=1)\n",
    "print(\"Vertical opened image\")\n",
    "show_image(vertical)\n",
    "# Opening with horizontal kernel \n",
    "horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (KERNEL_LENGTH,1))\n",
    "horizontal = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, horizontal_kernel, iterations=1)\n",
    "print(\"Horizontal opened image\")\n",
    "show_image(horizontal)\n",
    "# Combining openings\n",
    "img_opened = cv2.addWeighted(vertical, 0.5, horizontal, 0.5, 0.0)\n",
    "_, img_opened = cv2.threshold(img_opened, 0, 255, cv2.THRESH_BINARY)\n",
    "print(\"Combined opened image\")\n",
    "show_image(img_opened)\n",
    "\n",
    "#\n",
    "CANNY_KERNEL_SIZE = 100\n",
    "print(\"Canny edge detection\")\n",
    "img_canny = cv2.Canny(img, CANNY_KERNEL_SIZE, CANNY_KERNEL_SIZE)\n",
    "show_image(img_canny, cv2.COLOR_BGR2RGB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def draw_contours(contours):\n",
    "    show_image(cv2.drawContours(img.copy(), contours, -1, (0, 255, 0), 1), cv2.COLOR_BGR2RGB)\n",
    "\n",
    "def get_contours(img_bin):\n",
    "    contours, hierarchy = cv2.findContours(img_bin, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n",
    "    draw_contours(contours)\n",
    "    return contours\n",
    "\n",
    "print(\"Thresholded image\")\n",
    "thresholded_contours = get_contours(img_bin)\n",
    "print(\"Opened image\")\n",
    "opened_contours = get_contours(img_opened)\n",
    "print(\"Canny edge detection\")\n",
    "edged_contours = get_contours(img_canny)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contour Filtering\n",
    "Multiple methods to filter the resulting contours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_contours(contours, func):\n",
    "    _contours = [x for x in contours if func(x)]\n",
    "    draw_contours(_contours)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter by contour area, parameters: `expected_area` and `tolerance`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_correct_area(contour, expected_area=625, tolerance=200):\n",
    "    area = cv2.contourArea(contour)\n",
    "    return abs(area - expected_area) <= tolerance\n",
    "\n",
    "print(\"Thresholded image\")\n",
    "filter_contours(thresholded_contours, is_correct_area)\n",
    "print(\"Opened image\")\n",
    "filter_contours(opened_contours, is_correct_area)\n",
    "print(\"Canny edge detection\")\n",
    "filter_contours(edged_contours, is_correct_area)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evidently, this filtering is not robust enough. We also want only the square boxes to be detected\n",
    "\n",
    "Filtering by assessing bounding boxes for squareness. Parameters: `expected_area`, `tolerance` and `squareness_tolerance`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def are_bounding_dimensions_correct(contour, expected_area=625, tolerance=200, squareness_tolerance=5):\n",
    "    area = cv2.contourArea(contour)\n",
    "    x, y, w, h = cv2.boundingRect(contour)\n",
    "    return abs(area - expected_area) <= tolerance and abs(w - h) <= squareness_tolerance\n",
    "\n",
    "print(\"Thresholded image\")\n",
    "filter_contours(thresholded_contours, are_bounding_dimensions_correct)\n",
    "print(\"Opened image\")\n",
    "filter_contours(opened_contours, are_bounding_dimensions_correct)\n",
    "print(\"Canny edge detection\")\n",
    "filter_contours(edged_contours, are_bounding_dimensions_correct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous method still picks up the circle as a valid check box\n",
    "\n",
    "Filtering using shape matching, parameter -- `tolerance`\n",
    "\n",
    "The `template` can be modified to catch shapes different from a 25x25 square."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_contour_square(contour, contour_tolerance=0.0015, square_side=25, area_tolerance=200):\n",
    "    expected_area = square_side * square_side\n",
    "    area = cv2.contourArea(contour)\n",
    "    template = np.array([[[0, 0]], [[0, 1]], [[1, 1]], [[1, 0]]], dtype=np.int32)\n",
    "    return cv2.matchShapes(template, contour, 3, 0.0) <= contour_tolerance and abs(area - expected_area) <= area_tolerance\n",
    "\n",
    "print(\"Thresholded image\")\n",
    "filter_contours(thresholded_contours, is_contour_square)\n",
    "print(\"Opened image\")\n",
    "filter_contours(opened_contours, is_contour_square)\n",
    "print(\"Canny edge detection\")\n",
    "filter_contours(edged_contours, is_contour_square)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}