Updates

Update 04/06/2017 Article "Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods" have been accepted for publication in Pattern Recogntion (Elsevier).

Update 31/05/2017 Implementation of the new package saliency_map.py. The package contains an implementation of the FASA algorithm for saliency detection [example] [wiki]

Update 22/03/2017 Fixed a bug in mask_analysis.py and almost completed a more robust version of the CNN head pose estimator.

Update 21/11/2016 New package color_classification.py. The package contains an implementation of the histogram intersection algorithm for colour classification [example]. Read more on my blog post.

Update 04/11/2016 New package motion_tracking.py. The package contains an implementation of Particle Filter, which can be used to follow a target in presence of noisy measurements [example] [video]

Update 01/11/2016 Comparison of three different motion detection algorithms [example] [video]

Update 28/10/2016 New package motion_detection.py. Using the classes in this package it is possible to track moving objects through background subtraction. Possible applications of this algorithm are people detection, vehicle detection and tracking [example]

Update 21/10/2016: New package color_detection.py added. Using the classes inside this package it is possible to detect colors [example], skin [example] and faces [example]

Update 19/10/2016: Working example on how to use CNNs for the pitch estimation [code] Working example on how to use CNNs for both yaw and pitch estimation [code]

Update 05/10/2016: Working example on how to use CNNs for head pose estimation (for the moment only yaw angle) [code]

Update 20/09/2016: Work in progress. The code provided at the moment does not still implement gaze detection. There is a beta version of the class which implements the CNN head pose estimator of the yaw angle [code]. You can use it loading the variables stored in this [file].

What is deepgaze?

Deepgaze is a library for people detection and tracking which uses Convolutional Neural Networks (CNNs) to estimate the Focus of Attention (FOA) of users. The FOA can be approximately estimated finding the head orientation. This is particularly useful when the eyes are covered, or when the user is too far from the camera to grab the eye region with a good resolution. When the eye region is visible it is possible to estimate the gaze direction, which is much more informative and can give a good indication of the FOA. Deepgaze contains useful packages for:

• Head pose estimation (Perspective-n-Point, Convolutional Neural Networks)
• Face detection (Haar Cascade)
• Skin and color detection (Range detection, Backprojection)
• Histogram-based classification (Histogram Intersection)
• Motion detection (Frame differencing, MOG, MOG2)
• Motion tracking (Particle filter)
• Saliency map (FASA)

Deepgaze is based on OpenCV and Tensorflow, some of the best libraries in computer vision and machine learning. Deepgaze is an open source project and any contribution is appreciated, feel free to fork the repository and propose integrations.

This library is the result of my recent work, if you use the library in academic work please cite the following paper:

Patacchiola, M., & Cangelosi, A. (2017). Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods. Pattern Recognition, http://dx.doi.org/10.1016/j.patcog.2017.06.009.

What is a Convolutional Neural Network?

A convolutional neural network (CNN, or ConvNet) is a type of feed-forward artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field. Convolutional networks were inspired by biological processes and are variations of multilayer perceptrons designed to use minimal amounts of preprocessing. They have wide applications in image and video recognition, recommender systems and natural language processing [wiki]

Prerequisites

To use the libray you have to install:

• Numpy [link]

sudo pip install numpy

• OpenCV [link]

sudo apt-get install libopencv-dev python-opencv

• Tensorflow [link]

sudo pip install tensorflow

Some examples may require additional libraries:

• dlib [link]

Installation

Download the repository from [here] or clone it using git:

git clone https://github.com/mpatacchiola/deepgaze.git

To install the package you have to enter in the deepgaze folder and run the setup.py script (it may require root privileges):

cd deepgaze
sudo python setup.py install

If you want to track all the installed files you can record the installation process in a text file using the --record flag:

sudo python setup.py install --record record.txt

Done! Now give a look to the examples below.

Examples

• Head Pose Estimation using the Perspective-n-Point algorithm in OpenCV [code] [video]

• Head Pose Estimation in-the-wild using Perspective-n-Point and dlib face detector [code] [video]

• Head Pose Estimation in images using Convolutional Neural Networks [code]

• Color detection using the Histogram Backprojection algorithm [blog] [code]

• Skin detection using the HSV range color detector [code]

• Face detection using the HSV range color detector [code]

• Motion detection and tracking using frame differencing on a video streaming [code]

• Motion detection and tracking comparison of three algorithms on a video streaming [code] [video]

• Motion tracking with unstable measurements using Particle Filter [code] [video]

• Motion tracking with multiple backprojection for playing chrome's dinosaur game [blog] [code] [video]

• Classify object using their colour fingerprint (histogram intersection) [blog] [code]

• Implementation of the FASA (Fast, Accurate, and Size-Aware Salient Object Detection) algorithm [code] [wiki] [link]

Acknowledgments

• The example "head pose estimation using Perspective-n-Point" is partially based on the C++ version you can find here, and on the workshop "Developing an attention system for a social robot" which was part of the 2nd International Summer School on Social Human-Robot Interaction.

• To implement the Bayes and Particle Filters I followed the great repository of rlabbe which you can find here