Overview

This code is for action recognition using the two stream networks (rgb and optical flow). Each stream is constructed by CNN and LSTM. I implemented this code for practice because I have not written any deep learning code from scratch!

Setup

Environment

• Linux (I used Ubuntu 14.04)
• GPU (I used GTX TitanX x2)
• Keras + TensorFlow (I recommend to install Keras via docker)

Dataset

Download UCF-101 dataset, and set to arbitary directory.

Run

Convert video to RGB images or optical flows

Use scripts/extract_optical_flow.sh in TSN (ECCV2016).

Make train/val split file

Use scripts/build_file_list.sh in TSN (ECCV2016).

Train model and test it

You can train a model and test it.

bash run.sh

TODO

• Modify to accept the various length input sequences (Fixed number of frames are extracted from a sequence now)

Reference

CNN + LSTM

• Keras issue 401
• Keras issue 5527
• LRCN (CVPR2015)

Batch training using "fit_generator" method

• Hatena blog

Two stream CNN

• Two stream CNN (NIPS2014)