Online tracking of groups of small animals in a planar environment in real time.
xyTracker is a modular Matlab platform for real-time tracking of animals for group learning experiments in planar environments. It tracks animals in real time and performs an online classification of the animal identity to ensure that animal identity is not lost for long tracking experiments. The name "xy" derives from Chinese "XiaoYu" (小鱼).
##Installation
The tracking system includes 3 versions (in order of increasing performance): A purely matlab based version, an OpenCV/matlab version, and an optimized MEX/C++/OpenCV version. All versions have the same matlab interface. The system automatically chooses the fastest available version depending what additional packages are installed on the system.
For the matlab-based version, only matlab (including the image processing toolbox) has to be installed. For the OpenCV versions, one needs to install OpenCV version >=3.0. For the Matlab/OpenCV functionality one needs to install the excellent mexopencv project. Additionally, for grabbing form ptGray cameras one needs to have installed the FlyCapture SDK.
For stimulus presenting functionality, the PsychToolbox has to be installed and in the matlab path. Note, that the xy.Tracker also uses (and includes a copy of) networkComponents and parts of the project munkres-cpp.
For compilation, one needs to specify the Matlab path and the path to mexpoencv (if available) and the path to FlycaptureSDK (if available) to compile with
$ make MATLABDIR=/my/path/to/MATLAB/ FLYCAPINCLUDEDIR=/usr/include/flycapture MEXOPENCVDIR=/mypath/to/mexopencv
To avoid library incompatibility with Matlab's packaged libraries one needs to preload the OpenCV and other libraries. With Linux, eg., put an alias into the .bashrc file:
export PRELOAD_LIBS=/usr/lib64/libstdc++.so.6:/usr/lib64/libtiff.so.5:/usr/lib/libflycapture.so:/usr/lib64/libglibmm-2.4.so:/usr/lib64/libglib-2.0.so.0:/usr/lib64/libsigc-2.0.so.0:/usr/lib/libflycapture.so.2:`ls /usr/local/lib/libopencv*.so |xargs|tr ' ' ':'`
alias matlab='LD_PRELOAD=$PRELOAD_LIBS /opt/MATLAB/R2014b/bin/matlab '
##Test One can test the installation by running
>> xy.Tracker.runTest();
##Usage In MATLAB your need to add the path where the +xy package folder is located. Then, try (without arguments) to get some documentation about the parameters:
>> xy.Tracker;
To track a video file (with ui-dialog and automatic detection of the number of bodies :
>> xyT = xy.Tracker([]);
>> xyT.track();
>> xyT.plot(); % make some result plots
>> xyT.save(); % save the xyT object and all results
To track video file 'myvideo.avi' having 3 animals, write (use 'nindiv',-1 for GUI selection, default is 'nindiv', [] for auto selection of the number of animals) and known approximate length (e.g. 100 px) and width (e.g. 30 px), do
>> xyT = xy.Tracker('myvideo.avi','nindiv',3,bodywidth',30,'bodylength',100);
Set higher level of display and track first 20 seconds
>> xyT.setDisplay(3)
>> xyT.track([0,20])
Or turn off the display for fastest tracking.
>> xyT.setDisplay(0)
>> xyT.track()
For further analysis the results of the tracking process are saved in the xyT.res field which can be obtained by
>> res = xyT.getTrackingResults();
res.pos is a [nFrames x [x,y] x nindiv] array of the position of the animal with unique ID.
res.tracks has a number of fields (controlled be the property xyT.saveFields and changed by the methods xyT.addSaveFields/xyT.removeSaveField).
For example:
>> res.tracks
ans =
id: [2101x5 double]
t: [2101x5 double]
centroid: [2101x5x2 double]
classProb: [2101x5x5 double]
bbox: [2101x5x4 double]
assignmentCost: [2101x5 double]
velocity: [2101x5x2 double]
consecutiveInvisibleCount: [2101x5 double]
segment_Orientation: [2101x5 double]
centerLine: [4-D double]
thickness: [2101x5x7 double]
segment_MinorAxisLength: [2101x5 double]
segment_MajorAxisLength: [2101x5 double]
segment_reversed: [2101x5 double]
identityId: [2101x5 double]
Each of the field as the dimensions [nFrames x nindiv x nDims] where nDims are additional dimensions dependent on the field. For instance, to plot the x-velocity for each tracked body after tracking:
>> res = xyT.getTrackingResults([0,20]);
>> v = xyT.getResField(res,'velocity',1); % 1 means delete "invisible" frames
>> vx = squeeze(v(:,:,1));
>> plot(res.t,vx);
To plot the traces
>> xyT.plotTrace()
Or in 3D (the centroid positions versus time), one can do the following:
>> res = xyT.getTrackingResults();
>> pos = xyT.interpolateInvisible(res,'pos',5); % interpolate lost detections and Boxcar smooth with n=5 frames
>> plot3(res.t,squeeze(p(:,1,:)),squeeze(p(:,2,:)));
>> xlabel('Time [s]'); ylabel('x-position [px]');zlabel('y-position [px]');
Or a more fancy plot of the "centerline" the body-axis of the bodies for each frame. Moments of large amount of "bending" of the fish's body are plotted in red. The position of the head is marked by a circle.
>> xyT.plotCenterLine([10,20]) % 10 to 20 seconds for all identities
Further examples can be found in the 'exps' and 'figs' directories.
For Video grabbing and online tracking currently only PointGrey cameras are supported via the FlyCaptureSDK library. After installation and compilation (edit the Makefile to add the directories) animals can be tracked from a camera with camera index CamIdx and simultaneously saved to raw video file 'myfile.avi' (camera index and ROI can be set with the flycap2 tool)
>> xyT = xy.Tracker({0,'myfile.avi'});
>> xyT.track(); % tracks indefintely. Close tracking window for stop or set end time
>> xyT.save();
>> clear ft; % to stop the background video recording
Tracking stops when the tracking display window is closed and results are saved. Note that the video is continuously recorded in the background until the ft object is cleared.
Although xyTracker is currently only tested with ptGrey cameras, it should be also possible to use the OpenCV VideoCapture method for video grabbing. That is, try (with camara index 0)
>> xyT = xy.Tracker(0);
>> xyT.track();
Using the Matlab PsychToolbox it is possible to do online feedback experiments. To perform a new experiment one has to derive a class from xy.stimulus.Presenter and overload (at least) the "stepStimulus" method. See xy.stimulus.PresenterFlash for an example
To perform the experiment:
>> opts = [];
>> opts.stimulus.presenter = xy.stimulus.Presenter';
>> opts.stmif = 1;
>> opts.nindiv = 3;
>> opts.detector.inverted = 1; % tracking in IR
>> opts.stimulus.screen = 1; % X-window screen number
>> opts.bodywidth = 30;
>> opts.bodylength = 150; % set approx. length and width if known to avoid auto-estimation
>>
>> xyT = xy.Tracker({0,'savevideo.avi'},opts);
>> xyT.setDisplay(0); % to avoid displaying delays
>> xyT.track(); % tracks until xy.stimulus.Presenter/isFinished returns true.
>> xyT.save();
>> clear xyT; % stop background avi recording
For debugging, stimulus presenter objects can be tested on simulated animal tracks without the need for calling the xy.Tracker.track() method:
>> xy.helper.stimulusSimulator(xy.stimulus.Presenter')