02. Tracking
After you have opened BioTracker, you are presented the main interface window.
The main window of the BioTracker.
-
The "File" menu lets you select your input sources as well as tracking modules from a drop down menu. You may choose between a video file, a folder containing a set of pictures or a video stream. In the File menu you can also load additional trackers supplied as *.dll (Windows) or *.so (Linux) files. The binary of the biotracker comes with some trackers which are loaded automatically. These will appear in dropdown-box 5.9.
-
Via the "Edit" menu you can access the undo/redo functionality and the ‘settings’ menu
-
The "View" menu offers toggles to show or hide the toolbars and the right and bottom panel.
-
In the “Extras” menu you can access the application settings, the application infos and the list of available shortcuts.
The Settings Window. Here you can apply configurations which are valid throughout the entire application. Changes will not not take effect until restart of BioTracker.- Video codec for both recording buttons in the main window. Individual codecs each have their own characteristics of speed, compression performance and usability. Do not confuse this with the videos container format - which will always be MKV (Matroska Video). Being able to compress the video in the BioTracker does not imply being able to playback or use it in other 3rd party applications. It is recommended to try out the toolset before relying on it.
- When recording videos it might occur that frames come in faster than the BioTracker can turn them into videos. The default resolution is, that the BioTracker will wait until encoding is done. This will block the entire process, including tracking and acquisition of further frames, but assures that every frame which passed through the BioTracker is in the video. By clicking this checkbox this behaviour is changed into discarding any images not processed in time. This changes the tradeoff from integrity to performance (important for real time tracking applications).
- This checkbox toggles what will be recorded in the Tracking Window with the last recording button.
- The number N determines that only every N’th frame from the source video shall be processed. Hence the default N=1 means that every frame will be processed. N=5 for instance means that only every 5’th image in the source is processed, skipping its four predecessors.
- The BioTracker writes the data provided by the tracking algorithm to a file. To do so it is using a generic file exporter which can be selected in this dropdown-box. Currently writing as CSV, JSON and binary files is possible. The CSV files are readable by every common version of Microsoft Excel and Open Office Calc. They will be stored in the “Tracks” subfolder of the BioTracker. Tracking output files are named according to the source video and the current timestamp.
The version information of the Biotracker, as well as other included libraries are displayed.
The list of all currently available shortcuts is displayed.
-
Controls
- Undo action
- Redo action
- Load a video file
- Load a set of pictures
- Load a camera stream
- Load a tracker
- Load trajectory data from file. Needs to fit the currently loaded tracker.
- Save the trajectory to a file
- The dropdown-box lists all currently loaded tracking modules. Tracking modules located in the “Plugins” subdirectory of the BioTracker will be automatically loaded. You can select the tracking algorithm of your choice here or load additional ones via the “File” menu. Doing so will add the tracking algorithm specific user interface in (12).
- Start the tracking. If this is not activated, the tracking plugin will not be triggered every next frame. This switch automatically gets activated on starting an experiment.
-
Contains the tools to modify trajectories and entities and to add annotations to the current frame.
- Add a new trajectory to the data structure and show it on the medium
- Delete all currently selected trajectories
- Switch the ID of two selected trajectories
- Select all entities
- Change the border color of all selected entities
- Change the fill color of all selected entities
- Add a yellow label annotation to the current frame
- Add a yellow arrow annotation to the current frame
- Add a yellow rectangular annotation to the current frame
- Add a yellow elliptical annotation to the current frame
- Delete the currently selected annotation
-
This slider lets you seek in a video file or set of pictures once a source video is loaded. This is not available when using a camera (live stream).
-
This is the mediaplayer that grants you control over the source video. These buttons will be greyed out and dysfunctional when no source video is selected. The first button steps one frame in the source video back. The ‘play button’ plays back the video as fast as possible, to the limit of the source fps. It will turn into a pause button while playing so you can pause playback at any time. The stop button stops the source video and resets it to frame 1 . The next button skips to the next respective frame. The next three buttons will help you save the things you see in the tracking view to files. The camera button saves the currently displayed image to a file in the subfolder “screenshots” in the biotracker directory. The record button with the camera symbol will record the source video without any overlays. I.e. if you are running a tracker which is currently displaying trackpoints on top of the video, these will not be saved in the video recorded by this button. The last button records a video of what you see, including the previously mentioned tracking overlay. By default, it records the video and overlay at original resolution. You may change this in the “Settings Window” to record the scaled output, I.e. you can set it to record exactly what you see, zoom included. Videos are saved in the “Videos” subdirectory of BioTracker.
-
Displays source fps and currently played fps. The source fps is the fps defined by the source video. This means that if you are using a camera stream which is configured to run at X fps, X will be the source fps. However, tracking can be slow depending on your hardware, source video and tracking algorithm. For this reason the BioTracker might not be able to keep up to the source fps and run slower. This is not harmful to integrity of tracking results. But if you are tracking directly from camera stream, some frames will be dropped if the current fps are less than source fps. There is also an input method to put an upper limit to the playback speed of the BioTracker.
-
Selecting a tracker using (5.9) will not start tracking procedure yet. To do so, tick the button (5.10) and start playing the video. Alternatively, just click “start experiment” in the experiment tab (10). For details on how to use the individual tracking module, see the sections on them. The 'Finalize trial' button will finalize the current tracking session and trial. For CSV output this means the csv file is re-written in chronological order and replaces the current file. The file handle will be released so the operating system can move the file away. In any case the trajectory data structure will be cleared and all tracks removed. This effectively starts an entirely new tracking session on the source video.
-
In this tab you may adjust what and how the BioTracker shall display things to you. See visualization options.
-
Tracker: Every tracking module will have its specific user interface which is displayed here. For details the the section for the corresponding tracking plugin.
-
How to use BioTracker: Shows a short manual on how to use the BioTracker stepwise.
-
Shows notifications in a text browser. These can also be created by the tracker. Colors:
- Messages (black)
- Warnings (orange)
- Errors (red)
The visualization options tab
- Check this box to have the tracking visualized by the core. Not displaying it at all may hinder your work with the tracker, but slightly improve performance.
- This box provides access to configurations on how the individual entities are visualized over time and which shape. It also provides some general access functions
- The ‘number of tracks’ displays how many tracks are currently processed.
- (Color subbox) In the first row you can set a color for the inner area of a shape, either for the selected set or for all of them at once.
- (Color subbox) In the second row you can set a color for the borders of a shape, either for the selected set or for all of them at once.
- A trace is a visualization of a subjects position of the previous timesteps. In this view you may adjust this history and whether to see it at all.
- Tracing style denotes the type of marker. Available are points, arrowed lines and straight lines.
- Number N of past trackpoints to be visualized for each track. E.g. setting this to 5 will cause it to visualize the past 5 trackpoints using the method from (a).
- Here 1 means every timestep will be visualized in the trace. 2 means every second, etc.. The total number of trackpoints will be the number specified in (b) divided by the number specified in (c).
- The “Area descriptor” box offers functionality to adjust rectification and tracking area.
- Define the size (width and height) of the tracking area.
- Toggle visualization of rectification rectangle
- Toggle visualization of tracking area shape
- Toggle tracking area shape between rectangular and ellipsoid shape.
There is a switch to toggle advanced options.
The enabled and disabled switch to toggle advanced options in the ‘Visualization options’ tab
Advanced options of the options tab
- (‘Dimensions’ groupbox)
- Dimension setter buttons for all or selected tracks
- Button to revert the dimensions of all tracks back to its defaults
- Checkbox to toggle entities orientation lines
- Checkbox to toggle ID in the center of each entity
- (‘Tracer dimensions’ groupbox)
- Spinbox to control the tracer size by ratio to entity (0,5 means half size)
- Checkbox to toggle the orientation line of each tracer
- (‘Miscellaneous’ groupbox)
- Checkbox to toggle antialiasing for all entities (at the cost of performance)
- Checkbox to toggle antialiasing for the whole media view frame (at the cost of performance)
You may select an entity by left clicking it. All selected entities will be highlighted by a solid outline. Multiple entities can be selected by left click into the view and drag the mouse to select a set of entities. Entities can always be added to the selection by holding the left “ctrl” key while clicking the entity.
Menu appearing when right-clicking the tracking area
Upon right click on free area you will presented the menu as seen on the left hand side.
- “Add entity here” will add a new entity at the currently clicked point.
- “Swap ID’s” will swap the ID’s of two selected entities. Only available if exactly two are selected.
- “Unmark all…” will clear the current selection.
- “Remove selected shapes” removes the currently selected shape or shapes.
Right click on an entity will yield a menu as seen on the right hand side.
- “Show full info” opens up a Window with information about the entity in this frame
- “Change fill color” changes the color of the entities area
- “Change border color” changes the color of the entities area border
- “Transparency” changes the transparency of the entity
- “Remove track” removes the whole track, no matter whether selected or not.
- “Remove track entity” only removes the right clicked entity, no matter whether selected or not.
- “Fix Track” fixes the data structure of the track, so the tracker does not change it (but you can)
- “Mark” and “Unmark” mark or unmark the right clicked entity, respectively.
- The “Morph into…” menu changes the type of the entity into compatible types. Current compatibile types are point, rectangle and ellipse.
To start your tracking project, you need to select a source which can be a picture sequence, a previously recorded video file or a camera stream.
A set of images may be tracked using the context menu “File → Open Pictures...“. In the file dialogue multiple pictures may be selected, which will be tracked sequentially according to their filename ascendingly.
The software uses OpenCV 3.2 to decode videos and images. Hence the supported formats for images are as per documentation.
- Windows bitmaps - .bmp, .dib
- JPEG files - .jpeg, .jpg, .jpe
- JPEG 2000 files - .jp2
- Portable Network Graphics - .png
- Portable image format - .pbm, .pgm, .ppm
- Sun rasters - .sr, .ras
- TIFF files - .tiff, .tif
A video may be tracked using the context menu “File → Open Video...“. The framerate will be determined from the video file itself. The BioTracker only include free codecs for reading. However it supports most common container formats, such as mp4, avi or mkv. To read commonly used proprietary codecs, such as X264, you need to install these codecs from a third party. On Linux these can be acquired via repositories and are supplied via gstreamer. On Windows these can be acquired by downloading and installing codec packs and provided via directShow.
Selecting “File → Open Camera…” in the context menu opens a form for selecting and configuring the camera input.
Any generic USB camera may be used for displaying and tracking.
“Preview” will yield a preview image similar the example under given configuration, if the camera could be loaded properly.
The preview image is stretched to a smaller resolution and might not retain the given aspect ratio. Resolution and FPS values “Default”
mean the camera will not be reconfigured and remains at whatever configuration it comes up at loading. This enables the user to set these
properties in an external tool and just leave them like that for use in the BioTracker. Any other values will be set to the camera. Faulty values will result in failing to open the camera for recording.
Annotation examples.
The user can manually annotate videos and images with vectors or labels. The annotations will be saved to a .csv file that corresponds to the image or video filename with the added suffix “.annotations.csv”. Each annotation will be fixed to the frame it is created in. In other frames it will be greyed out. Annotations have circled handles which can be used to change the position of the annotation. To remove an annotation click on one of its handles, which will be marked red, and hit the ALT + delete keys or use the ‘Delete selected annotation’ tool in the toolbar (6) To place a vector press ALT + A, then left click into the video to start the arrow. While keeping the mouse button pressed, drag the cursor to the end point of the vector and release the mouse button. To place a label press ALT + L, left click into the video and release the mouse button. Press ALT + R to create a rectangular and ALT + E to create an elliptical Annotation. There also exists a tool in the left toolbar (6) for each type of annotation.
Upon loading a video file and enabling “display of rectification” and “tracking area” in the view tab, four blue squares will appear connected by fine black lines. This rectangle defines the rectification. Often the camera view is not perfectly orthogonal to the recorded area. In fact the camera could be viewing the tracked area from an arbitrary angle and distance. This original image, which is perspectively distorted, is the input to the tracker. The tracker wants to estimate - according to it’s algorithm and limitations - an object’s position in the source video. To do so it is useful for the tracker to use real world distances instead of pixels to get a realistic measure for distances. The user might be interested in an object's real world coordinates instead of pixels as well. To achieve this a rectification is necessary. In the BioTracker a rectification is defined via four points of known distance in the real world. These four points define a coordinate system in centimeters which can be used by the tracker and which are output for the user. Note that when recording at an angle, a rectangle in the real world might look distorted on the video. This can be corrected by selecting the corner points of the distorted rectangle for the rectification. The red corners define the tracking area. This definition is passed to the tracking plugin, which may use it to check whether a point is in- or outside the given area. This way points in irrelevant portions of the source video may be ignored to speed up tracking. Note that this behaviour is tracker dependent.
There are several tracking modules currently available and others will be available in future as they can be developed independent of the main BioTracker software. The main package of BioTracker already contains two tracking modules while other can be downloaded at github releases page.
The following tracking modules are currently available:
- Background subtraction tracker
- Lucas-Kanade-tracker