Long-Term Topological Localisation

This repository contains the benchmarking software that was used for our research on long-term localisation, which we presented at the IROS 2014 conference [1], and ICRA 2015 Workshop on Long-Term Autonomy [2]. The software in this repository reproduces experiments described in these two papers.

Overview

The method presented in [1,2] is using the concept of Frequency Map Enhancement (FreMEn) to predict appearance of locations in a hybrid feature-topological map. Predicting appearance for a given time improves robot localisation in changing environments, see the picture and video below for a detailed explanation. The long-term observations of a particular image feature visibility (red,centre), are transferred to the spectral domain (left). Transferring the most prominent spectral components (left, green) to the time domain provides an analytic expression (centre) representing the probability of the feature being visible at a given time (green, centre). This allows to predict the feature visibility at a time when the robot performs self-localization (blue). In this case, the FreMEn model is applied to all features visible at a given location, which allows to predict its appearance for a specific time.

[](https://www.youtube.com/watch?v=aTr9KD4XMGc"FreMEn for Visual Localization") Click the picture to see a detailed explanation - make sure you have sound on.

Automated benchmark

This repository allows for automated benchmarking using the approach described in [3]. If you would like to quickly test any improvements of our method, for the repository, do your changes in the code and open a pull request. A Jenkins server will automatically compile your code, compare it against the method presented in [1,2] and send you a detailed report.

How to use the software

The best way to start with the software is to reproduce the experiments described in [1,2]. You just need to perform a few simple steps:

1. Clone or fork this repository.
2. Download relevant datasets.
3. Compile the package.
4. Build the spatio-temporal models of individual locations.
5. Benchmark the individual models.
6. Summarize the results.

Getting the software

Just fork of clone this repository: https://github.com/gestom/long-term_topological_localisation.git.

Dataset

You need to get a relevant dataset to test with. There are two datasets that we tested our software with, the Witham Wharf RGB-D and the Stromovka-FreLoc. You will need to download them if you want to test the software by repeating the experiments of [1,2]. For quick testing, download the Stromovka-FreLoc and unzip it in the src folder.

Compilation

The project requires openCV, and it was tested with openCV version 3.2. The project compilation is Makefile-based, so simply cd src to the src folder and call make. If successful, the bin folder should contain the fremen binary. To adjust to the partular dataset, change the timeQuantum variable in main/fremen.cpp.

OpenCV issues

In case you are getting into trouble with xfeatures2d.hpp, you might need to recompile openCV.

1. Create a folder to perform the compilation and switch to it: mkdir ~/opencv;cd ~/opencv
2. Download opencv: git clone -b 3.4 --single-branch https://github.com/opencv/opencv.git
3. Download opencv-contrib: git clone -b 3.4 --single-branch https://github.com/opencv/opencv_contrib.git
4. Go to opencv folder, create a build folder and switch to it: mkdir opencv/build;cd opencv/build
5. Tell opencv to compile with the contrib: cmake -DOPENCV_ENABLE_NONFREE:BOOL=ON -DOPENCV_EXTRA_MODULES_PATH=~/opencv/opencv_contrib/modules ~/opencv/opencv
6. Compile it: make -j5.
7. Install it: sudo make install

Build the spatio-temporal models

To build a feature-based model of a particular location, you need to

1. Build a set of features visible at a given location. This is done by calling the fremen binary with the build argument, e.g.: ../bin/fremen build stromovka/training/place_0/ stromovka/maps/place_0.all takes all the images in the stromovka/training/place_0/ and creates a feature-based map of that location in stromovka/maps/place_0.all.
2. Remove unstable features from the build map. This is done by calling the fremen binary with the reduce argument, e.g.: ../bin/fremen reduce stromovka/maps/place_0.all 0.05 stromovka/maps/place_0.red removes features that were visible in less than 5% of images.
3. Rebuild the temporal models of features' visibility. This is done by calling the fremen binary with the recalculate argument, e.g.: ../bin/fremen recalculate stromovka/training/place_0/ stromovka/maps/place_0.red stromovka/maps/place_0.map registers the features of images stored in stromovka/training/place_0/ with the map in stromovka/maps/place_0.red and stores their re-detections in stromovka/maps/place_0.map.

Alternatively, you can simply call the scripts/buildmaps.sh FOLDER, which performs the three aforementioned steps for all locations stored in FOLDER/traninng, i.e.

./scripts/buildmaps.sh stromovka

builds the maps, removes unstable features and creates FreMEN visibility models of all images arranged in stromovka/training/place_X folders.

Benchmark localisation

To run the benchmarks, run the fremen with the test argument. The other arguments are image folder, map_folder, num_features, and fremen_order e.g.:

../bin/fremen test stromovka/testing/ stromovka/maps/ 15 1

tries to perform the localisation using 15 most likely visible features generates by FreMEn order 1 from temporal models stored in stromovka/maps and images stored in stromovka/testing. To perform the test for all FreMEn orders and 1-50 feature numbers, simply call the available test.sh script with the dataset folder as an parameter, e.g.:

./scripts/test.sh stromovka/.

This will fill the stromovka/reports folder with resuts of the localisation.

Evaluate results

Finally, you can call the script

./scripts/summarize.sh stromovka/

and the stromovka/summary folder will contain the results of the localization for different FreMEn orders in text format and as a graph in xfig, png and pdf. The graph stromovka/summary/error.pdf should look like the Figure 3 of [2].

Troubleshoting

opencv2/xfeatures2d.hpp not found Please, see the section OpenCV issues above.

/usr/bin/ld: final link failed Try make clean. If that doesn't help, make sure that all *.o were removed. If you find any though, delete them. After that, try it again.

../bin/fremen: error while loading shared libraries: libopencv_highgui.so.3.4: cannot open shared object file: No such file or directory Try sudo ldconfig -v and launch it again.

###References

1. T.Krajnik, J.P.Fentanes, O.M.Mozos, T.Duckett, J.Ekekrantz, M.Hanheide: Long-term topological localisation for service robots in dynamic environments using spectral maps. In proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2014. [bibtex]
2. T.Krajnik, J.P.Fentanes, J.Santos, K.Kusumam, T.Duckett: FreMEn: Frequency Map Enhancement for Long-Term Mobile Robot Autonomy in Changing Environments. In proceedings of the ICRA Workshop on Visual Place Recognition in Changing Environments (VPRiCE), 2014. [bibtex]