This my dissertation project's public code repository. The main goal of was to improve on a novel bio-informatics based stereo matching pipeline.
Stereo matching is a subset within the field of computer vision concerned with finding corresponding pixels between the views of two stereo images. The output of such algorithm is a so-called "disparity map". Within such image, each pixel's color intensity represents the horizontal movement of a pixel with regards to its corresponding pair in the other view. This, knowing the geometry of the stereo-setup can be used to reconstruct depth.
DNA sequence alignment is the process of attempting to find two or more DNA sequences common elements by insertion or deletion of characters in either sequence. One of them, namely, the Niedelman and Wunch algorithm takes advantage on dynamic programming to find the best overlap between two sequences.
As it turns out, an adapted form of this algorithm can be used on image scan-lines as well.
The graphs below shows the performance improvements achieved by introducing different cost-aggregation methods. The tests were run at quater resolution against the metric called "bad1".
The experiments are located under the "experiments/notebooks" folder. Please launch the main binder link above, experiment specific below or download the repository and run it locally. Requirements are listed in the "requirements.txt" file. If run via binder, it is advised to run only the visualisation notebooks for each experiment (the limitations of that environment: 2 cores + 2 GB ram). In that case, please look for the "-VIS" suffix in the notebook's filename.
| Notebook | Matcher used | Dataset | Experiment discussion | Contents | Direct binder link |
|---|---|---|---|---|---|
| ALG_000_EXP_001_Metrics_validation.ipynb | - | Middlebury 2003 | Yes | Unit tests run, then local implementation of Middlebury metrics implementation is cross-checked with published results. |  |
| ALG_001-EXP_001-Baseline-VIS.ipynb | OriginalMatcher | Middlebury 2003 | Yes | 3 diffferent initialisation methods are tested, produced disparity maps are output at the end of the experiment. | |
| ALG_002_EXP_001-Baseline-MacLean_et_al.ipynb | OriginalMatcher3 | Middlebury 2003 | No | Implementing disparity constraints for runtime improvements proposed by MacLean, W. J., Sabihuddin, S. and Islam, J. (2010). | |
| ALG_002_EXP_001-VIS.ipynb | OriginalMatcher3 | Middlebury 2003 | Yes | Runtime and error comparision with baseline, histograms, "hit n miss" errors, 3d visualisation of disparities with intensity values back projected as their surface. | |
| ALG_003_EXP_001-Baseline-MacLean_et_al-Parallel_version.ipynb | ParallelMatcherNWMacLean | Middlebury 2003 | No | Scanline-wise parallelisation of the pipeline in python. Disparity outputs displayed. | |
| ALG_003_EXP_001-VIS.ipynb | ParallelMatcherNWMacLean | Middlebury 2003 | Yes | Runtime and error comparision with baseline, output log file. | |
| ALG_004_EXP_001-Baseline-MacLean_et_al-Numba.ipynb | NumbaSimpleMatcher | Middlebury 2003 | No | Converting the pipeline to Numba. Disparity outputs displayed. | |
| ALG_004_EXP_001-VIS.ipynb | NumbaSimpleMatcher | Middlebury 2003 | Yes | Runtime and error comparision with baseline, parallel baseline, output log file. | |
| ALG_004_EXP_002-Baseline-MacLean_et_al-param_search.ipynb | NumbaSimpleMatcher | Middlebury 2003 | No | Running the pipeline with a range of match values (1-60). | |
| ALG_004_EXP_002-VIS.ipynb | NumbaSimpleMatcher | Middlebury 2003 | Yes | Interactive visual experiment analysis using Plotly.Tabular output. | |
| ALG_005_EXP_001-PatchMatch-2003.ipynb | NumbaPatchMatcher | Middlebury 2003 | No | Implementing constant-weight cost aggregation into the pipeline. A set of window sizes are tested. Tabular output of logged results. | |
| ALG_005_EXP_001-VIS.ipynb | NumbaPatchMatcher | Middlebury 2003 | Yes | Interactive visual experiment analysis using Plotly. | |
| ALG_005_EXP_002-PatchMatch-2003.ipynb | NumbaPatchMatcher | Middlebury 2003 | No | Testing a set of support windows, used edge detectors and gaussian weights. Weights dipslayed, results are in tabular format. | |
| ALG_005_EXP_002-VIS.ipynb | NumbaPatchMatcher | Middlebury 2003 | Yes | Interactive visual experiment analysis using Plotly. | |
| ALG_005_EXP_003-PatchMatch_2014-VIS.ipynb | NumbaPatchMatcher | Middlebury 2014 | Yes | Interactive visual experiment analysis using Plotly. | |
| ALG_006_EXP_001-Bilateral_product_2003.ipynb | NumbaPatchMatcher | Middlebury 2003 | No | Interactive visual experiment analysis using Plotly. | |
| ALG_006_EXP_002-Bilateral_summed_2003.ipynb | NumbaPatchMatcherBilateral | Middlebury 2003 | No | Interactive visual experiment analysis using Plotly. | |
| ALG_006_EXP_003-Bilateral_product_2014-VIS.ipynb | NumbaPatchMatcherBilateral | Middlebury 2014 | No | Interactive visual experiment analysis using Plotly. | |
| ALG_006_EXP_004-Bilateral_summed_2014-VIS.ipynb | NumbaPatchMatcherBilateral | Middlebury 2014 | No | Interactive visual experiment analysis using Plotly. | |
| ALG_006_EXP_005-Comparative_analysis-VIS.ipynb | NumbaPatchMatcherBilateral | Middlebury 2014 | Yes | Comparative interactive visual experiment analysis using Plotly. | |
| ALG_006_EXP_006-Bilateral_summed_truncated_2003.ipynb | NumbaPatchMatcherBilateral (with truncated cost) | Middlebury 2014 | No | Notebook guiding parameter search. | |
| ALG_006_EXP_007-Bilateral_sum_truncated_2014-VIS.ipynb | NumbaPatchMatcherBilateral (with truncated cost) | Middlebury 2014 | No | Interactive visual experiment analysis using Plotly. | |
(test set images are coming soon)
- MacLean, W. J., Sabihuddin, S. and Islam, J. (2010) ‘Leveraging cost matrix structure for hardware implementation of stereo disparity computation using dynamic programming’, in Computer Vision and Image Understanding, pp. 1126–1138. doi: 10.1016/j.cviu.2010.03.011.