This project is largely inspired by the Udactiy Sensor Fusion Nanodegree course project (see https://github.com/udacity/SFND_2D_Feature_Tracking). It is used as a solution project submission for the 2D Feature Tracking module of the mentioned course.

The executable that is implemented in this project matches keypoints of successive camera frames to each other in order to track cars that are located in front of an ego vehicle over multiple frames. For this purpose different keypoint extractors and descriptors are implemented that can be compared to each other. The project makes use of the OpenCV library.

• The keypoint detection methods that are implemented are: Shi-Tomasi, Harris, FAST, BRISK, ORB, AKAZE and SIFT.
• The keypoint description methods that are implemented are: BRISK, BRIEF, ORB, FREAK, AKAZE and SIFT.
• For descriptor matching a brute force method or FLANN can be combined with nearest-neighbor- or kNN-matching.

The software was tested on Ubuntu 18.04.

The following guide was used to install OpenCV (4.3) on Ubuntu 18.04:

• https://linuxize.com/post/how-to-install-opencv-on-ubuntu-18-04/

Various other tutorials can be found on the internet for installing OpenCV on other operating systems.

Additional dependecies are cmake (>= 2.8), make (>= 4.1) and gcc/g++ (>= 5.4).

git clone
cd 2d-feature tracking
mkdir build && cd build
cmake ..
make

./2D_feature_tracking

In order to evaluate different detector/descriptor pairs and give a recommendation on which pair to use, different metrics were defined. The performances of the detectors/descriptors based on these metrics are shown in the following tables.

Number of detected keypoints per frame and total average:

Average time for detection in ms:

Average number of matches for all detector/descriptor combinations:

Average time required for description with corresponding detector in ms:

Some general remarks:

• The AKAZE descriptor works only when also using AKAZE for finding keypoints.
• The ORB descriptor does not work together with the SIFT detector.
• The SIFT descriptor only works with FLANN-based matching, not with the Brute Force Matching approach that was used with all other descriptors.

Performance results:

• The Harris detector, as it was parameterized here, finds the least amount of keypoints (24 in average), whereas BRISK finds the most (271). All other detectors are able to find a similar amount of keypoints betwenn 105 and 165 per frame.
• The FAST method is by far the fastest method for keypoint detection, whereas the SIFT method is by far the slowest. AKAZE and BRISK are relatively slow as well, whereas the execution time of Shi-Tomasi, Harris and ORB lies in between.
• Most descriptors are able to find the most matches in combination with BRISK and AKAZE detectors. It can be observed that the number of matches found depends very much on the number of keypoints that were detected by the detector and given as an input to the descriptor.
• Comparing the execution time of each descriptor it can be observed that FREAK, SIFT and AKAZE are relatively slow, whereas BRISK and BRIEF are very fast. The execution time of the descriptors are largely independent of the keypoint detection method that they are combined with.

A good combination of detector and descriptor should have a realatively short runtime while being able to find a large number of matches. Based on this, the following detector/descriptor pairs are recommended:

• For a good relation between speed and the number of matches use FAST-BRIEF.
• If the number of matches is more important than speed use BRISK-BRIEF.
• As a third alternative with high speed and a realtively large number of matches use BRISK-BRISK.

Be aware that a high number of matches does not necessarily result in a high number of correct matches since some matches could be false positives. However, analyzing the ratio of false positives to total matches would require a deeper analysis that would extend the purpose of this project.