FM-Bench: Feature Matching Benchmark (BMVC 2019)

Abstract

Matching two images while estimating their relative geometry is a key step in many computer vision applications. For decades, a well-established pipeline, consisting of SIFT, RANSAC, and 8-point algorithm, has been used for this task. Recently, many new approaches were proposed and shown to outperform previous alternatives on standard benchmarks, including the learned features, correspondence pruning algorithms, and robust estimators. However, whether it is beneficial to incorporate them into the classic pipeline is less-investigated. To this end, we are interested in i) evaluating the performance of these recent algorithms in the context of image matching and epipolar geometry estimation, and ii) leveraging them to design more practical registration systems. The experiments are conducted in four large-scale datasets using strictly defined evaluation metrics, and the promising results provide insight into which algorithms suit which scenarios. According to this, we propose three high-quality matching systems and a Coarse-to-Fine RANSAC estimator. They show remarkable performances and have potentials to a large part of computer vision tasks. To facilitate future research, the full evaluation pipeline and the proposed methods are made publicly available.

Publication

Jia-Wang Bian, Yu-Huan Wu, Ji Zhao, Yun Liu, Le Zhang, Ming-Ming Cheng, Ian Reid, An Evaluation of Feature Matchers for Fundamental Matrix Estimation, British Machine Vision Conference (BMVC), 2019 [PDF] [Code] [Chinese Translation]

@inproceedings{bian2019bench,
  title={An Evaluation of Feature Matchers for Fundamental Matrix Estimation},
  author={Bian, Jia-Wang and Wu, Yu-Huan and Zhao, Ji and Liu, Yun and Zhang, Le and Cheng, Ming-Ming and Reid, Ian},
  booktitle= {British Machine Vision Conference (BMVC)},
  year={2019}
}

Contribution

1. We present an evaluation protocol for local features, robust stimators, and especially correspondence pruning algorithms such as [7, 26, 47] which have not been carefully investigated.
2. We evaluate algorithms on four large-scale datasets using strictly defined metrics. The results provide insights into which datasets are particularly challenging and which algorithms suit which scenarios.
3. We propose three high-quality and efficient matching systems, which perform on par with the powerful CODE [23] system but are several orders of magnitude faster.
4. Interestingly, we observe that the recent GC-RANSAC [6] (also USAC [34]) does not show consistently high performance on geometry estimation but permits effective outlier pruning. We hence propose to first use it for outlier removal, and then apply LMedS based estimator [36] for model fitting. The resulting approach, termed Coarse-to-Fine RANSAC, shows significant superiority over other alternatives.

Datasets

Metrics

• %Recall refers to the success ratio of fundamental matrix estimation.
• %Inlier(-m) refers to the inlier rate after (before) RANSAC-like estimators.
• #Corrs(-m) refers to the number of correspondences after (before) RANSAC.

Results

Updated Results

1. By adjusting parameters, GC-RSC shows better results than those reported in the original paper. The new results are shown below:

TUM: 66.40    KITTI: 90.40    T&T: 86.40    CPC:60.00