Image Matching: An Application-oriented Benchmark

Abstract

Image matching approaches have been widely used in computer vision applications in which the image-level matching performance of matchers is critical. However, it has not been well investigated by previous works which place more emphases on evaluating local features. To this end, we present a uniform benchmark with novel evaluation metrics and a large-scale dataset for evaluating the overall performance of image matching methods. The proposed metrics are application-oriented as they emphasize application requirements for matchers. The dataset contains two portions for benchmarking video frame matching and unordered image matching separately, where each portion consists of real-world image sequences and each sequence has a specific attribute. Subsequently, we carry out a comprehensive performance evaluation of different state-of-the-art methods and conduct in-depth analyses regarding various aspects such as application requirements, matching types, and data diversity. Moreover, we shed light on how to choose appropriate approaches for different applications based on empirical results and analyses. Conclusions in this benchmark can be used as general guidelines to design practical matching systems and also advocate potential future research directions in this field.

Publication

JiaWang Bian, Le Zhang, Yun Liu, Wen-Yan Lin, Ming-Ming Cheng and Ian D. Reid, Image Matching: An Application-oriented Benchmark, [PDF][Project Page][Dataset (Baidu Yun)][Dataset (Google Drive)][Github]

 

Benchmark

Application requirements:

  1. Robust.
  2. Accurate.
  3. Sufficient.
  4. Fast.

Evaluation Metrics:

  1. Success ratio / Pose error thresholds (SP).
  2. Averaged verified match numbers / Pose error thresholds (AP).

Dataset

The capture of image sequences are shown in the following figure:

Image sequences are introduced in Table 1:

 

Experimental Results

Six features (SIFT, SURF, ORB, AKAZE, BRISK, KAZE) with two matching methods (RATIO, GMS), in total 12 features , are evaluated on the benchmark. Here, the implementation of features is from OpenCV library and all methods are used with their default thresholds. The experimental results are illustrated below:

The score for robustness, accuracy, and efficiency is illustrated in Table 2:

 

The time consumption of algorithms is listed in Table 3:

 

 

3474total visits,21visits today

Leave a Reply

Your email address will not be published.