Image Matching Benchmark

Abstract

Image matching has been one of the most fundamental and active research areas in computer vision community. In this field, existing evaluation protocols overemphasize feature design and ignore the performance of image level matching. However, the latter is critical for many high-level vision and robotics tasks. To address this, we present the first application-oriented image matching benchmark to facilitate the analysis of matching algorithms in application level. In addition, we construct a large-scale dataset of real-world images that cover a wide range of scene types, on which state-of-the-art matchers (local features with correspondence selection methods) are exhaustively evaluated and analyzed. Moreover, we demonstrate the effectiveness of a simple technique which is readily pluggable into any matching system to improve performance. The results and analysis will provide researchers with a practical guide to utilizing and improving image matching. Finally, we will release our evaluation code on the GitHub.

Publication

JiaWang Bian, Le Zhang, Yun Liu, Wen-Yan Lin, Ming-Ming Cheng and Ian D. Reid, Image Matching Benchmark, [PDF][Project Page][Dataset (Baidu Yun)][Dataset (Google Drive)][Github]

 

Benchmark

Requirements:

  1. Image matching should be robust.
  2. Image matching should be accurate.
  3. Verified correspondences should be sufficient.
  4. Image matching should be fast.

Metrics:

  1. Success ratio / Pose error thresholds (SP).
  2. Averaged verified match numbers / Pose error thresholds (AP).
  3. Consumption Time (CT).

Dataset

The capture of image sequences are shown in the following figure:

Image sequences are introduced in Table 1:

 

Experimental Results

Six features (SIFT, SURF, ORB, AKAZE, BRISK, KAZE) with 2 matching methods (RATIO, GMS), in total 12 feature matchers, are evaluated on the benchmark. Here, the implementation of features is from OpenCV library and all methods are used with their default thresholds. The experimental results are illustrated as below: 

 

Time consumption of algorithms are listed in Table 2:

We discover that matching performance can be improved by just lowing detectors’ threshold and accepting more weak features, and we test this idea by setting up an additional experiment. The results of matchers with the low threshold are shown as below:

 

 

 

975total visits,17visits today

Leave a Reply

Your email address will not be published.