Highly-Accurate Performance Evaluation of Region Detectors

The detection of features is a fundamental step in many computer vision applications. Standing at the beginning of a processing pipeline, the accuracy of such an application is often determined by the accuracy of the detected features. Thus, the development and the evaluation of feature detectors is of high interest in the computer vision community.
Our evaluation is focused on affine invariant region detectors, e.g. (Cordes et al. 2010 ) :

The data set includes:

• Image sequences showing planar scenes with changes in illumination and perspective.
• Highly-accurate homography matrices H for each image pair. The matrices determine the mapping between the images. In order to provide fair results, the matrices are computed without using any feature detector.

For evaluation of region detectors

• Use matlab code provided by Mikolajczyk et al. for detector and descriptor evaluation

NEWS:

• The Hannover Region Detector Evaluation Data Set has been included in the HSequences data set proposed at the ECCV 2016 workshop: Local Features: State of the art, open problems and performance evaluation

• Projection of scene point P to camera planes:

• Scene restricted to planar geometry
• With homogeneous coordinates: H*p=p'

• GRACE sequence [viewpoint change]
  • (1536x1024) (2048x1365) (3072x2048) (3456x2304)

  

• UNDERGROUND sequence [viewpoint change + pattern repetitions]
  • (1536x1024) (2048x1365) (3072x2048) (3456x2304)

  

• COLORS sequence [viewpoint change + pattern repetitions]
  • (1536x1024) (2048x1365) (3072x2048) (3456x2304)

  

• POSTERS sequence [viewpoint change + texture repetitions]
  • (1536x1024) (2048x1365) (3072x2048) (3456x2304)

  

• THERE sequence [strong viewpoint change]
  • (1536x1024) (2048x1365) (3072x2048) (3456x2304)

  

• The data sets contain
  • image sequences in different resolutions (generated from the RAW data of a Canon EOS 350D camera)
  • homographies for the accurate mapping between image pairs of a sequence [1], [2]
• Our data sets overcome the problem of limited accuracy of the data provided by Mikolajczyk et al. (cf. [1] for details)
• Additionally, we provide
  • Homographies with increased accuracy for Mikolajczyk's image sequences [3]
    • for Bark : BarkHomAcc
    • for Graffiti : GrafHomAcc (modified images included)
    • for Bikes : BikesHomAcc
• Easy to use and compare: use Matlab evaluation protocol (repeatability, matching score), with our ppm-images and homography-textfiles; evaluation examples: [4,5,6,7,8,9,11,12]
• If you use the data for evaluation, please cite the corresponding paper(s)
• Our data set has been included in the HBench challenge [10]

• [1] Kai Cordes, Bodo Rosenhahn, and Jörn Ostermann: Increasing the Accuracy of Feature Evaluation Benchmarks Using Differential Evolution, Symposium on Differential Evolution (SDE), IEEE, 2011
• [2] Kai Cordes, Bodo Rosenhahn, and Jörn Ostermann: High-Resolution Feature Evaluation Benchmark, Computer Analysis of Images and Patterns (CAIP), Springer, 2013
• [3] Krystian Mikolajczyk, Cordelia Schmid: Scale & affine invariant interest point detectors. International Journal of Computer Vision (IJCV) 60, 63--86 (2004)
• [4] Hossein Mobahi, C. Lawrence Zitnick, Yi Ma.: Seeing through the blur. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1736--1743 (2012)
• [5] Kai Cordes, Bodo Rosenhahn, and Jörn Ostermann: Localization Accuracy of Interest Point Detectors with Different Scale Space Representations, Advanced Video and Signal-Based Surveillance (AVSS), pp. 247--252, IEEE, 2014
• [6] Kai Cordes, Jörn Ostermann: Increasing the Precision of Junction Shaped Features, International Conference on Machine Vision Applications (MVA), IEEE Computer Society, 2015
• [7] Thao-Ngoc Nguyen and Kazunori Miyata: Multi-scale Region Perpendicular local binary pattern: An Effective Feature for Interest Region Description. The Visual Computer 31(4), 391--406, Springer, 2015
• [8] Kai Cordes, Lukas Grundmann, and Jörn Ostermann: Feature Evaluation with High-Resolution Images, Computer Analysis of Images and Patterns (CAIP), Springer, 2015
• [9] Zhao Chunyang and Zhao Huaici: Accurate and robust feature-based homography estimation using HALF-SIFT and feature localization error weighting. Journal of Visual Communication and Image Representation, 40, 2016: 288-299
• [10]  Vassileios Balntas, Karel Lenc: Local Features: State of the art, open problems and performance evaluation,  Computer Vision – ECCV 2016 Workshops, Springer International Publishing
• [11] Elena Ranguelova, "A Salient Region Detector for structured images," 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pp. 1-8
• [12] Kohei Kawai et al. "Compactification of Affine Transformation Filter Using Tensor Decomposition", ICIP 2018, pp.2162-2166

• Conference Contributions
  • 

    Kai Cordes, Lukas Grundmann, Jörn Ostermann

    Feature Evaluation with High-Resolution Images

    The 16th International Conference on Computer Analysis of Images and Patterns (CAIP), Lecture Notes in Computer Science (LNCS), Springer International Publishing Switzerland, Vol. 9256, pp. 374--386, Valletta, Malta, September 2015, edited by George Azzopardi and Nicolai Petkov

    (pdfDOI) BibTeX
  • 

    Kai Cordes, Bodo Rosenhahn, Jörn Ostermann

    High-Resolution Feature Evaluation Benchmark

    The 15th International Conference on Computer Analysis of Images and Patterns (CAIP), Lecture Notes in Computer Science (LNCS), Springer Berlin/Heidelberg, Vol. 8047, pp. 327-334, York, UK, August 2013, edited by Richard Wilson et al.

    (pdfDOI) BibTeX (paper page)
  • 

    Kai Cordes, Bodo Rosenhahn, Jörn Ostermann

    Increasing the Accuracy of Feature Evaluation Benchmarks Using Differential Evolution

    IEEE Symposium Series on Computational Intelligence (SSCI) - IEEE Symposium on Differential Evolution (SDE), IEEE Computer Society, Paris, France, April 2011

    (pdfDOI) BibTeX (paper page)