Minutes of ECVnet workshop on Benchmarking

This workshop was attended by 11 delegates: Pascal Brand, LIFIA, France; Henrik I. Christensen, Aalborg University, Denmark; Adrian Clark, University of Essex, England; Patrick Courtney, ITMI-Aptor, France; Jim Crowley, LIFIA-IMAG, France; Wolfgang Foerstner, University Bonn, Germany; Christophe Guizard, Cemagref, France; Claus B. Madsen, Royal Institute of Technology, Sweden; Jan Nielsen, AITEK S.r.l., Italy; Patrick Stelmaszyk, ITMI-Aptor, France; Neil Thacker, University of Sheffield, England.

copies of overheads available from Patrick.Courtney@itmi.cgs.fr

INTRODUCTION

what is benchmarking and why do it ? Patrick Stelmaszyk

Benchmarking is defined here as "qualitative and quantitative evaluation of performance" where performance has many dimensions. Beneficiaries of benchmarking are identified and the usefulness of benchmarking at five key stages of system design (tuning; selection; combination; improvement and resue) are presented.

STATE OF THE ART - THEORETICAL AND NUMERICAL APPROACHES

1: studies in pose estimation stability, Claus Madsen

A study of pose estimation stability is motivated by the observation that some views of 3D objects seem subjectively preferable to others, and that an understanding would be helpful in developing strategies for camera placement. It is subsequently demonstrated that theoretical analysis and experiments result in knowledge about the stability of edge segments and that this knowledge can be used to study the stability of a pose estimation technique. The non-intuitiveness of some of the results obtained emphasise the importance of studying stability issues.

2: 10 con's and pro's for testing vision algorithms, Wolfgang Foerstner

Responses to ten common counter-arguments to benchmarking are presented (it is task-dependent; vision is only one module; vision is too complex; the models used are wrong; measures are not comparable; there is no theory; there are too many parameters; ground truth is too costly; simulations are not reality; testing is not acknowledged), together with recommendation for circumventing them.

3: statistical analysis of correspondence algorithms, Neil Thacker

A procedure for algorithmic modeling is presented. This involves identifying the statistical assumptions underlying the algorithms and sampling the corresponding data distributions. From this, algorithm performance parameters can be calculated directly. This has the advantages of needing less data, allowing cross-comparison; being quick and giving data independent behaviour. Examples are given for both area and feature based correspondance algorithms.

4: characterisation of vision algorithms: an experimental approach, Jan Nielsen

A space pose algorithm is described comprising several cameras, an edge detector, line linker, line grouper and search engine. Benchmarking of this system in terms of success rate and pose error is carried out for variable numbers of observations, distance between observers, layout and object. Recommendations are made regarding this system and benchmarking in general.

5: evaluating reliability and robustness - an example in estimation techniques, Patrick Courtney

An unreliable feature detector is considered to given valid but noisy as well as erroneous outlier data. Criteria for reliability (percentage of estimates within given error limits) and robustness (variation in reliability with proportion of outliers) are described. The performance of various well-known estimation techniques are given with respect to these criteria, indicating when they should be used.

6: benchmarking for recognition and indexing, Henrik Christensen

An analysis of the issues surrounding benchmarking for recognition system is presented, outlining the underlying goals which should be addressed, the diversity of the problem and methods. Possible approaches based on standards and objective measures are presented.

7: testing an industrial camera - practical experiences, Christophe Guizard

A survey of CCD array cameras is presented to address the problem of the lack of available information. The workplan is outlined covering electrical, optical and mechanical considerations. An extension to consider line scan and colour cameras is outlined. Conclusions are made as to future related studies.

8: sharing software and image datasets, Adrian Clark

Reasons for sharing code and data are proposed (difficulty in obtaining data; to permit comparisons; improve efficiency of research). The history of the development of PEIPA - the pilot european image processing archive is also presented. Recommendations are made for future use of this archive for data, software, its use and dissemination.

9: testing image matching procedures, Wolfgang Foerstner

The automation of a satellite image scanning task is described which requires external (controlled tests) and internal evaluation (self diagnosis). A comparison is made of cost with and without self diagnosis using a cost matrix. Some of the issues and problems surrounding self diagnosis are discussed.

afternoon - identification of blocks and proposals for the future

DISCUSSION AND SYNTHESIS

Points of Consensus

• benchmarking is POSSIBLE. There is often a large parameter space to be searched but this is feasible as it may be reduced or searched by running tests over long periods of time. The large space suggests that initial trials, to build confidence and trust, should be run over smaller spaces;
• benchmarking is WORTHWHILE. It can help us to gain knowledge of the algorithms we are studying; develop the field of computer vision; add value to our existing algorithms (since an algorithm which has been subjected to many tests and is still used is to be trusted more than a novel one); and helps us to build complex multi-module systems;
• we have ANSWERS to the classic criticisms against benchmarking. (See talk "10 con's and pro's for testing vision algorithms"). These answers are only the first pass and should evolve as those unconvinced by benchmarking refine their criticisms;
• agreement on some terms of VOCABULARY seem to be missing. The terms signal, noise, reliability, robustness seem to be used in different senses by different workers. It would be useful to prepare a list of definitions. Some progress has already been made in this area for certain terms by other groups.

Is There a Methodology ?

• modularity, to reduce the scale of the problem and permit reuse of algorithms in different applications;
• abstraction in the description of behavior away from application-specific measures and criteria. Tests need to be fair and objective to be useful;
• statistics seems to offer the principal theoretical framework to use;
• mixing theory and practice seems to be important. Theory alone does cannot guarantee a full description of algorithm performance, while empirical testing alone do not allow full understanding of algorithm behaviour.

Several approaches have been successfully demonstrated including:

• qualitative singularity identification;
• mapping input distributions to output distributions;
• statistical modeling of algorithms;

Resistance to Benchmarking - A Cultural Attitude

• promoting techniques at PhD research level;
• rewarding benchmarking activities. Many conference seek "novel" approaches and developing such novelty was felt to be easier and less useful than developing and verifying existing ones;
• using common databases, properly described;
• involving end-users (associations, medicine, photogrammetry, defence who have a real need for usable machine vision) and ask them to promote or co-promote their results.

Proposals for Action

• ask the managment board of ECVnet to organise a full peer-reviewed workshop at ECCV96, covering methodology, testing and replication of results;
• create a special group on "performance characterisation" (a term preferred to benchmarking) within ECVnet;
• create a group of people interested by these methodologies with: each participant to send a list of interested people to ITMI; ITMI to send minutes of the meeting to these people;
• start discussion on a glossary of terms.

For more information please contact : Patrick.Courtney@itmi.cgs.fr