  The field of industrial defect detection using machine learning and deep
learning is a subject of active research. Datasets, also called benchmarks, are
used to compare and assess research results. There is a number of datasets in
industrial visual inspection, of varying quality. Thus, it is a difficult task
to determine which dataset to use. Generally speaking, datasets which include a
testing set, with precise labeling and made in real-world conditions should be
preferred. We propose a study of existing benchmarks to compare and expose
their characteristics and their use-cases. A study of industrial metrics
requirements, as well as testing procedures, will be presented and applied to
the studied benchmarks. We discuss our findings by examining the current state
of benchmarks for industrial visual inspection, and by exposing guidelines on
the usage of benchmarks.
