  The learning of hierarchical representations for image classification has
experienced an impressive series of successes due in part to the availability
of large-scale labeled data for training. On the other hand, the trained
classifiers have traditionally been evaluated on small and fixed sets of test
images, which are deemed to be extremely sparsely distributed in the space of
all natural images. It is thus questionable whether recent performance
improvements on the excessively re-used test sets generalize to real-world
natural images with much richer content variations. Inspired by efficient
stimulus selection for testing perceptual models in psychophysical and
physiological studies, we present an alternative framework for comparing image
classifiers, which we name the MAximum Discrepancy (MAD) competition. Rather
than comparing image classifiers using fixed test images, we adaptively sample
a small test set from an arbitrarily large corpus of unlabeled images so as to
maximize the discrepancies between the classifiers, measured by the distance
over WordNet hierarchy. Human labeling on the resulting model-dependent image
sets reveals the relative performance of the competing classifiers, and
provides useful insights on potential ways to improve them. We report the MAD
competition results of eleven ImageNet classifiers while noting that the
framework is readily extensible and cost-effective to add future classifiers
into the competition. Codes can be found at https://github.com/TAMU-VITA/MAD.
