  Distinct scientific theories can make similar predictions. To adjudicate
between theories, we must design experiments for which the theories make
distinct predictions. Here we consider the problem of comparing deep neural
networks as models of human visual recognition. To efficiently compare models'
ability to predict human responses, we synthesize controversial stimuli: images
for which different models produce distinct responses. We applied this approach
to two visual recognition tasks, handwritten digits (MNIST) and objects in
small natural images (CIFAR-10). For each task, we synthesized controversial
stimuli to maximize the disagreement among models which employed different
architectures and recognition algorithms. Human subjects viewed hundreds of
these stimuli, as well as natural examples, and judged the probability of
presence of each digit/object category in each image. We quantified how
accurately each model predicted the human judgments. The best performing models
were a generative Analysis-by-Synthesis model (based on variational
autoencoders) for MNIST and a hybrid discriminative-generative Joint Energy
Model for CIFAR-10. These DNNs, which model the distribution of images,
performed better than purely discriminative DNNs, which learn only to map
images to labels. None of the candidate models fully explained the human
responses. Controversial stimuli generalize the concept of adversarial
examples, obviating the need to assume a ground-truth model. Unlike natural
images, controversial stimuli are not constrained to the stimulus distribution
models are trained on, thus providing severe out-of-distribution tests that
reveal the models' inductive biases. Controversial stimuli therefore provide
powerful probes of discrepancies between models and human perception.
