  A few years ago, the first CNN surpassed human performance on ImageNet.
However, it soon became clear that machines lack robustness on more challenging
test cases, a major obstacle towards deploying machines "in the wild" and
towards obtaining better computational models of human visual perception. Here
we ask: Are we making progress in closing the gap between human and machine
vision? To answer this question, we tested human observers on a broad range of
out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials
across 90 participants. We then investigated a range of promising machine
learning developments that crucially deviate from standard supervised CNNs
along three axes: objective function (self-supervised, adversarially trained,
CLIP language-image training), architecture (e.g. vision transformers), and
dataset size (ranging from 1M to 1B).
  Our findings are threefold. (1.) The longstanding distortion robustness gap
between humans and CNNs is closing, with the best models now exceeding human
feedforward performance on most of the investigated OOD datasets. (2.) There is
still a substantial image-level consistency gap, meaning that humans make
different errors than models. In contrast, most models systematically agree in
their categorisation errors, even substantially different ones like contrastive
self-supervised vs. standard supervised models. (3.) In many cases,
human-to-model consistency improves when training dataset size is increased by
one to three orders of magnitude. Our results give reason for cautious
optimism: While there is still much room for improvement, the behavioural
difference between human and machine vision is narrowing. In order to measure
future progress, 17 OOD datasets with image-level human behavioural data and
evaluation code are provided as a toolbox and benchmark at:
https://github.com/bethgelab/model-vs-human/
