  Image classification accuracy on the ImageNet dataset has been a barometer
for progress in computer vision over the last decade. Several recent papers
have questioned the degree to which the benchmark remains useful to the
community, yet innovations continue to contribute gains to performance, with
today's largest models achieving 90%+ top-1 accuracy. To help contextualize
progress on ImageNet and provide a more meaningful evaluation for today's
state-of-the-art models, we manually review and categorize every remaining
mistake that a few top models make in order to provide insight into the
long-tail of errors on one of the most benchmarked datasets in computer vision.
We focus on the multi-label subset evaluation of ImageNet, where today's best
models achieve upwards of 97% top-1 accuracy. Our analysis reveals that nearly
half of the supposed mistakes are not mistakes at all, and we uncover new valid
multi-labels, demonstrating that, without careful review, we are significantly
underestimating the performance of these models. On the other hand, we also
find that today's best models still make a significant number of mistakes (40%)
that are obviously wrong to human reviewers. To calibrate future progress on
ImageNet, we provide an updated multi-label evaluation set, and we curate
ImageNet-Major: a 68-example "major error" slice of the obvious mistakes made
by today's top models -- a slice where models should achieve near perfection,
but today are far from doing so.
