  Image classifiers are information-discarding machines, by design. Yet, how
these models discard information remains mysterious. We hypothesize that one
way for image classifiers to reach high accuracy is to first zoom to the most
discriminative region in the image and then extract features from there to
predict image labels, discarding the rest of the image. Studying six popular
networks ranging from AlexNet to CLIP, we find that proper framing of the input
image can lead to the correct classification of 98.91% of ImageNet images.
Furthermore, we uncover positional biases in various datasets, especially a
strong center bias in two popular datasets: ImageNet-A and ObjectNet. Finally,
leveraging our insights into the potential of zooming, we propose a test-time
augmentation (TTA) technique that improves classification accuracy by forcing
models to explicitly perform zoom-in operations before making predictions. Our
method is more interpretable, accurate, and faster than MEMO, a
state-of-the-art (SOTA) TTA method. We introduce ImageNet-Hard, a new benchmark
that challenges SOTA classifiers including large vision-language models even
when optimal zooming is allowed.
