  Perceptual distances between images, as measured in the space of pre-trained
deep features, have outperformed prior low-level, pixel-based metrics on
assessing perceptual similarity. While the capabilities of older and less
accurate models such as AlexNet and VGG to capture perceptual similarity are
well known, modern and more accurate models are less studied. In this paper, we
present a large-scale empirical study to assess how well ImageNet classifiers
perform on perceptual similarity. First, we observe a inverse correlation
between ImageNet accuracy and Perceptual Scores of modern networks such as
ResNets, EfficientNets, and Vision Transformers: that is better classifiers
achieve worse Perceptual Scores. Then, we examine the ImageNet
accuracy/Perceptual Score relationship on varying the depth, width, number of
training steps, weight decay, label smoothing, and dropout. Higher accuracy
improves Perceptual Score up to a certain point, but we uncover a Pareto
frontier between accuracies and Perceptual Score in the mid-to-high accuracy
regime. We explore this relationship further using a number of plausible
hypotheses such as distortion invariance, spatial frequency sensitivity, and
alternative perceptual functions. Interestingly we discover shallow ResNets and
ResNets trained for less than 5 epochs only on ImageNet, whose emergent
Perceptual Score matches the prior best networks trained directly on supervised
human perceptual judgements. The checkpoints for the models in our study are
available at
https://console.cloud.google.com/storage/browser/gresearch/perceptual_similarity.
