  Deep generative models (DGMs) of images are now sufficiently mature that they
produce nearly photorealistic samples and obtain scores similar to the data
distribution on heuristics such as Frechet Inception Distance (FID). These
results, especially on large-scale datasets such as ImageNet, suggest that DGMs
are learning the data distribution in a perceptually meaningful space and can
be used in downstream tasks. To test this latter hypothesis, we use
class-conditional generative models from a number of model
classes---variational autoencoders, autoregressive models, and generative
adversarial networks (GANs)---to infer the class labels of real data. We
perform this inference by training an image classifier using only synthetic
data and using the classifier to predict labels on real data. The performance
on this task, which we call Classification Accuracy Score (CAS), reveals some
surprising results not identified by traditional metrics and constitute our
contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1
and Top-5 accuracy decrease by 27.9\% and 41.6\%, respectively, compared to the
original data; and conditional generative models from other model classes, such
as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical
Autoregressive Models (HAMs), substantially outperform GANs on this benchmark.
Second, CAS automatically surfaces particular classes for which generative
models failed to capture the data distribution, and were previously unknown in
the literature. Third, we find traditional GAN metrics such as Inception Score
(IS) and FID neither predictive of CAS nor useful when evaluating non-GAN
models. Furthermore, in order to facilitate better diagnoses of generative
models, we open-source the proposed metric.
