  Automatically discovering failures in vision models under real-world settings
remains an open challenge. This work demonstrates how off-the-shelf,
large-scale, image-to-text and text-to-image models, trained on vast amounts of
data, can be leveraged to automatically find such failures. In essence, a
conditional text-to-image generative model is used to generate large amounts of
synthetic, yet realistic, inputs given a ground-truth label. Misclassified
inputs are clustered and a captioning model is used to describe each cluster.
Each cluster's description is used in turn to generate more inputs and assess
whether specific clusters induce more failures than expected. We use this
pipeline to demonstrate that we can effectively interrogate classifiers trained
on ImageNet to find specific failure cases and discover spurious correlations.
We also show that we can scale the approach to generate adversarial datasets
targeting specific classifier architectures. This work serves as a
proof-of-concept demonstrating the utility of large-scale generative models to
automatically discover bugs in vision models in an open-ended manner. We also
describe a number of limitations and pitfalls related to this approach.
