  Models that are learned from real-world data are often biased because the
data used to train them is biased. This can propagate systemic human biases
that exist and ultimately lead to inequitable treatment of people, especially
minorities. To characterize bias in learned classifiers, existing approaches
rely on human oracles labeling real-world examples to identify the "blind
spots" of the classifiers; these are ultimately limited due to the human labor
required and the finite nature of existing image examples. We propose a
simulation-based approach for interrogating classifiers using generative
adversarial models in a systematic manner. We incorporate a progressive
conditional generative model for synthesizing photo-realistic facial images and
Bayesian Optimization for an efficient interrogation of independent facial
image classification systems. We show how this approach can be used to
efficiently characterize racial and gender biases in commercial systems.
