  Computer vision systems have been deployed in various applications involving
biometrics like human faces. These systems can identify social media users,
search for missing persons, and verify identity of individuals. While computer
vision models are often evaluated for accuracy on available benchmarks, more
annotated data is necessary to learn about their robustness and fairness
against semantic distributional shifts in input data, especially in face data.
Among annotated data, counterfactual examples grant strong explainability
characteristics. Because collecting natural face data is prohibitively
expensive, we put forth a generative AI-based framework to construct targeted,
counterfactual, high-quality synthetic face data. Our synthetic data pipeline
has many use cases, including face recognition systems sensitivity evaluations
and image understanding system probes. The pipeline is validated with multiple
user studies. We showcase the efficacy of our face generation pipeline on a
leading commercial vision model. We identify facial attributes that cause
vision systems to fail.
