  Even as deep neural networks (DNNs) have achieved remarkable success on
vision-related tasks, their performance is brittle to transformations in the
input. Of particular interest are semantic transformations that model changes
that have a basis in the physical world, such as rotations, translations,
changes in lighting or camera pose. In this paper, we show how differentiable
rendering can be utilized to generate images that are informative, yet
realistic, and which can be used to analyze DNN performance and improve its
robustness through data augmentation. Given a differentiable renderer and a
DNN, we show how to use off-the-shelf attacks from adversarial machine learning
to generate semantic counterexamples -- images where semantic features are
changed as to produce misclassifications or misdetections. We validate our
approach on DNNs for image classification and object detection. For
classification, we show that semantic counterexamples, when used to augment the
dataset, (i) improve generalization performance (ii) enhance robustness to
semantic transformations, and (iii) transfer between models. Additionally, in
comparison to sampling-based semantic augmentation, our technique generates
more informative data in a sample efficient manner.
