  The robustness of image classifiers is essential to their deployment in the
real world. The ability to assess this resilience to manipulations or
deviations from the training data is thus crucial. These modifications have
traditionally consisted of minimal changes that still manage to fool
classifiers, and modern approaches are increasingly robust to them. Semantic
manipulations that modify elements of an image in meaningful ways have thus
gained traction for this purpose. However, they have primarily been limited to
style, color, or attribute changes. While expressive, these manipulations do
not make use of the full capabilities of a pretrained generative model. In this
work, we aim to bridge this gap. We show how a pretrained image generator can
be used to semantically manipulate images in a detailed, diverse, and
photorealistic way while still preserving the class of the original image.
Inspired by recent GAN-based image inversion methods, we propose a method
called Adversarial Pivotal Tuning (APT). Given an image, APT first finds a
pivot latent space input that reconstructs the image using a pretrained
generator. It then adjusts the generator's weights to create small yet semantic
manipulations in order to fool a pretrained classifier. APT preserves the full
expressive editing capabilities of the generative model. We demonstrate that
APT is capable of a wide range of class-preserving semantic image manipulations
that fool a variety of pretrained classifiers. Finally, we show that
classifiers that are robust to other benchmarks are not robust to APT
manipulations and suggest a method to improve them. Code available at:
https://captaine.github.io/apt/
