  One of the remarkable properties of robust computer vision models is that
their input-gradients are often aligned with human perception, referred to in
the literature as perceptually-aligned gradients (PAGs). Despite only being
trained for classification, PAGs cause robust models to have rudimentary
generative capabilities, including image generation, denoising, and
in-painting. However, the underlying mechanisms behind these phenomena remain
unknown. In this work, we provide a first explanation of PAGs via
\emph{off-manifold robustness}, which states that models must be more robust
off- the data manifold than they are on-manifold. We first demonstrate
theoretically that off-manifold robustness leads input gradients to lie
approximately on the data manifold, explaining their perceptual alignment. We
then show that Bayes optimal models satisfy off-manifold robustness, and
confirm the same empirically for robust models trained via gradient norm
regularization, randomized smoothing, and adversarial training with projected
gradient descent. Quantifying the perceptual alignment of model gradients via
their similarity with the gradients of generative models, we show that
off-manifold robustness correlates well with perceptual alignment. Finally,
based on the levels of on- and off-manifold robustness, we identify three
different regimes of robustness that affect both perceptual alignment and model
accuracy: weak robustness, bayes-aligned robustness, and excessive robustness.
Code is available at \url{https://github.com/tml-tuebingen/pags}.
