  Perceptually Aligned Gradients (PAG) refer to an intriguing property observed
in robust image classification models, wherein their input gradients align with
human perception and pose semantic meanings. While this phenomenon has gained
significant research attention, it was solely studied in the context of
unimodal vision-only architectures. In this work, we extend the study of PAG to
Vision-Language architectures, which form the foundations for diverse
image-text tasks and applications. Through an adversarial robustification
finetuning of CLIP, we demonstrate that robust Vision-Language models exhibit
PAG in contrast to their vanilla counterparts. This work reveals the merits of
CLIP with PAG (CLIPAG) in several vision-language generative tasks. Notably, we
show that seamlessly integrating CLIPAG in a "plug-n-play" manner leads to
substantial improvements in vision-language generative applications.
Furthermore, leveraging its PAG property, CLIPAG enables text-to-image
generation without any generative model, which typically requires huge
generators.
