  Prompt learning is effective for fine-tuning foundation models to improve
their generalization across a variety of downstream tasks. However, the prompts
that are independently optimized along a single modality path, may sacrifice
the vision-language alignment of pre-trained models in return for improved
performance on specific tasks and classes, leading to poorer generalization. In
this paper, we first demonstrate that prompt tuning along only one single
branch of CLIP (e.g., language or vision) is the reason why the misalignment
occurs. Without proper regularization across the learnable parameters in
different modalities, prompt learning violates the original pre-training
constraints inherent in the two-tower architecture. To address such
misalignment, we first propose feature shift, which is defined as the variation
of embeddings after introducing the learned prompts, to serve as an explanatory
tool. We dive into its relation with generalizability and thereafter propose
RESTORE, a multi-modal prompt learning method that exerts explicit constraints
on cross-modal consistency. To be more specific, to prevent feature
misalignment, a feature shift consistency is introduced to synchronize
inter-modal feature shifts by measuring and regularizing the magnitude of
discrepancy during prompt tuning. In addition, we propose a "surgery" block to
avoid short-cut hacking, where cross-modal misalignment can still be severe if
the feature shift of each modality varies drastically at the same rate. It is
implemented as feed-forward adapters upon both modalities to alleviate the
misalignment problem. Extensive experiments on 15 datasets demonstrate that our
method outperforms the state-of-the-art prompt tuning methods without
compromising feature alignment.
