  Despite the proliferation of large vision-language foundation models,
estimation of the learning and forgetting outcomes following fine-tuning of
these models remains largely unexplored. Inspired by work highlighting the
significance of the modality gap in contrastive dual-encoders, we propose the
Inter-Intra Modal Measure (IIMM). Combining terms quantifying the similarity
between image embeddings and the similarity between incorrect image and label
embedding pairs, the IIMM functions as a strong predictor of performance
changes with fine-tuning. Our extensive empirical analysis across four
state-of-the-art vision-language models (CLIP, SigLIP, CoCa, EVA-02-CLIP) and
five fine-tuning techniques (full fine-tuning, BitFit, attention-weight tuning,
LoRA, CLIP-Adapter) demonstrates a strong, statistically significant linear
relationship: fine-tuning on tasks with higher IIMM scores produces greater
in-domain performance gains but also induces more severe out-of-domain
performance degradation, with some parameter-efficient fine-tuning (PEFT)
methods showing extreme forgetting. We compare our measure against transfer
scores from state-of-the-art model selection methods and show that the IIMM is
significantly more predictive of accuracy gains. With only a single forward
pass of the target data, practitioners can leverage this key insight to
heuristically evaluate the degree to which a model can be expected to improve
following fine-tuning. Given additional knowledge about the model's performance
on a few diverse tasks, this heuristic further evolves into a strong predictor
of expected performance changes when training for new tasks.
