  Recent studies indicate that large multimodal models (LMMs) are highly robust
against natural distribution shifts, often surpassing previous baselines.
Despite this, domain-specific adaptation is still necessary, particularly in
specialized areas like healthcare. Due to the impracticality of fine-tuning
LMMs given their vast parameter space, this work investigates in-context
learning (ICL) as an effective alternative for enhancing LMMs' adaptability. We
find that the success of ICL heavily relies on the choice of demonstration,
mirroring challenges seen in large language models but introducing unique
complexities for LMMs facing distribution shifts. Our study addresses this by
evaluating an unsupervised ICL method, TopKNearestPR, which selects in-context
examples through a nearest example search based on feature similarity. We
uncover that its effectiveness is limited by the deficiencies of pre-trained
vision encoders under distribution shift scenarios. To address these
challenges, we propose InvariantSelectPR, a novel method leveraging
Class-conditioned Contrastive Invariance (CCI) for more robust demonstration
selection. Specifically, CCI enhances pre-trained vision encoders by improving
their discriminative capabilities across different classes and ensuring
invariance to domain-specific variations. This enhancement allows the encoders
to effectively identify and retrieve the most informative examples, which are
then used to guide LMMs in adapting to new query samples under varying
distributions. Our experiments show that InvariantSelectPR substantially
improves the adaptability of LMMs, achieving significant performance gains on
benchmark datasets, with a 34.2%$\uparrow$ accuracy increase in 7-shot on
Camelyon17 and 16.9%$\uparrow$ increase in 7-shot on HAM10000 compared to the
baseline zero-shot performance.
