  Pretrained machine learning models need to be adapted to distribution shifts
when deployed in new target environments. When obtaining labeled data from the
target distribution is expensive, few-shot adaptation with only a few examples
from the target distribution becomes essential. In this work, we propose
MixPro, a lightweight and highly data-efficient approach for few-shot
adaptation. MixPro first generates a relatively large dataset by mixing
(linearly combining) pre-trained embeddings of large source data with those of
the few target examples. This process preserves important features of both
source and target distributions, while mitigating the specific noise in the
small target data. Then, it trains a linear classifier on the mixed embeddings
to effectively adapts the model to the target distribution without overfitting
the small target data. Theoretically, we demonstrate the advantages of MixPro
over previous methods. Our experiments, conducted across various model
architectures on 8 datasets featuring different types of distribution shifts,
reveal that MixPro can outperform baselines by up to 7\%, with only 2-4 target
examples.
