  How do we transfer the relevant knowledge from ever larger foundation models
into small, task-specific downstream models that can run at much lower costs?
Standard transfer learning using pre-trained weights as the initialization
transfers limited information and commits us to often massive pre-trained
architectures. This procedure also precludes combining multiple pre-trained
models that learn complementary information. To address these shortcomings, we
introduce Adaptive Feature Transfer (AFT). Instead of transferring weights, AFT
operates purely on features, thereby decoupling the choice of the pre-trained
model from the smaller downstream model. Rather than indiscriminately
compressing all pre-trained features, AFT adaptively transfers pre-trained
features that are most useful for performing the downstream task, using a
simple regularization that adds minimal overhead. Across multiple vision,
language, and multi-modal datasets, AFT achieves significantly better
downstream performance compared to alternatives with a similar computational
cost. Furthermore, AFT reliably translates improvement in pre-trained models
into improvement in downstream performance, even if the downstream model is
over $50\times$ smaller, and can effectively transfer complementary information
learned by multiple pre-trained models.
