  Vision foundation models exhibit impressive power, benefiting from the
extremely large model capacity and broad training data. However, in practice,
downstream scenarios may only support a small model due to the limited
computational resources or efficiency considerations. Moreover, the data used
for pretraining foundation models are usually invisible and very different from
the target data of downstream tasks. This brings a critical challenge for the
real-world application of foundation models: one has to transfer the knowledge
of a foundation model to the downstream task that has a quite different
architecture with only downstream target data. Existing transfer learning or
knowledge distillation methods depend on either the same model structure or
finetuning of the foundation model. Thus, naively introducing these methods can
be either infeasible or very inefficient. To address this, we propose a
Task-Driven Model Reprogramming (TDMR) framework. Specifically, we reprogram
the foundation model to project the knowledge into a proxy space, which
alleviates the adverse effect of task mismatch and domain inconsistency. Then,
we reprogram the target model via progressive distillation from the proxy space
to efficiently learn the knowledge from the reprogrammed foundation model. TDMR
is compatible with different pre-trained model types (CNN, transformer or their
mix) and limited target data, and promotes the wide applications of vision
foundation models to downstream tasks in a cost-effective manner. Extensive
experiments on different downstream classification tasks and target model
structures demonstrate the effectiveness of our methods with both CNNs and
transformer foundation models.
