  With the development of deep networks on various large-scale datasets, a
large zoo of pretrained models are available. When transferring from a model
zoo, applying classic single-model based transfer learning methods to each
source model suffers from high computational burden and cannot fully utilize
the rich knowledge in the zoo. We propose \emph{Zoo-Tuning} to address these
challenges, which learns to adaptively transfer the parameters of pretrained
models to the target task. With the learnable channel alignment layer and
adaptive aggregation layer, Zoo-Tuning \emph{adaptively aggregates channel
aligned pretrained parameters} to derive the target model, which promotes
knowledge transfer by simultaneously adapting multiple source models to
downstream tasks. The adaptive aggregation substantially reduces the
computation cost at both training and inference. We further propose lite
Zoo-Tuning with the temporal ensemble of batch average gating values to reduce
the storage cost at the inference time. We evaluate our approach on a variety
of tasks, including reinforcement learning, image classification, and facial
landmark detection. Experiment results demonstrate that the proposed adaptive
transfer learning approach can transfer knowledge from a zoo of models more
effectively and efficiently.
