  We address the challenge of getting efficient yet accurate recognition
systems with limited labels. While recognition models improve with model size
and amount of data, many specialized applications of computer vision have
severe resource constraints both during training and inference. Transfer
learning is an effective solution for training with few labels, however often
at the expense of a computationally costly fine-tuning of large base models. We
propose to mitigate this unpleasant trade-off between compute and accuracy via
semi-supervised cross-domain distillation from a set of diverse source models.
Initially, we show how to use task similarity metrics to select a single
suitable source model to distill from, and that a good selection process is
imperative for good downstream performance of a target model. We dub this
approach DistillNearest. Though effective, DistillNearest assumes a single
source model matches the target task, which is not always the case. To
alleviate this, we propose a weighted multi-source distillation method to
distill multiple source models trained on different domains weighted by their
relevance for the target task into a single efficient model (named
DistillWeighted). Our methods need no access to source data, and merely need
features and pseudo-labels of the source models. When the goal is accurate
recognition under computational constraints, both DistillNearest and
DistillWeighted approaches outperform both transfer learning from strong
ImageNet initializations as well as state-of-the-art semi-supervised techniques
such as FixMatch. Averaged over 8 diverse target tasks our multi-source method
outperforms the baselines by 5.6%-points and 4.5%-points, respectively.
