  While recent studies on semi-supervised learning have shown remarkable
progress in leveraging both labeled and unlabeled data, most of them presume a
basic setting of the model is randomly initialized. In this work, we consider
semi-supervised learning and transfer learning jointly, leading to a more
practical and competitive paradigm that can utilize both powerful pre-trained
models from source domain as well as labeled/unlabeled data in the target
domain. To better exploit the value of both pre-trained weights and unlabeled
target examples, we introduce adaptive consistency regularization that consists
of two complementary components: Adaptive Knowledge Consistency (AKC) on the
examples between the source and target model, and Adaptive Representation
Consistency (ARC) on the target model between labeled and unlabeled examples.
Examples involved in the consistency regularization are adaptively selected
according to their potential contributions to the target task. We conduct
extensive experiments on popular benchmarks including CIFAR-10, CUB-200, and
MURA, by fine-tuning the ImageNet pre-trained ResNet-50 model. Results show
that our proposed adaptive consistency regularization outperforms
state-of-the-art semi-supervised learning techniques such as Pseudo Label, Mean
Teacher, and FixMatch. Moreover, our algorithm is orthogonal to existing
methods and thus able to gain additional improvements on top of MixMatch and
FixMatch. Our code is available at
https://github.com/SHI-Labs/Semi-Supervised-Transfer-Learning.
