  Semi-supervised learning (SSL) addresses the lack of labeled data by
exploiting large unlabeled data through pseudolabeling. However, in the
extremely low-label regime, pseudo labels could be incorrect, a.k.a. the
confirmation bias, and the pseudo labels will in turn harm the network
training. Recent studies combined finetuning (FT) from pretrained weights with
SSL to mitigate the challenges and claimed superior results in the low-label
regime. In this work, we first show that the better pretrained weights brought
in by FT account for the state-of-the-art performance, and importantly that
they are universally helpful to off-the-shelf semi-supervised learners. We
further argue that direct finetuning from pretrained weights is suboptimal due
to covariate shift and propose a contrastive target pretraining step to adapt
model weights towards target dataset. We carried out extensive experiments on
both classification and segmentation tasks by doing target pretraining then
followed by semi-supervised finetuning. The promising results validate the
efficacy of target pretraining for SSL, in particular in the low-label regime.
