  Unsupervised pretraining has achieved great success and many recent works
have shown unsupervised pretraining can achieve comparable or even slightly
better transfer performance than supervised pretraining on downstream target
datasets. But in this paper, we find this conclusion may not hold when the
target dataset has very few labeled samples for finetuning, \ie, few-label
transfer. We analyze the possible reason from the clustering perspective: 1)
The clustering quality of target samples is of great importance to few-label
transfer; 2) Though contrastive learning is essential to learn how to cluster,
its clustering quality is still inferior to supervised pretraining due to lack
of label supervision. Based on the analysis, we interestingly discover that
only involving some unlabeled target domain into the unsupervised pretraining
can improve the clustering quality, subsequently reducing the transfer
performance gap with supervised pretraining. This finding also motivates us to
propose a new progressive few-label transfer algorithm for real applications,
which aims to maximize the transfer performance under a limited annotation
budget. To support our analysis and proposed method, we conduct extensive
experiments on nine different target datasets. Experimental results show our
proposed method can significantly boost the few-label transfer performance of
unsupervised pretraining.
