  It is common within the deep learning community to first pre-train a deep
neural network from a large-scale dataset and then fine-tune the pre-trained
model to a specific downstream task. Recently, both supervised and unsupervised
pre-training approaches to learning representations have achieved remarkable
advances, which exploit the discriminative knowledge of labels and the
intrinsic structure of data, respectively. It follows natural intuition that
both discriminative knowledge and intrinsic structure of the downstream task
can be useful for fine-tuning, however, existing fine-tuning methods mainly
leverage the former and discard the latter. A question arises: How to fully
explore the intrinsic structure of data for boosting fine-tuning? In this
paper, we propose Bi-tuning, a general learning framework to fine-tuning both
supervised and unsupervised pre-trained representations to downstream tasks.
Bi-tuning generalizes the vanilla fine-tuning by integrating two heads upon the
backbone of pre-trained representations: a classifier head with an improved
contrastive cross-entropy loss to better leverage the label information in an
instance-contrast way, and a projector head with a newly-designed categorical
contrastive learning loss to fully exploit the intrinsic structure of data in a
category-consistent way. Comprehensive experiments confirm that Bi-tuning
achieves state-of-the-art results for fine-tuning tasks of both supervised and
unsupervised pre-trained models by large margins (e.g. 10.7\% absolute rise in
accuracy on CUB in low-data regime).
