  Aiming towards a holistic understanding of multiple downstream tasks
simultaneously, there is a need for extracting features with better
transferability. Though many latest self-supervised pre-training methods have
achieved impressive performance on various vision tasks under the prevailing
pretrain-finetune paradigm, their generalization capacity to multi-task
learning scenarios is yet to be explored. In this paper, we extensively
investigate the transfer performance of various types of self-supervised
methods, e.g., MoCo and SimCLR, on three downstream tasks, including semantic
segmentation, drivable area segmentation, and traffic object detection, on the
large-scale driving dataset BDD100K. We surprisingly find that their
performances are sub-optimal or even lag far behind the single-task baseline,
which may be due to the distinctions of training objectives and architectural
design lied in the pretrain-finetune paradigm. To overcome this dilemma as well
as avoid redesigning the resource-intensive pre-training stage, we propose a
simple yet effective pretrain-adapt-finetune paradigm for general multi-task
training, where the off-the-shelf pretrained models can be effectively adapted
without increasing the training overhead. During the adapt stage, we utilize
learnable multi-scale adapters to dynamically adjust the pretrained model
weights supervised by multi-task objectives while leaving the pretrained
knowledge untouched. Furthermore, we regard the vision-language pre-training
model CLIP as a strong complement to the pretrain-adapt-finetune paradigm and
propose a novel adapter named LV-Adapter, which incorporates language priors in
the multi-task model via task-specific prompting and alignment between visual
and textual features.
