  Despite the fast progress in training specialized models for various tasks,
learning a single general model that works well for many tasks is still
challenging for computer vision. Here we introduce multi-task self-training
(MuST), which harnesses the knowledge in independent specialized teacher models
(e.g., ImageNet model on classification) to train a single general student
model. Our approach has three steps. First, we train specialized teachers
independently on labeled datasets. We then use the specialized teachers to
label an unlabeled dataset to create a multi-task pseudo labeled dataset.
Finally, the dataset, which now contains pseudo labels from teacher models
trained on different datasets/tasks, is then used to train a student model with
multi-task learning. We evaluate the feature representations of the student
model on 6 vision tasks including image recognition (classification, detection,
segmentation)and 3D geometry estimation (depth and surface normal estimation).
MuST is scalable with unlabeled or partially labeled datasets and outperforms
both specialized supervised models and self-supervised models when training on
large scale datasets. Lastly, we show MuST can improve upon already strong
checkpoints trained with billions of examples. The results suggest
self-training is a promising direction to aggregate labeled and unlabeled
training data for learning general feature representations.
