  We propose universally slimmable self-supervised learning (dubbed as US3L) to
achieve better accuracy-efficiency trade-offs for deploying self-supervised
models across different devices. We observe that direct adaptation of
self-supervised learning (SSL) to universally slimmable networks misbehaves as
the training process frequently collapses. We then discover that temporal
consistent guidance is the key to the success of SSL for universally slimmable
networks, and we propose three guidelines for the loss design to ensure this
temporal consistency from a unified gradient perspective. Moreover, we propose
dynamic sampling and group regularization strategies to simultaneously improve
training efficiency and accuracy. Our US3L method has been empirically
validated on both convolutional neural networks and vision transformers. With
only once training and one copy of weights, our method outperforms various
state-of-the-art methods (individually trained or not) on benchmarks including
recognition, object detection and instance segmentation. Our code is available
at https://github.com/megvii-research/US3L-CVPR2023.
