  Self-supervised learning makes significant progress in pre-training large
models, but struggles with small models. Mainstream solutions to this problem
rely mainly on knowledge distillation, which involves a two-stage procedure:
first training a large teacher model and then distilling it to improve the
generalization ability of smaller ones. In this work, we introduce another
one-stage solution to obtain pre-trained small models without the need for
extra teachers, namely, slimmable networks for contrastive self-supervised
learning (SlimCLR). A slimmable network consists of a full network and several
weight-sharing sub-networks, which can be pre-trained once to obtain various
networks, including small ones with low computation costs. However,
interference between weight-sharing networks leads to severe performance
degradation in self-supervised cases, as evidenced by gradient magnitude
imbalance and gradient direction divergence. The former indicates that a small
proportion of parameters produce dominant gradients during backpropagation,
while the main parameters may not be fully optimized. The latter shows that the
gradient direction is disordered, and the optimization process is unstable. To
address these issues, we introduce three techniques to make the main parameters
produce dominant gradients and sub-networks have consistent outputs. These
techniques include slow start training of sub-networks, online distillation,
and loss re-weighting according to model sizes. Furthermore, theoretical
results are presented to demonstrate that a single slimmable linear layer is
sub-optimal during linear evaluation. Thus a switchable linear probe layer is
applied during linear evaluation. We instantiate SlimCLR with typical
contrastive learning frameworks and achieve better performance than previous
arts with fewer parameters and FLOPs. The code is at
https://github.com/mzhaoshuai/SlimCLR.
