  Recent research has reported a performance degradation in self-supervised
contrastive learning for specially designed efficient networks, such as
MobileNet and EfficientNet. A common practice to address this problem is to
introduce a pretrained contrastive teacher model and train the lightweight
networks with distillation signals generated by the teacher. However, it is
time and resource consuming to pretrain a teacher model when it is not
available. In this work, we aim to establish a stronger baseline for
lightweight contrastive models without using a pretrained teacher model.
Specifically, we show that the optimal recipe for efficient models is different
from that of larger models, and using the same training settings as ResNet50,
as previous research does, is inappropriate. Additionally, we observe a common
issu e in contrastive learning where either the positive or negative views can
be noisy, and propose a smoothed version of InfoNCE loss to alleviate this
problem. As a result, we successfully improve the linear evaluation results
from 36.3\% to 62.3\% for MobileNet-V3-Large and from 42.2\% to 65.8\% for
EfficientNet-B0 on ImageNet, closing the accuracy gap to ResNet50 with
$5\times$ fewer parameters. We hope our research will facilitate the usage of
lightweight contrastive models.
