  Self-supervised learning (SSL) has developed rapidly in recent years.
However, most of the mainstream methods are computationally expensive and rely
on two (or more) augmentations for each image to construct positive pairs.
Moreover, they mainly focus on large models and large-scale datasets, which
lack flexibility and feasibility in many practical applications. In this paper,
we propose an efficient single-branch SSL method based on non-parametric
instance discrimination, aiming to improve the algorithm, model, and data
efficiency of SSL. By analyzing the gradient formula, we correct the update
rule of the memory bank with improved performance. We further propose a novel
self-distillation loss that minimizes the KL divergence between the probability
distribution and its square root version. We show that this alleviates the
infrequent updating problem in instance discrimination and greatly accelerates
convergence. We systematically compare the training overhead and performance of
different methods in different scales of data, and under different backbones.
Experimental results show that our method outperforms various baselines with
significantly less overhead, and is especially effective for limited amounts of
data and small models.
