  Siamese-network-based self-supervised learning (SSL) suffers from slow
convergence and instability in training. To alleviate this, we propose a
framework to exploit intermediate self-supervisions in each stage of deep nets,
called the Ladder Siamese Network. Our self-supervised losses encourage the
intermediate layers to be consistent with different data augmentations to
single samples, which facilitates training progress and enhances the
discriminative ability of the intermediate layers themselves. While some
existing work has already utilized multi-level self supervisions in SSL, ours
is different in that 1) we reveal its usefulness with non-contrastive Siamese
frameworks in both theoretical and empirical viewpoints, and 2) ours improves
image-level classification, instance-level detection, and pixel-level
segmentation simultaneously. Experiments show that the proposed framework can
improve BYOL baselines by 1.0% points in ImageNet linear classification, 1.2%
points in COCO detection, and 3.1% points in PASCAL VOC segmentation. In
comparison with the state-of-the-art methods, our Ladder-based model achieves
competitive and balanced performances in all tested benchmarks without causing
large degradation in one.
