  Self supervised learning (SSL) has become a very successful technique to
harness the power of unlabeled data, with no annotation effort. A number of
developed approaches are evolving with the goal of outperforming supervised
alternatives, which have been relatively successful. One main issue in SSL is
robustness of the approaches under different settings. In this paper, for the
first time, we recognize the fundamental limits of SSL coming from the use of a
single-supervisory signal. To address this limitation, we leverage the power of
uncertainty representation to devise a robust and general standard hierarchical
learning/training protocol for any SSL baseline, regardless of their
assumptions and approaches. Essentially, using the information bottleneck
principle, we decompose feature learning into a two-stage training procedure,
each with a distinct supervision signal. This double supervision approach is
captured in two key steps: 1) invariance enforcement to data augmentation, and
2) fuzzy pseudo labeling (both hard and soft annotation). This simple, yet,
effective protocol which enables cross-class/cluster feature learning, is
instantiated via an initial training of an ensemble of models through
invariance enforcement to data augmentation as first training phase, and then
assigning fuzzy labels to the original samples for the second training phase.
We consider multiple alternative scenarios with double supervision and evaluate
the effectiveness of our approach on recent baselines, covering four different
SSL paradigms, including geometrical, contrastive, non-contrastive, and
hard/soft whitening (redundancy reduction) baselines. Extensive experiments
under multiple settings show that the proposed training protocol consistently
improves the performance of the former baselines, independent of their
respective underlying principles.
