  Self-supervised learning aims to learn representation that can be effectively
generalized to downstream tasks. Many self-supervised approaches regard two
views of an image as both the input and the self-supervised signals, assuming
that either view contains the same task-relevant information and the shared
information is (approximately) sufficient for predicting downstream tasks.
Recent studies show that discarding superfluous information not shared between
the views can improve generalization. Hence, the ideal representation is
sufficient for downstream tasks and contains minimal superfluous information,
termed minimal sufficient representation. One can learn this representation by
maximizing the mutual information between the representation and the supervised
view while eliminating superfluous information. Nevertheless, the computation
of mutual information is notoriously intractable. In this work, we propose an
objective termed multi-view entropy bottleneck (MVEB) to learn minimal
sufficient representation effectively. MVEB simplifies the minimal sufficient
learning to maximizing both the agreement between the embeddings of two views
and the differential entropy of the embedding distribution. Our experiments
confirm that MVEB significantly improves performance. For example, it achieves
top-1 accuracy of 76.9\% on ImageNet with a vanilla ResNet-50 backbone on
linear evaluation. To the best of our knowledge, this is the new
state-of-the-art result with ResNet-50.
