  In this paper, we conduct a comprehensive analysis of two dual-branch
(Siamese architecture) self-supervised learning approaches, namely Barlow Twins
and spectral contrastive learning, through the lens of matrix mutual
information. We prove that the loss functions of these methods implicitly
optimize both matrix mutual information and matrix joint entropy. This insight
prompts us to further explore the category of single-branch algorithms,
specifically MAE and U-MAE, for which mutual information and joint entropy
become the entropy. Building on this intuition, we introduce the Matrix
Variational Masked Auto-Encoder (M-MAE), a novel method that leverages the
matrix-based estimation of entropy as a regularizer and subsumes U-MAE as a
special case. The empirical evaluations underscore the effectiveness of M-MAE
compared with the state-of-the-art methods, including a 3.9% improvement in
linear probing ViT-Base, and a 1% improvement in fine-tuning ViT-Large, both on
ImageNet.
