  Minimum redundancy among different elements of an embedding in a latent space
is a fundamental requirement or major preference in representation learning to
capture intrinsic informational structures. Current self-supervised learning
methods minimize a pair-wise covariance matrix to reduce the feature redundancy
and produce promising results. However, such representation features of
multiple variables may contain the redundancy among more than two feature
variables that cannot be minimized via the pairwise regularization. Here we
propose the High-Order Mixed-Moment-based Embedding (HOME) strategy to reduce
the redundancy between any sets of feature variables, which is to our best
knowledge the first attempt to utilize high-order statistics/information in
this context. Multivariate mutual information is minimum if and only if
multiple variables are mutually independent, which suggests the necessary
conditions of factorized mixed moments among multiple variables. Based on these
statistical and information theoretic principles, our general HOME framework is
presented for self-supervised representation learning. Our initial experiments
show that a simple version in the form of a three-order HOME scheme already
significantly outperforms the current two-order baseline method (i.e., Barlow
Twins) in terms of the linear evaluation on representation features.
