  Simplicial Embeddings (SEM) are representations learned through
self-supervised learning (SSL), wherein a representation is projected into $L$
simplices of $V$ dimensions each using a softmax operation. This procedure
conditions the representation onto a constrained space during pretraining and
imparts an inductive bias for group sparsity. For downstream classification, we
formally prove that the SEM representation leads to better generalization than
an unnormalized representation. Furthermore, we empirically demonstrate that
SSL methods trained with SEMs have improved generalization on natural image
datasets such as CIFAR-100 and ImageNet. Finally, when used in a downstream
classification task, we show that SEM features exhibit emergent semantic
coherence where small groups of learned features are distinctly predictive of
semantically-relevant classes.
