  Deep ensembles (DE) have been successful in improving model performance by
learning diverse members via the stochasticity of random initialization. While
recent works have attempted to promote further diversity in DE via
hyperparameters or regularizing loss functions, these methods primarily still
rely on a stochastic approach to explore the hypothesis space. In this work, we
present Multi-Symmetry Ensembles (MSE), a framework for constructing diverse
ensembles by capturing the multiplicity of hypotheses along symmetry axes,
which explore the hypothesis space beyond stochastic perturbations of model
weights and hyperparameters. We leverage recent advances in contrastive
representation learning to create models that separately capture opposing
hypotheses of invariant and equivariant functional classes and present a simple
ensembling approach to efficiently combine appropriate hypotheses for a given
task. We show that MSE effectively captures the multiplicity of conflicting
hypotheses that is often required in large, diverse datasets like ImageNet. As
a result of their inherent diversity, MSE improves classification performance,
uncertainty quantification, and generalization across a series of transfer
tasks.
