  Latent representations are used extensively for downstream tasks, such as
visualization, interpolation or feature extraction of deep learning models.
Invariant and equivariant neural networks are powerful and well-established
models for enforcing inductive biases. In this paper, we demonstrate that the
inductive bias imposed on the by an equivariant model must also be taken into
account when using latent representations. We show how not accounting for the
inductive biases leads to decreased performance on downstream tasks, and vice
versa, how accounting for inductive biases can be done effectively by using an
invariant projection of the latent representations. We propose principles for
how to choose such a projection, and show the impact of using these principles
in two common examples: First, we study a permutation equivariant variational
auto-encoder trained for molecule graph generation; here we show that invariant
projections can be designed that incur no loss of information in the resulting
invariant representation. Next, we study a rotation-equivariant representation
used for image classification. Here, we illustrate how random invariant
projections can be used to obtain an invariant representation with a high
degree of retained information. In both cases, the analysis of invariant latent
representations proves superior to their equivariant counterparts. Finally, we
illustrate that the phenomena documented here for equivariant neural networks
have counterparts in standard neural networks where invariance is encouraged
via augmentation. Thus, while these ambiguities may be known by experienced
developers of equivariant models, we make both the knowledge as well as
effective tools to handle the ambiguities available to the broader community.
