  Learning invariant representations has been the longstanding approach to
self-supervised learning. However, recently progress has been made in
preserving equivariant properties in representations, yet do so with highly
prescribed architectures. In this work, we propose an invariant-equivariant
self-supervised architecture that employs Capsule Networks (CapsNets) which
have been shown to capture equivariance with respect to novel viewpoints. We
demonstrate that the use of CapsNets in equivariant self-supervised
architectures achieves improved downstream performance on equivariant tasks
with higher efficiency and fewer network parameters. To accommodate the
architectural changes of CapsNets, we introduce a new objective function based
on entropy minimisation. This approach which we name CapsIE (Capsule Invariant
Equivariant Network) achieves state-of-the-art performance across invariant and
equivariant tasks on the 3DIEBench dataset compared to prior equivariant SSL
methods, while outperforming supervised baselines. Our results demonstrate the
ability of CapsNets to learn complex and generalised representations for
large-scale, multi-task datasets compared to previous CapsNet benchmarks. Code
is available at https://github.com/AberdeenML/CapsIE.
