  Deep learning has led to remarkable advances in computer vision. Even so,
today's best models are brittle when presented with variations that differ even
slightly from those seen during training. Minor shifts in the pose, color, or
illumination of an object can lead to catastrophic misclassifications.
State-of-the art models struggle to understand how a set of variations can
affect different objects. We propose a framework for instilling a notion of how
objects vary in more realistic settings. Our approach applies the formalism of
Lie groups to capture continuous transformations to improve models' robustness
to distributional shifts. We apply our framework on top of state-of-the-art
self-supervised learning (SSL) models, finding that explicitly modeling
transformations with Lie groups leads to substantial performance gains of
greater than 10% for MAE on both known instances seen in typical poses now
presented in new poses, and on unknown instances in any pose. We also apply our
approach to ImageNet, finding that the Lie operator improves performance by
almost 4%. These results demonstrate the promise of learning transformations to
improve model robustness.
