  A robot's deployment environment often involves perceptual changes that
differ from what it has experienced during training. Standard practices such as
data augmentation attempt to bridge this gap by augmenting source images in an
effort to extend the support of the training distribution to better cover what
the agent might experience at test time. In many cases, however, it is
impossible to know test-time distribution-shift a priori, making these schemes
infeasible. In this paper, we introduce a general approach, called Invariance
Through Latent Alignment (ILA), that improves the test-time performance of a
visuomotor control policy in deployment environments with unknown perceptual
variations. ILA performs unsupervised adaptation at deployment-time by matching
the distribution of latent features on the target domain to the agent's prior
experience, without relying on paired data. Although simple, we show that this
idea leads to surprising improvements on a variety of challenging adaptation
scenarios, including changes in lighting conditions, the content in the scene,
and camera poses. We present results on calibrated control benchmarks in
simulation -- the distractor control suite -- and a physical robot under a
sim-to-real setup.
