  We seek to learn a representation on a large annotated data source that
generalizes to a target domain using limited new supervision. Many prior
approaches to this problem have focused on learning "disentangled"
representations so that as individual factors vary in a new domain, only a
portion of the representation need be updated. In this work, we seek the
generalization power of disentangled representations, but relax the requirement
of explicit latent disentanglement and instead encourage linearity of
individual factors of variation by requiring them to be manipulable by learned
linear transformations. We dub these transformations latent canonicalizers, as
they aim to modify the value of a factor to a pre-determined (but arbitrary)
canonical value (e.g., recoloring the image foreground to black). Assuming a
source domain with access to meta-labels specifying the factors of variation
within an image, we demonstrate experimentally that our method helps reduce the
number of observations needed to generalize to a similar target domain when
compared to a number of supervised baselines.
