  Data augmentation is a key element in training high-dimensional models. In
this approach, one synthesizes new observations by applying pre-specified
transformations to the original training data; e.g.~new images are formed by
rotating old ones. Current augmentation schemes, however, rely on manual
specification of the applied transformations, making data augmentation an
implicit form of feature engineering. With an eye towards true end-to-end
learning, we suggest learning the applied transformations on a per-class basis.
Particularly, we align image pairs within each class under the assumption that
the spatial transformation between images belongs to a large class of
diffeomorphisms. We then learn a class-specific probabilistic generative models
of the transformations in a Riemannian submanifold of the Lie group of
diffeomorphisms. We demonstrate significant performance improvements in
training deep neural nets over manually-specified augmentation schemes. Our
code and augmented datasets are available online.
