  Data augmentation has been proven to be an effective technique for developing
machine learning models that are robust to known classes of distributional
shifts (e.g., rotations of images), and alignment regularization is a technique
often used together with data augmentation to further help the model learn
representations invariant to the shifts used to augment the data. In this
paper, motivated by a proliferation of options of alignment regularizations, we
seek to evaluate the performances of several popular design choices along the
dimensions of robustness and invariance, for which we introduce a new test
procedure. Our synthetic experiment results speak to the benefits of squared l2
norm regularization. Further, we also formally analyze the behavior of
alignment regularization to complement our empirical study under assumptions we
consider realistic. Finally, we test this simple technique we identify
(worst-case data augmentation with squared l2 norm alignment regularization)
and show that the benefits of this method outrun those of the specially
designed methods. We also release a software package in both TensorFlow and
PyTorch for users to use the method with a couple of lines at
https://github.com/jyanln/AlignReg.
