  Assumptions about invariances or symmetries in data can significantly
increase the predictive power of statistical models. Many commonly used models
in machine learning are constraint to respect certain symmetries in the data,
such as translation equivariance in convolutional neural networks, and
incorporation of new symmetry types is actively being studied. Yet, efforts to
learn such invariances from the data itself remains an open research problem.
It has been shown that marginal likelihood offers a principled way to learn
invariances in Gaussian Processes. We propose a weight-space equivalent to this
approach, by minimizing a lower bound on the marginal likelihood to learn
invariances in neural networks resulting in naturally higher performing models.
