  It is common in machine learning and statistics to use symmetries derived
from expert knowledge to simplify problems or improve performance, using
methods like data augmentation or penalties. In this paper we consider the
unsupervised and semi-supervised problems of learning such symmetries in a
distribution directly from data in a model-free fashion. We show that in the
worst case this problem is as difficult as the graph automorphism problem.
However, if we restrict to the case where the covariance matrix has unique
eigenvalues, then the eigenvectors will also be eigenvectors of the symmetry
transformation. If we further restrict to finding orthogonal symmetries, then
the eigenvalues will be either be 1 or -1, and the problem reduces to
determining which eigenvectors are which. We develop and compare theoretically
and empirically the effectiveness of different methods of selecting which
eigenvectors should have eigenvalue -1 in the symmetry transformation, and
discuss how to extend this approach to non-orthogonal cases where we have
labels
