  We present a novel nonparametric algorithm for symmetry-based disentangling
of data manifolds, the Geometric Manifold Component Estimator (GEOMANCER).
GEOMANCER provides a partial answer to the question posed by Higgins et al.
(2018): is it possible to learn how to factorize a Lie group solely from
observations of the orbit of an object it acts on? We show that fully
unsupervised factorization of a data manifold is possible if the true metric of
the manifold is known and each factor manifold has nontrivial holonomy -- for
example, rotation in 3D. Our algorithm works by estimating the subspaces that
are invariant under random walk diffusion, giving an approximation to the de
Rham decomposition from differential geometry. We demonstrate the efficacy of
GEOMANCER on several complex synthetic manifolds. Our work reduces the question
of whether unsupervised disentangling is possible to the question of whether
unsupervised metric learning is possible, providing a unifying insight into the
geometric nature of representation learning.
