  Linear feature extraction at the presence of nonlinear dependencies among the
data is a fundamental challenge in unsupervised learning. We propose using a
probabilistic Gram-Schmidt (GS) type orthogonalization process in order to
detect and map out redundant dimensions. Specifically, by applying the GS
process over a family of functions which presumably captures the nonlinear
dependencies in the data, we construct a series of covariance matrices that can
either be used to identify new large-variance directions, or to remove those
dependencies from the principal components. In the former case, we provide
information-theoretic guarantees in terms of entropy reduction. In the latter,
we prove that under certain assumptions the resulting algorithms detect and
remove nonlinear dependencies whenever those dependencies lie in the linear
span of the chosen function family. Both proposed methods extract linear
features from the data while removing nonlinear redundancies. We provide
simulation results on synthetic and real-world datasets which show improved
performance over PCA and state-of-the-art linear feature extraction algorithms,
both in terms of variance maximization of the extracted features, and in terms
of improved performance of classification algorithms. Additionally, our methods
are comparable and often outperform the non-linear method of kernel PCA.
