  Principal Component Analysis (PCA) is a well known procedure to reduce
intrinsic complexity of a dataset, essentially through simplifying the
covariance structure or the correlation structure. We introduce a novel
algebraic, model-based point of view and provide in particular an extension of
the PCA to distributions without second moments by formulating the PCA as a
best low rank approximation problem. In contrast to hitherto existing
approaches, the approximation is based on a kind of spectral representation,
and not on the real space. Nonetheless, the prominent role of the eigenvectors
is here reduced to define the approximating surface and its maximal dimension.
In this perspective, our approach is close to the original idea of Pearson
(1901) and hence to autoencoders. Since variable selection in linear regression
can be seen as a special case of our extension, our approach gives some
insight, why the various variable selection methods, such as forward selection
and best subset selection, cannot be expected to coincide. The linear
regression model itself and the PCA regression appear as limit cases.
