  Even with the rise in popularity of over-parameterized models, simple
dimensionality reduction and clustering methods, such as PCA and k-means, are
still routinely used in an amazing variety of settings. A primary reason is the
combination of simplicity, interpretability and computational efficiency. The
focus of this article is on improving upon PCA and k-means, by allowing
non-linear relations in the data and more flexible cluster shapes, without
sacrificing the key advantages. The key contribution is a new framework for
Principal Elliptical Analysis (PEA), defining a simple and computationally
efficient alternative to PCA that fits the best elliptical approximation
through the data. We provide theoretical guarantees on the proposed PEA
algorithm using Vapnik-Chervonenkis (VC) theory to show strong consistency and
uniform concentration bounds. Toy experiments illustrate the performance of
PEA, and the ability to adapt to non-linear structure and complex cluster
shapes. In a rich variety of real data clustering applications, PEA is shown to
do as well as k-means for simple datasets, while dramatically improving
performance in more complex settings.
