  This paper enhances and develops bridges between statistics, mechanics, and
geometry. For a given system of points in $\mathbb R^k$ representing a sample
of full rank, we construct an explicit pencil of confocal quadrics with the
following properties: (i) All the hyperplanes for which the hyperplanar moments
of inertia for the given system of points are equal, are tangent to the same
quadrics from the pencil of quadrics. As an application, we develop
regularization procedures for the orthogonal least square method, analogues of
lasso and ridge methods from linear regression. (ii) For any given point $P$
among all the hyperplanes that contain it, the best fit is the tangent
hyperplane to the quadric from the confocal pencil corresponding to the maximal
Jacobi coordinate of the point $P$; the worst fit among the hyperplanes
containing $P$ is the tangent hyperplane to the ellipsoid from the confocal
pencil that contains $P$. The confocal pencil of quadrics provides a universal
tool to solve the restricted principal component analysis restricted at any
given point. Both results (i) and (ii) can be seen as generalizations of the
classical result of Pearson on orthogonal regression. They have natural and
important applications in the statistics of the errors-in-variables models
(EIV). For the classical linear regressions we provide a geometric
characterization of hyperplanes of least squares in a given direction among all
hyperplanes which contain a given point. The obtained results have applications
in restricted regressions, both ordinary and orthogonal ones. For the latter, a
new formula for test statistic is derived. The developed methods and results
are illustrated in natural statistics examples.
