  It has been over 200 years since Gauss's and Legendre's famous priority
dispute on who discovered the method of least squares. Nevertheless, we argue
that the normal equations are still relevant in many facets of modern
statistics, particularly in the domain of high-dimensional inference. Even
today, we are still learning new things about the law of large numbers, first
described in Bernoulli's Ars Conjectandi 300 years ago, as it applies to high
dimensional inference. The other insight the normal equations provide is the
asymptotic Gaussianity of the least squares estimators. The general form of the
Gaussian distribution, Gaussian processes, are another tool used in modern
high-dimensional inference. The Gaussian distribution also arises via the
central limit theorem in describing weak convergence of the usual least squares
estimators. In terms of high-dimensional inference, we are still missing the
right notion of weak convergence. In this mostly expository work, we try to
describe how both the normal equations and the theory of Gaussian processes,
what we refer to as the "geometry of least squares," apply to many questions of
current interest.
