  We consider the question of learning in general topological vector spaces. By
exploiting known (or parametrized) covariance structures, our Main Theorem
demonstrates that any continuous linear map corresponds to a certain
isomorphism of embedded Hilbert spaces. By inverting this isomorphism and
extending continuously, we construct a version of the Ordinary Least Squares
estimator in absolute generality. Our Gauss-Markov theorem demonstrates that
OLS is a "best linear unbiased estimator", extending the classical result. We
construct a stochastic version of the OLS estimator, which is a continuous
disintegration exactly for the class of "uncorrelated implies independent"
(UII) measures. As a consequence, Gaussian measures always exhibit continuous
disintegrations through continuous linear maps, extending a theorem of the
first author. Applying this framework to some problems in machine learning, we
prove a useful representation theorem for covariance tensors, and show that OLS
defines a good kriging predictor for vector-valued arrays on general index
spaces. We also construct a support-vector machine classifier in this setting.
We hope that our article shines light on some deeper connections between
probability theory, statistics and machine learning, and may serve as a point
of intersection for these three communities.
