  Echo State Networks (ESNs) are a class of single-layer recurrent neural
networks with randomly generated internal weights, and a single layer of
tuneable outer weights, which are usually trained by regularised linear least
squares regression. Remarkably, ESNs still enjoy the universal approximation
property despite the training procedure being entirely linear. In this paper,
we prove that an ESN trained on a sequence of observations from an ergodic
dynamical system (with invariant measure $\mu$) using Tikhonov least squares
regression against a set of targets, will approximate the target function in
the $L^2(\mu)$ norm. In the special case that the targets are future
observations, the ESN is learning the next step map, which allows time series
forecasting. We demonstrate the theory numerically by training an ESN using
Tikhonov least squares on a sequence of scalar observations of the Lorenz
system.
