  The problem of function approximation by neural dynamical systems has
typically been approached in a top-down manner: Any continuous function can be
approximated to an arbitrary accuracy by a sufficiently complex model with a
given architecture. This can lead to high-complexity controls which are
impractical in applications. In this paper, we take the opposite, constructive
approach: We impose various structural restrictions on system dynamics and
consequently characterize the class of functions that can be realized by such a
system. The systems are implemented as a cascade interconnection of a neural
stochastic differential equation (Neural SDE), a deterministic dynamical
system, and a readout map. Both probabilistic and geometric (Lie-theoretic)
methods are used to characterize the classes of functions realized by such
systems.
