  We study the dynamics of supervised on-line learning of realizable tasks in
feed-forward neural networks. We focus on the regime where the number of
examples used for training is proportional to the number of input channels N.
Using generating function techniques from spin glass theory, we are able to
average over the composition of the training set and transform the problem for
N to infinity to an effective single pattern system, described completely by
the student autocovariance, the student-teacher overlap and the student
response function, with exact closed equations. Our method applies to arbitrary
learning rules, i.e. not necessarily of a gradient-descent type. The resulting
exact macroscopic dynamical equations can be integrated without finite-size
effects up to any degree of accuracy, but their main value is in providing an
exact and simple starting point for analytical approximation schemes. Finally,
we show how, in the region of absent anomalous response and using the
hypothesis that (as in detailed balance systems) the short-time part of the
various operators can be transformed away, one can describe the stationary
state of the network succesfully by a set of coupled equations involving only
four scalar order parameters.
