  Equilibrium states of large layered neural networks with differentiable
activation function and a single, linear output unit are investigated using the
replica formalism. The quenched free energy of a student network with a very
large number of hidden units learning a rule of perfectly matching complexity
is calculated analytically. The system undergoes a first order phase transition
from unspecialized to specialized student configurations at a critical size of
the training set. Computer simulations of learning by stochastic gradient
descent from a fixed training set demonstrate that the equilibrium results
describe quantitatively the plateau states which occur in practical training
procedures at sufficiently small but finite learning rates.
