  Substantial work indicates that the dynamics of neural networks (NNs) is
closely related to their initialization of parameters. Inspired by the phase
diagram for two-layer ReLU NNs with infinite width (Luo et al., 2021), we make
a step towards drawing a phase diagram for three-layer ReLU NNs with infinite
width. First, we derive a normalized gradient flow for three-layer ReLU NNs and
obtain two key independent quantities to distinguish different dynamical
regimes for common initialization methods. With carefully designed experiments
and a large computation cost, for both synthetic datasets and real datasets, we
find that the dynamics of each layer also could be divided into a linear regime
and a condensed regime, separated by a critical regime. The criteria is the
relative change of input weights (the input weight of a hidden neuron consists
of the weight from its input layer to the hidden neuron and its bias term) as
the width approaches infinity during the training, which tends to $0$,
$+\infty$ and $O(1)$, respectively. In addition, we also demonstrate that
different layers can lie in different dynamical regimes in a training process
within a deep NN. In the condensed regime, we also observe the condensation of
weights in isolated orientations with low complexity. Through experiments under
three-layer condition, our phase diagram suggests a complicated dynamical
regimes consisting of three possible regimes, together with their mixture, for
deep NNs and provides a guidance for studying deep NNs in different
initialization regimes, which reveals the possibility of completely different
dynamics emerging within a deep NN for its different layers.
