  We analyze architectural features of Deep Neural Networks (DNNs) using the
so-called Neural Tangent Kernel (NTK), which describes the training and
generalization of DNNs in the infinite-width setting. In this setting, we show
that for fully-connected DNNs, as the depth grows, two regimes appear: "order",
where the (scaled) NTK converges to a constant, and "chaos", where it converges
to a Kronecker delta. Extreme order slows down training while extreme chaos
hinders generalization. Using the scaled ReLU as a nonlinearity, we end up in
the ordered regime. In contrast, Layer Normalization brings the network into
the chaotic regime. We observe a similar effect for Batch Normalization (BN)
applied after the last nonlinearity. We uncover the same order and chaos modes
in Deep Deconvolutional Networks (DC-NNs). Our analysis explains the appearance
of so-called checkerboard patterns and border artifacts. Moving the network
into the chaotic regime prevents checkerboard patterns; we propose a
graph-based parametrization which eliminates border artifacts; finally, we
introduce a new layer-dependent learning rate to improve the convergence of
DC-NNs. We illustrate our findings on DCGANs: the ordered regime leads to a
collapse of the generator to a checkerboard mode, which can be avoided by
tuning the nonlinearity to reach the chaotic regime. As a result, we are able
to obtain good quality samples for DCGANs without BN.
