  Neural Tangent Kernel (NTK) theory is widely used to study the dynamics of
infinitely-wide deep neural networks (DNNs) under gradient descent. But do the
results for infinitely-wide networks give us hints about the behavior of real
finite-width ones? In this paper, we study empirically when NTK theory is valid
in practice for fully-connected ReLU and sigmoid DNNs. We find out that whether
a network is in the NTK regime depends on the hyperparameters of random
initialization and the network's depth. In particular, NTK theory does not
explain the behavior of sufficiently deep networks initialized so that their
gradients explode as they propagate through the network's layers: the kernel is
random at initialization and changes significantly during training in this
case, contrary to NTK theory. On the other hand, in the case of vanishing
gradients, DNNs are in the the NTK regime but become untrainable rapidly with
depth. We also describe a framework to study generalization properties of DNNs,
in particular the variance of network's output function, by means of NTK theory
and discuss its limits.
