  The Neural Tangent Kernel (NTK) is an important milestone in the ongoing
effort to build a theory for deep learning. Its prediction that sufficiently
wide neural networks behave as kernel methods, or equivalently as random
feature models, has been confirmed empirically for certain wide architectures.
It remains an open question how well NTK theory models standard neural network
architectures of widths common in practice, trained on complex datasets such as
ImageNet. We study this question empirically for two well-known convolutional
neural network architectures, namely AlexNet and LeNet, and find that their
behavior deviates significantly from their finite-width NTK counterparts. For
wider versions of these networks, where the number of channels and widths of
fully-connected layers are increased, the deviation decreases.
