  Gradient descent yields zero training loss in polynomial time for deep neural
networks despite non-convex nature of the objective function. The behavior of
network in the infinite width limit trained by gradient descent can be
described by the Neural Tangent Kernel (NTK) introduced in
\cite{Jacot2018Neural}. In this paper, we study dynamics of the NTK for finite
width Deep Residual Network (ResNet) using the neural tangent hierarchy (NTH)
proposed in \cite{Huang2019Dynamics}. For a ResNet with smooth and Lipschitz
activation function, we reduce the requirement on the layer width $m$ with
respect to the number of training samples $n$ from quartic to cubic. Our
analysis suggests strongly that the particular skip-connection structure of
ResNet is the main reason for its triumph over fully-connected network.
