  Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics
of neural networks and their generalization bounds. The study on NTK has been
devoted to typical neural network architectures, but it is incomplete for
neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial
neural networks (PNNs). In this work, we derive the finite-width NTK
formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We
prove their equivalence to the kernel regression predictor with the associated
NTK, which expands the application scope of NTK. Based on our results, we
elucidate the separation of PNNs over standard neural networks with respect to
extrapolation and spectral bias. Our two key insights are that when compared to
standard neural networks, PNNs can fit more complicated functions in the
extrapolation regime and admit a slower eigenvalue decay of the respective NTK,
leading to a faster learning towards high-frequency functions. Besides, our
theoretical results can be extended to other types of NNs-Hp, which expand the
scope of our work. Our empirical results validate the separations in broader
classes of NNs-Hp, which provide a good justification for a deeper
understanding of neural architectures.
