  Past decades have witnessed a great interest in the distinction and
connection between neural network learning and kernel learning. Recent
advancements have made theoretical progress in connecting infinite-wide neural
networks and Gaussian processes. Two predominant approaches have emerged: the
Neural Network Gaussian Process (NNGP) and the Neural Tangent Kernel (NTK). The
former, rooted in Bayesian inference, represents a zero-order kernel, while the
latter, grounded in the tangent space of gradient descents, is a first-order
kernel. In this paper, we present the Unified Neural Kernel (UNK), which
characterizes the learning dynamics of neural networks with gradient descents
and parameter initialization. The proposed UNK kernel maintains the limiting
properties of both NNGP and NTK, exhibiting behaviors akin to NTK with a finite
learning step and converging to NNGP as the learning step approaches infinity.
Besides, we also theoretically characterize the uniform tightness and learning
convergence of the UNK kernel, providing comprehensive insights into this
unified kernel. Experimental results underscore the effectiveness of our
proposed method.
