  The underlying mechanism of neural networks in capturing precise knowledge
has been the subject of consistent research efforts. In this work, we propose a
theoretical approach based on Neural Tangent Kernels (NTKs) to investigate such
mechanisms. Specifically, considering the infinite network width, we
hypothesize the learning dynamics of target models may intuitively unravel the
features they acquire from training data, deepening our insights into their
internal mechanisms. We apply our approach to several fundamental models and
reveal how these models leverage statistical features during gradient descent
and how they are integrated into final decisions. We also discovered that the
choice of activation function can affect feature extraction. For instance, the
use of the \textit{ReLU} activation function could potentially introduce a bias
in features, providing a plausible explanation for its replacement with
alternative functions in recent pre-trained language models. Additionally, we
find that while self-attention and CNN models may exhibit limitations in
learning n-grams, multiplication-based models seem to excel in this area. We
verify these theoretical findings through experiments and find that they can be
applied to analyze language modeling tasks, which can be regarded as a special
variant of classification. Our contributions offer insights into the roles and
capacities of fundamental components within large language models, thereby
aiding the broader understanding of these complex systems.
