  Large language models (LLMs) have shown strong arithmetic reasoning
capabilities when prompted with Chain-of-Thought (CoT) prompts. However, we
have only a limited understanding of how they are processed by LLMs. To
demystify it, prior work has primarily focused on ablating different components
in the CoT prompt and empirically observing their resulting LLM performance
change. Yet, the reason why these components are important to LLM reasoning is
not explored. To fill this gap, in this work, we investigate ``neuron
activation'' as a lens to provide a unified explanation to observations made by
prior work. Specifically, we look into neurons within the feed-forward layers
of LLMs that may have activated their arithmetic reasoning capabilities, using
Llama2 as an example. To facilitate this investigation, we also propose an
approach based on GPT-4 to automatically identify neurons that imply arithmetic
reasoning. Our analyses revealed that the activation of reasoning neurons in
the feed-forward layers of an LLM can explain the importance of various
components in a CoT prompt, and future research can extend it for a more
complete understanding.
