  Identifying important neurons for final predictions is essential for
understanding the mechanisms of large language models. Due to computational
constraints, current attribution techniques struggle to operate at neuron
level. In this paper, we propose a static method for pinpointing significant
neurons for different outputs. Compared to seven other methods, our approach
demonstrates superior performance across three metrics. Additionally, since
most static methods typically only identify "value neurons" directly
contributing to the final prediction, we introduce a static method for
identifying "query neurons" which activate these "value neurons". Finally, we
apply our methods to analyze the localization of six distinct types of
knowledge across both attention and feed-forward network (FFN) layers. Our
method and analysis are helpful for understanding the mechanisms of knowledge
storage and set the stage for future research in knowledge editing. We will
release our data and code on github.
