  Pre-trained language models (PLMs) contain vast amounts of factual knowledge,
but how the knowledge is stored in the parameters remains unclear. This paper
delves into the complex task of understanding how factual knowledge is stored
in multilingual PLMs, and introduces the Architecture-adapted Multilingual
Integrated Gradients method, which successfully localizes knowledge neurons
more precisely compared to current methods, and is more universal across
various architectures and languages. Moreover, we conduct an in-depth
exploration of knowledge neurons, leading to the following two important
discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which
store factual knowledge in a form that transcends language. We design
cross-lingual knowledge editing experiments, demonstrating that the PLMs can
accomplish this task based on language-independent neurons; (2) The discovery
of Degenerate Knowledge Neurons, a novel type of neuron showing that different
knowledge neurons can store the same fact. Its property of functional overlap
endows the PLMs with a robust mastery of factual knowledge. We design
fact-checking experiments, proving that the degenerate knowledge neurons can
help the PLMs to detect wrong facts. Experiments corroborate these findings,
shedding light on the mechanisms of factual knowledge storage in multilingual
PLMs, and contribute valuable insights to the field. The code is available at
https://github.com/heng840/AMIG.
