  Language Models (LMs) acquire parametric knowledge from their training
process, embedding it within their weights. The increasing scalability of LMs,
however, poses significant challenges for understanding a model's inner
workings and further for updating or correcting this embedded knowledge without
the significant cost of retraining. This underscores the importance of
unveiling exactly what knowledge is stored and its association with specific
model components. Instance Attribution (IA) and Neuron Attribution (NA) offer
insights into this training-acquired knowledge, though they have not been
compared systematically. Our study introduces a novel evaluation framework to
quantify and compare the knowledge revealed by IA and NA. To align the results
of the methods we introduce the attribution method NA-Instances to apply NA for
retrieving influential training instances, and IA-Neurons to discover important
neurons of influential instances discovered by IA. We further propose a
comprehensive list of faithfulness tests to evaluate the comprehensiveness and
sufficiency of the explanations provided by both methods. Through extensive
experiments and analysis, we demonstrate that NA generally reveals more diverse
and comprehensive information regarding the LM's parametric knowledge compared
to IA. Nevertheless, IA provides unique and valuable insights into the LM's
parametric knowledge, which are not revealed by NA. Our findings further
suggest the potential of a synergistic approach of combining the diverse
findings of IA and NA for a more holistic understanding of an LM's parametric
knowledge.
