  Mechanistic Interpretability (MI) promises a path toward fully understanding
how neural networks make their predictions. Prior work demonstrates that even
when trained to perform simple arithmetic, models can implement a variety of
algorithms (sometimes concurrently) depending on initialization and
hyperparameters. Does this mean neuron-level interpretability techniques have
limited applicability? We argue that high-dimensional neural networks can learn
low-dimensional representations of their training data that are useful beyond
simply making good predictions. Such representations can be understood through
the mechanistic interpretability lens and provide insights that are
surprisingly faithful to human-derived domain knowledge. This indicates that
such approaches to interpretability can be useful for deriving a new
understanding of a problem from models trained to solve it. As a case study, we
extract nuclear physics concepts by studying models trained to reproduce
nuclear data.
