  Computationally explicit hypotheses of brain function derived from machine
learning (ML)-based models have recently revolutionized neuroscience. Despite
the unprecedented ability of these artificial neural networks (ANNs) to capture
responses in biological neural networks (brains), and our full access to all
internal model components (unlike the brain), ANNs are often referred to as
black-boxes with limited interpretability. Interpretability, however, is a
multi-faceted construct that is used differently across fields. In particular,
interpretability, or explainability, efforts in Artificial Intelligence (AI)
focus on understanding how different model components contribute to its output
(i.e., decision making). In contrast, the neuroscientific interpretability of
ANNs requires explicit alignment between model components and neuroscientific
constructs (e.g., different brain areas or phenomena, like recurrence or
top-down feedback). Given the widespread calls to improve the interpretability
of AI systems, we here highlight these different notions of interpretability
and argue that the neuroscientific interpretability of ANNs can be pursued in
parallel with, but independently from, the ongoing efforts in AI. Certain ML
techniques (e.g., deep dream) can be leveraged in both fields, to ask what
stimulus optimally activates the specific model features (feature visualization
by optimization), or how different features contribute to the model's output
(feature attribution). However, without appropriate brain alignment, certain
features will remain uninterpretable to neuroscientists.
