  An explanation of a machine learning model is considered "faithful" if it
accurately reflects the model's decision-making process. However, explanations
such as feature attributions for deep learning are not guaranteed to be
faithful, and can produce potentially misleading interpretations. In this work,
we develop Sum-of-Parts (SOP), a class of models whose predictions come with
grouped feature attributions that are faithful-by-construction. This model
decomposes a prediction into an interpretable sum of scores, each of which is
directly attributable to a sparse group of features. We evaluate SOP on
benchmarks with standard interpretability metrics, and in a case study, we use
the faithful explanations from SOP to help astrophysicists discover new
knowledge about galaxy formation.
