  Self-interpreting neural networks have garnered significant interest in
research. Existing works in this domain often (1) lack a solid theoretical
foundation ensuring genuine interpretability or (2) compromise model
expressiveness. In response, we formulate a generic Additive Self-Attribution
(ASA) framework. Observing the absence of Shapley value in Additive
Self-Attribution, we propose Shapley Additive Self-Attributing Neural Network
(SASANet), with theoretical guarantees for the self-attribution value equal to
the output's Shapley values. Specifically, SASANet uses a marginal
contribution-based sequential schema and internal distillation-based training
strategies to model meaningful outputs for any number of features, resulting in
un-approximated meaningful value function. Our experimental results indicate
SASANet surpasses existing self-attributing models in performance and rivals
black-box models. Moreover, SASANet is shown more precise and efficient than
post-hoc methods in interpreting its own predictions.
