  Most recent work on interpretability of complex machine learning models has
focused on estimating $\textit{a posteriori}$ explanations for previously
trained models around specific predictions. $\textit{Self-explaining}$ models
where interpretability plays a key role already during learning have received
much less attention. We propose three desiderata for explanations in general --
explicitness, faithfulness, and stability -- and show that existing methods do
not satisfy them. In response, we design self-explaining models in stages,
progressively generalizing linear classifiers to complex yet architecturally
explicit models. Faithfulness and stability are enforced via regularization
specifically tailored to such models. Experimental results across various
benchmark datasets show that our framework offers a promising direction for
reconciling model complexity and interpretability.
