  A salient approach to interpretable machine learning is to restrict modeling
to simple models. In the Bayesian framework, this can be pursued by restricting
the model structure and prior to favor interpretable models. Fundamentally,
however, interpretability is about users' preferences, not the data generation
mechanism; it is more natural to formulate interpretability as a utility
function. In this work, we propose an interpretability utility, which
explicates the trade-off between explanation fidelity and interpretability in
the Bayesian framework. The method consists of two steps. First, a reference
model, possibly a black-box Bayesian predictive model which does not compromise
accuracy, is fitted to the training data. Second, a proxy model from an
interpretable model family that best mimics the predictive behaviour of the
reference model is found by optimizing the interpretability utility function.
The approach is model agnostic -- neither the interpretable model nor the
reference model are restricted to a certain class of models -- and the
optimization problem can be solved using standard tools. Through experiments on
real-word data sets, using decision trees as interpretable models and Bayesian
additive regression models as reference models, we show that for the same level
of interpretability, our approach generates more accurate models than the
alternative of restricting the prior. We also propose a systematic way to
measure stability of interpretabile models constructed by different
interpretability approaches and show that our proposed approach generates more
stable models.
