  We present an interpretable companion model for any pre-trained black-box
classifiers. The idea is that for any input, a user can decide to either
receive a prediction from the black-box model, with high accuracy but no
explanations, or employ a companion rule to obtain an interpretable prediction
with slightly lower accuracy. The companion model is trained from data and the
predictions of the black-box model, with the objective combining area under the
transparency--accuracy curve and model complexity. Our model provides flexible
choices for practitioners who face the dilemma of choosing between always using
interpretable models and always using black-box models for a predictive task,
so users can, for any given input, take a step back to resort to an
interpretable prediction if they find the predictive performance satisfying, or
stick to the black-box model if the rules are unsatisfying. To show the value
of companion models, we design a human evaluation on more than a hundred people
to investigate the tolerable accuracy loss to gain interpretability for humans.
