  Recent efforts in Machine Learning (ML) interpretability have focused on
creating methods for explaining black-box ML models. However, these methods
rely on the assumption that simple approximations, such as linear models or
decision-trees, are inherently human-interpretable, which has not been
empirically tested. Additionally, past efforts have focused exclusively on
comprehension, neglecting to explore the trust component necessary to convince
non-technical experts, such as clinicians, to utilize ML models in practice. In
this paper, we posit that reinforcement learning (RL) can be used to learn what
is interpretable to different users and, consequently, build their trust in ML
models. To validate this idea, we first train a neural network to provide risk
assessments for heart failure patients. We then design a RL-based clinical
decision-support system (DSS) around the neural network model, which can learn
from its interactions with users. We conduct an experiment involving a diverse
set of clinicians from multiple institutions in three different countries. Our
results demonstrate that ML experts cannot accurately predict which system
outputs will maximize clinicians' confidence in the underlying neural network
model, and suggest additional findings that have broad implications to the
future of research into ML interpretability and the use of ML in medicine.
