  With machine learning models being increasingly used to aid decision making
even in high-stakes domains, there has been a growing interest in developing
interpretable models. Although many supposedly interpretable models have been
proposed, there have been relatively few experimental studies investigating
whether these models achieve their intended effects, such as making people more
closely follow a model's predictions when it is beneficial for them to do so or
enabling them to detect when a model has made a mistake. We present a sequence
of pre-registered experiments (N=3,800) in which we showed participants
functionally identical models that varied only in two factors commonly thought
to make machine learning models more or less interpretable: the number of
features and the transparency of the model (i.e., whether the model internals
are clear or black box). Predictably, participants who saw a clear model with
few features could better simulate the model's predictions. However, we did not
find that participants more closely followed its predictions. Furthermore,
showing participants a clear model meant that they were less able to detect and
correct for the model's sizable mistakes, seemingly due to information
overload. These counterintuitive findings emphasize the importance of testing
over intuition when developing interpretable models.
