  The act of explaining across two parties is a feedback loop, where one
provides information on what needs to be explained and the other provides an
explanation relevant to this information. We apply a reinforcement learning
framework which emulates this format by providing explanations based on the
explainee's current mental model. We conduct novel online human experiments
where explanations generated by various explanation methods are selected and
presented to participants, using policies which observe participants' mental
models, in order to optimize an interpretability proxy. Our results suggest
that mental model-based policies (anchored in our proposed state
representation) may increase interpretability over multiple sequential
explanations, when compared to a random selection baseline. This work provides
insight into how to select explanations which increase relevant information for
users, and into conducting human-grounded experimentation to understand
interpretability.
