  As practitioners increasingly deploy machine learning models in critical
domains such as health care, finance, and policy, it becomes vital to ensure
that domain experts function effectively alongside these models. Explainability
is one way to bridge the gap between human decision-makers and machine learning
models. However, most of the existing work on explainability focuses on
one-off, static explanations like feature importances or rule lists. These
sorts of explanations may not be sufficient for many use cases that require
dynamic, continuous discovery from stakeholders. In the literature, few works
ask decision-makers about the utility of existing explanations and other
desiderata they would like to see in an explanation going forward. In this
work, we address this gap and carry out a study where we interview doctors,
healthcare professionals, and policymakers about their needs and desires for
explanations. Our study indicates that decision-makers would strongly prefer
interactive explanations in the form of natural language dialogues. Domain
experts wish to treat machine learning models as "another colleague", i.e., one
who can be held accountable by asking why they made a particular decision
through expressive and accessible natural language interactions. Considering
these needs, we outline a set of five principles researchers should follow when
designing interactive explanations as a starting place for future work.
Further, we show why natural language dialogues satisfy these principles and
are a desirable way to build interactive explanations. Next, we provide a
design of a dialogue system for explainability and discuss the risks,
trade-offs, and research opportunities of building these systems. Overall, we
hope our work serves as a starting place for researchers and engineers to
design interactive explainability systems.
