  In the era of "information overload", effective information provision is
essential for enabling rapid response and critical decision making. In making
sense of diverse information sources, data dashboards have become an
indispensable tool, providing fast, effective, adaptable, and personalized
access to information for professionals and the general public alike. However,
these objectives place a heavy requirement on dashboards as information
systems, resulting in poor usability and ineffective design. Understanding
these shortfalls is a challenge given the absence of a consistent and
comprehensive approach to dashboard evaluation. In this paper we systematically
review literature on dashboard implementation in the healthcare domain, a field
where dashboards have been employed widely, and in which there is widespread
interest for improving the current state of the art, and subsequently analyse
approaches taken towards evaluation. We draw upon consolidated dashboard
literature and our own observations to introduce a general definition of
dashboards which is more relevant to current trends, together with a dashboard
task-based classification, which underpin our subsequent analysis. From a total
of 81 papers we derive seven evaluation scenarios - task performance, behaviour
change, interaction workflow, perceived engagement, potential utility,
algorithm performance and system implementation. These scenarios distinguish
different evaluation purposes which we illustrate through measurements, example
studies, and common challenges in evaluation study design. We provide a
breakdown of each evaluation scenario, and highlight some of the subtle and
less well posed questions. We conclude by outlining a number of active
discussion points and a set of dashboard evaluation best practices for the
academic, clinical and software development communities alike.
