  Decision-making with information displays is a key focus of research in areas
like explainable AI, human-AI teaming, and data visualization. However, what
constitutes a decision problem, and what is required for an experiment to be
capable of concluding that human decisions are flawed in some way, remain open
to speculation. We present a widely applicable definition of a decision problem
synthesized from statistical decision theory and information economics. We
argue that to attribute loss in human performance to forms of bias, an
experiment must provide participants with the information that a rational agent
would need to identify the normative decision. We evaluate the extent to which
recent evaluations of decision-making from the literature on AI-assisted
decisions achieve this criteria. We find that only 10 (26\%) of 39 studies that
claim to identify biased behavior present participants with sufficient
information to characterize their behavior as deviating from good
decision-making in at least one treatment condition. We motivate the value of
studying well-defined decision problems by describing a characterization of
performance losses they allow us to conceive. In contrast, the ambiguities of a
poorly communicated decision problem preclude normative interpretation. We
conclude with recommendations for practice.
