  Details of the designs and mechanisms in support of human-AI collaboration
must be considered in the real-world fielding of AI technologies. A critical
aspect of interaction design for AI-assisted human decision making are policies
about the display and sequencing of AI inferences within larger decision-making
workflows. We have a poor understanding of the influences of making AI
inferences available before versus after human review of a diagnostic task at
hand. We explore the effects of providing AI assistance at the start of a
diagnostic session in radiology versus after the radiologist has made a
provisional decision. We conducted a user study where 19 veterinary
radiologists identified radiographic findings present in patients' X-ray
images, with the aid of an AI tool. We employed two workflow configurations to
analyze (i) anchoring effects, (ii) human-AI team diagnostic performance and
agreement, (iii) time spent and confidence in decision making, and (iv)
perceived usefulness of the AI. We found that participants who are asked to
register provisional responses in advance of reviewing AI inferences are less
likely to agree with the AI regardless of whether the advice is accurate and,
in instances of disagreement with the AI, are less likely to seek the second
opinion of a colleague. These participants also reported the AI advice to be
less useful. Surprisingly, requiring provisional decisions on cases in advance
of the display of AI inferences did not lengthen the time participants spent on
the task. The study provides generalizable and actionable insights for the
deployment of clinical AI tools in human-in-the-loop systems and introduces a
methodology for studying alternative designs for human-AI collaboration. We
make our experimental platform available as open source to facilitate future
research on the influence of alternate designs on human-AI workflows.
