  Human-AI collaboration for decision-making strives to achieve team
performance that exceeds the performance of humans or AI alone. However, many
factors can impact success of Human-AI teams, including a user's domain
expertise, mental models of an AI system, trust in recommendations, and more.
This work examines users' interaction with three simulated algorithmic models,
all with similar accuracy but different tuning on their true positive and true
negative rates. Our study examined user performance in a non-trivial blood
vessel labeling task where participants indicated whether a given blood vessel
was flowing or stalled.
  Our results show that while recommendations from an AI-Assistant can aid user
decision making, factors such as users' baseline performance relative to the AI
and complementary tuning of AI error types significantly impact overall team
performance. Novice users improved, but not to the accuracy level of the AI.
Highly proficient users were generally able to discern when they should follow
the AI recommendation and typically maintained or improved their performance.
Mid-performers, who had a similar level of accuracy to the AI, were most
variable in terms of whether the AI recommendations helped or hurt their
performance. In addition, we found that users' perception of the AI's
performance relative on their own also had a significant impact on whether
their accuracy improved when given AI recommendations. This work provides
insights on the complexity of factors related to Human-AI collaboration and
provides recommendations on how to develop human-centered AI algorithms to
complement users in decision-making tasks.
