  The use of Artificial Intelligence (AI) based on data-driven algorithms has
become ubiquitous in today's society. Yet, in many cases and especially when
stakes are high, humans still make final decisions. The critical question,
therefore, is whether AI helps humans make better decisions as compared to a
human alone or AI an alone. We introduce a new methodological framework that
can be used to answer experimentally this question with no additional
assumptions. We measure a decision maker's ability to make correct decisions
using standard classification metrics based on the baseline potential outcome.
We consider a single-blinded experimental design, in which the provision of
AI-generated recommendations is randomized across cases with a human making
final decisions. Under this experimental design, we show how to compare the
performance of three alternative decision-making systems--human-alone,
human-with-AI, and AI-alone. We apply the proposed methodology to the data from
our own randomized controlled trial of a pretrial risk assessment instrument.
We find that AI recommendations do not improve the classification accuracy of a
judge's decision to impose cash bail. Our analysis also shows that AI-alone
decisions generally perform worse than human decisions with or without AI
assistance. Finally, AI recommendations tend to impose cash bail on non-white
arrestees more often than necessary when compared to white arrestees.
