  The increased use of algorithmic predictions in sensitive domains has been
accompanied by both enthusiasm and concern. To understand the opportunities and
risks of these technologies, it is key to study how experts alter their
decisions when using such tools. In this paper, we study the adoption of an
algorithmic tool used to assist child maltreatment hotline screening decisions.
We focus on the question: Are humans capable of identifying cases in which the
machine is wrong, and of overriding those recommendations? We first show that
humans do alter their behavior when the tool is deployed. Then, we show that
humans are less likely to adhere to the machine's recommendation when the score
displayed is an incorrect estimate of risk, even when overriding the
recommendation requires supervisory approval. These results highlight the risks
of full automation and the importance of designing decision pipelines that
provide humans with autonomy.
