  Prior research in psychology has found that people's decisions are often
inconsistent. An individual's decisions vary across time, and decisions vary
even more across people. Inconsistencies have been identified not only in
subjective matters, like matters of taste, but also in settings one might
expect to be more objective, such as sentencing, job performance evaluations,
or real estate appraisals. In our study, we explore whether algorithmic
decision aids can be used to moderate the degree of inconsistency in human
decision-making in the context of real estate appraisal. In a large-scale
human-subject experiment, we study how different forms of algorithmic
assistance influence the way that people review and update their estimates of
real estate prices. We find that both (i) asking respondents to review their
estimates in a series of algorithmically chosen pairwise comparisons and (ii)
providing respondents with traditional machine advice are effective strategies
for influencing human responses. Compared to simply reviewing initial estimates
one by one, the aforementioned strategies lead to (i) a higher propensity to
update initial estimates, (ii) a higher accuracy of post-review estimates, and
(iii) a higher degree of consistency between the post-review estimates of
different respondents. While these effects are more pronounced with traditional
machine advice, the approach of reviewing algorithmically chosen pairs can be
implemented in a wider range of settings, since it does not require access to
ground truth data.
