  Interactive AI systems increasingly employ a human-in-the-loop strategy. This
creates new challenges for the HCI community when designing such systems. We
reveal and investigate some of these challenges in a case study with an
industry partner, and developed a prototype human-in-the-loop system for
preference-guided 3D model processing. Two 3D artists used it in their daily
work for 3 months. We found that the human-AI loop often did not converge
towards a satisfactory result and designed a lab study (N=20) to investigate
this further. We analyze interaction data and user feedback through the lens of
theories of human judgment to explain the observed human-in-the-loop failures
with two key insights: 1) optimization using preferential choices lacks
mechanisms to deal with inconsistent and contradictory human judgments; 2)
machine outcomes, in turn, influence future user inputs via heuristic biases
and loss aversion. To mitigate these problems, we propose descriptive UI design
guidelines. Our case study draws attention to challenging and practically
relevant imperfections in human-AI loops that need to be considered when
designing human-in-the-loop systems.
