  Historically, much of machine learning research has focused on the
performance of the algorithm alone, but recently more attention has been
focused on optimizing joint human-algorithm performance. Here, we analyze a
specific type of human-algorithm collaboration where the algorithm has access
to a set of $n$ items, and presents a subset of size $k$ to the human, who
selects a final item from among those $k$. This scenario could model content
recommendation, route planning, or any type of labeling task. Because both the
human and algorithm have imperfect, noisy information about the true ordering
of items, the key question is: which value of $k$ maximizes the probability
that the best item will be ultimately selected? For $k=1$, performance is
optimized by the algorithm acting alone, and for $k=n$ it is optimized by the
human acting alone. Surprisingly, we show that for multiple of noise models, it
is optimal to set $k \in [2, n-1]$ - that is, there are strict benefits to
collaborating, even when the human and algorithm have equal accuracy
separately. We demonstrate this theoretically for the Mallows model and
experimentally for the Random Utilities models of noisy permutations. However,
we show this pattern is reversed when the human is anchored on the algorithm's
presented ordering - the joint system always has strictly worse performance. We
extend these results to the case where the human and algorithm differ in their
accuracy levels, showing that there always exist regimes where a more accurate
agent would strictly benefit from collaborating with a less accurate one, but
these regimes are asymmetric between the human and the algorithm's accuracy.
