  For humans and computers, the first step in answering an open-domain question
is retrieving a set of relevant documents from a large corpus. However, the
strategies that computers use fundamentally differ from those of humans. To
better understand these differences, we design a gamified interface for data
collection -- Cheater's Bowl -- where a human answers complex questions with
access to both traditional and modern search tools. We collect a dataset of
human search sessions, analyze human search strategies, and compare them to
state-of-the-art multi-hop QA models. Humans query logically, apply dynamic
search chains, and use world knowledge to boost searching. We demonstrate how
human queries can improve the accuracy of existing systems and propose
improving the future design of QA models.
