  In addition to the traditional task of getting machines to answer questions,
a major research question in question answering is to create interesting,
challenging questions that can help systems learn how to answer questions and
also reveal which systems are the best at answering questions. We argue that
creating a question answering dataset -- and the ubiquitous leaderboard that
goes with it -- closely resembles running a trivia tournament: you write
questions, have agents (either humans or machines) answer the questions, and
declare a winner. However, the research community has ignored the decades of
hard-learned lessons from decades of the trivia community creating vibrant,
fair, and effective question answering competitions. After detailing problems
with existing QA datasets, we outline the key lessons -- removing ambiguity,
discriminating skill, and adjudicating disputes -- that can transfer to QA
research and how they might be implemented for the QA community.
