  Adversarial evaluation stress tests a model's understanding of natural
language. While past approaches expose superficial patterns, the resulting
adversarial examples are limited in complexity and diversity. We propose
human-in-the-loop adversarial generation, where human authors are guided to
break models. We aid the authors with interpretations of model predictions
through an interactive user interface. We apply this generation framework to a
question answering task called Quizbowl, where trivia enthusiasts craft
adversarial questions. The resulting questions are validated via live
human--computer matches: although the questions appear ordinary to humans, they
systematically stump neural and information retrieval models. The adversarial
questions cover diverse phenomena from multi-hop reasoning to entity type
distractors, exposing open challenges in robust question answering.
