  We describe a Question Answering (QA) dataset that contains complex questions
with conditional answers, i.e. the answers are only applicable when certain
conditions apply. We call this dataset ConditionalQA. In addition to
conditional answers, the dataset also features: (1) long context documents with
information that is related in logically complex ways; (2) multi-hop questions
that require compositional logical reasoning; (3) a combination of extractive
questions, yes/no questions, questions with multiple answers, and
not-answerable questions; (4) questions asked without knowing the answers. We
show that ConditionalQA is challenging for many of the existing QA models,
especially in selecting answer conditions. We believe that this dataset will
motivate further research in answering complex questions over long documents.
Data and leaderboard are publicly available at
\url{https://github.com/haitian-sun/ConditionalQA}.
