  Recent pretrained language models "solved" many reading comprehension
benchmarks, where questions are written with access to the evidence document.
However, datasets containing information-seeking queries where evidence
documents are provided after the queries are written independently remain
challenging. We analyze why answering information-seeking queries is more
challenging and where their prevalent unanswerabilities arise, on Natural
Questions and TyDi QA. Our controlled experiments suggest two headrooms --
paragraph selection and answerability prediction, i.e. whether the paired
evidence document contains the answer to the query or not. When provided with a
gold paragraph and knowing when to abstain from answering, existing models
easily outperform a human annotator. However, predicting answerability itself
remains challenging. We manually annotate 800 unanswerable examples across six
languages on what makes them challenging to answer. With this new data, we
conduct per-category answerability prediction, revealing issues in the current
dataset collection as well as task formulation. Together, our study points to
avenues for future research in information-seeking question answering, both for
dataset creation and model development.
