  Open-domain question answering answers a question based on evidence retrieved
from a large corpus. State-of-the-art neural approaches require intermediate
evidence annotations for training. However, such intermediate annotations are
expensive, and methods that rely on them cannot transfer to the more common
setting, where only question-answer pairs are available. This paper
investigates whether models can learn to find evidence from a large corpus,
with only distant supervision from answer labels for model training, thereby
generating no additional annotation cost. We introduce a novel approach
(DistDR) that iteratively improves over a weak retriever by alternately finding
evidence from the up-to-date model and encouraging the model to learn the most
likely evidence. Without using any evidence labels, DistDR is on par with
fully-supervised state-of-the-art methods on both multi-hop and single-hop QA
benchmarks. Our analysis confirms that DistDR finds more accurate evidence over
iterations, which leads to model improvements.
