  We address the problem of extractive question answering using document-level
distant super-vision, pairing questions and relevant documents with answer
strings. We compare previously used probability space and distant super-vision
assumptions (assumptions on the correspondence between the weak answer string
labels and possible answer mention spans). We show that these assumptions
interact, and that different configurations provide complementary benefits. We
demonstrate that a multi-objective model can efficiently combine the advantages
of multiple assumptions and out-perform the best individual formulation. Our
approach outperforms previous state-of-the-art models by 4.3 points in F1 on
TriviaQA-Wiki and 1.7 points in Rouge-L on NarrativeQA summaries.
