  We address the task of evidence retrieval for long document question
answering, which involves locating relevant paragraphs within a document to
answer a question. We aim to assess the applicability of large language models
(LLMs) in the task of zero-shot long document evidence retrieval, owing to
their unprecedented performance across various NLP tasks. However, currently
the LLMs can consume limited context lengths as input, thus providing document
chunks as inputs might overlook the global context while missing out on
capturing the inter-segment dependencies. Moreover, directly feeding the large
input sets can incur significant computational costs, particularly when
processing the entire document (and potentially incurring monetary expenses
with enterprise APIs like OpenAI's GPT variants). To address these challenges,
we propose a suite of techniques that exploit the discourse structure commonly
found in documents. By utilizing this structure, we create a condensed
representation of the document, enabling a more comprehensive understanding and
analysis of relationships between different parts. We retain $99.6\%$ of the
best zero-shot approach's performance, while processing only $26\%$ of the
total tokens used by the best approach in the information seeking evidence
retrieval setup. We also show how our approach can be combined with
\textit{self-ask} reasoning agent to achieve best zero-shot performance in
complex multi-hop question answering, just $\approx 4\%$ short of zero-shot
performance using gold evidence.
