  Large Language Models (LLMs) have been applied to many research problems
across various domains. One of the applications of LLMs is providing
question-answering systems that cater to users from different fields. The
effectiveness of LLM-based question-answering systems has already been
established at an acceptable level for users posing questions in popular and
public domains such as trivia and literature. However, it has not often been
established in niche domains that traditionally require specialized expertise.
To this end, we construct the NEPAQuAD1.0 benchmark to evaluate the performance
of three frontier LLMs -- Claude Sonnet, Gemini, and GPT-4 -- when answering
questions originating from Environmental Impact Statements prepared by U.S.
federal government agencies in accordance with the National Environmental
Environmental Act (NEPA). We specifically measure the ability of LLMs to
understand the nuances of legal, technical, and compliance-related information
present in NEPA documents in different contextual scenarios. For example, we
test the LLMs' internal prior NEPA knowledge by providing questions without any
context, as well as assess how LLMs synthesize the contextual information
present in long NEPA documents to facilitate the question/answering task. We
compare the performance of the long context LLMs and RAG powered models in
handling different types of questions (e.g., problem-solving, divergent). Our
results suggest that RAG powered models significantly outperform the long
context models in the answer accuracy regardless of the choice of the frontier
LLM. Our further analysis reveals that many models perform better answering
closed questions than divergent and problem-solving questions.
