  We introduce SciQAG, a novel framework for automatically generating
high-quality science question-answer pairs from a large corpus of scientific
literature based on large language models (LLMs). SciQAG consists of a QA
generator and a QA evaluator, which work together to extract diverse and
research-level questions and answers from scientific papers. Utilizing this
framework, we construct a large-scale, high-quality, open-ended science QA
dataset containing 188,042 QA pairs extracted from 22,743 scientific papers
across 24 scientific domains. We also introduce SciQAG-24D, a new benchmark
task designed to evaluate the science question-answering ability of LLMs.
Extensive experiments demonstrate that fine-tuning LLMs on the SciQAG dataset
significantly improves their performance on both open-ended question answering
and scientific tasks. To foster research and collaboration, we make the
datasets, models, and evaluation codes publicly available, contributing to the
advancement of science question answering and developing more interpretable and
reasoning-capable AI systems.
