  Question Answering (QA) effectively evaluates language models' reasoning and
knowledge depth. While QA datasets are plentiful in areas like general domain
and biomedicine, academic chemistry is less explored. Chemical QA plays a
crucial role in both education and research by effectively translating complex
chemical information into readily understandable format. Addressing this gap,
we introduce ScholarChemQA, a large-scale QA dataset constructed from chemical
papers. This dataset reflects typical real-world challenges, including an
imbalanced data distribution and a substantial amount of unlabeled data that
can be potentially useful. Correspondingly, we introduce a QAMatch model,
specifically designed to effectively answer chemical questions by fully
leveraging our collected data. We first address the issue of imbalanced label
distribution by re-weighting the instance-wise loss based on the inverse
frequency of each class, ensuring minority classes are not dominated by
majority ones during optimization. Next, we utilize the unlabeled data to
enrich the learning process, generating a variety of augmentations based on a
SoftMix operation and ensuring their predictions align with the same target,
i.e., pseudo-labels. To ensure the quality of the pseudo-labels, we propose a
calibration procedure aimed at closely aligning the pseudo-label estimates of
individual samples with a desired ground truth distribution. Experiments show
that our QAMatch significantly outperforms the recent similar-scale baselines
and Large Language Models (LLMs) not only on our ScholarChemQA dataset but also
on four benchmark datasets. We hope our benchmark and model can facilitate and
promote more research on chemical QA.
