  Large multimodal models (LMMs) have proven flexible and generalisable across
many tasks and fields. Although they have strong potential to aid scientific
research, their capabilities in this domain are not well characterised. A key
aspect of scientific research is the ability to understand and interpret
figures, which serve as a rich, compressed source of complex information. In
this work, we present SciFIBench, a scientific figure interpretation benchmark.
Our main benchmark consists of a 1000-question gold set of multiple-choice
questions split between two tasks across 12 categories. The questions are
curated from CS arXiv paper figures and captions, using adversarial filtering
to find hard negatives and human verification for quality control. We evaluate
26 LMMs on SciFIBench, finding it to be a challenging benchmark. Finally, we
investigate the alignment and reasoning faithfulness of the LMMs on augmented
question sets from our benchmark. We release SciFIBench to encourage progress
in this domain.
