  Scientific understanding is a fundamental goal of science, allowing us to
explain the world. There is currently no good way to measure the scientific
understanding of agents, whether these be humans or Artificial Intelligence
systems. Without a clear benchmark, it is challenging to evaluate and compare
different levels of and approaches to scientific understanding. In this
Roadmap, we propose a framework to create a benchmark for scientific
understanding, utilizing tools from philosophy of science. We adopt a
behavioral notion according to which genuine understanding should be recognized
as an ability to perform certain tasks. We extend this notion by considering a
set of questions that can gauge different levels of scientific understanding,
covering information retrieval, the capability to arrange information to
produce an explanation, and the ability to infer how things would be different
under different circumstances. The Scientific Understanding Benchmark (SUB),
which is formed by a set of these tests, allows for the evaluation and
comparison of different approaches. Benchmarking plays a crucial role in
establishing trust, ensuring quality control, and providing a basis for
performance evaluation. By aligning machine and human scientific understanding
we can improve their utility, ultimately advancing scientific understanding and
helping to discover new insights within machines.
