  The prediction of chemical reactions has gained significant interest within
the machine learning community in recent years, owing to its complexity and
crucial applications in chemistry. However, model evaluation for this task has
been mostly limited to simple metrics like top-k accuracy, which obfuscates
fine details of a model's limitations. Inspired by progress in other fields, we
propose a new assessment scheme that builds on top of current approaches,
steering towards a more holistic evaluation. We introduce the following key
components for this goal: CHORISO, a curated dataset along with multiple
tailored splits to recreate chemically relevant scenarios, and a collection of
metrics that provide a holistic view of a model's advantages and limitations.
Application of this method to state-of-the-art models reveals important
differences on sensitive fronts, especially stereoselectivity and chemical
out-of-distribution generalization. Our work paves the way towards robust
prediction models that can ultimately accelerate chemical discovery.
