  Evaluating automatically generated text is generally hard due to the
inherently subjective nature of many aspects of the output quality. This
difficulty is compounded in automatic consultation note generation by differing
opinions between medical experts both about which patient statements should be
included in generated notes and about their respective importance in arriving
at a diagnosis. Previous real-world evaluations of note-generation systems saw
substantial disagreement between expert evaluators. In this paper we propose a
protocol that aims to increase objectivity by grounding evaluations in
Consultation Checklists, which are created in a preliminary step and then used
as a common point of reference during quality assessment. We observed good
levels of inter-annotator agreement in a first evaluation study using the
protocol; further, using Consultation Checklists produced in the study as
reference for automatic metrics such as ROUGE or BERTScore improves their
correlation with human judgements compared to using the original human note.
