  Summarization evaluation remains an open research problem: current metrics
such as ROUGE are known to be limited and to correlate poorly with human
judgments. To alleviate this issue, recent work has proposed evaluation metrics
which rely on question answering models to assess whether a summary contains
all the relevant information in its source document. Though promising, the
proposed approaches have so far failed to correlate better than ROUGE with
human judgments.
  In this paper, we extend previous approaches and propose a unified framework,
named QuestEval. In contrast to established metrics such as ROUGE or BERTScore,
QuestEval does not require any ground-truth reference. Nonetheless, QuestEval
substantially improves the correlation with human judgments over four
evaluation dimensions (consistency, coherence, fluency, and relevance), as
shown in the extensive experiments we report.
