  Maintaining factual consistency is a critical issue in abstractive text
summarisation, however, it cannot be assessed by traditional automatic metrics
used for evaluating text summarisation, such as ROUGE scoring. Recent efforts
have been devoted to developing improved metrics for measuring factual
consistency using pre-trained language models, but these metrics have
restrictive token limits, and are therefore not suitable for evaluating long
document text summarisation. Moreover, there is limited research and resources
available for evaluating whether existing automatic evaluation metrics are fit
for purpose when applied in long document settings. In this work, we evaluate
the efficacy of automatic metrics for assessing the factual consistency of long
document text summarisation. We create a human-annotated data set for
evaluating automatic factuality metrics, LongSciVerify, which contains
fine-grained factual consistency annotations for long document summaries from
the scientific domain. We also propose a new evaluation framework,
LongDocFACTScore, which is suitable for evaluating long document summarisation.
This framework allows metrics to be efficiently extended to any length document
and outperforms existing state-of-the-art metrics in its ability to correlate
with human measures of factuality when used to evaluate long document
summarisation data sets. We make our code and LongSciVerify data set publicly
available: https://github.com/jbshp/LongDocFACTScore.
