  Reference-based metrics that operate at the sentence-level typically
outperform quality estimation metrics, which have access only to the source and
system output. This is unsurprising, since references resolve ambiguities that
may be present in the source. In this paper, we investigate whether additional
source context can effectively substitute for a reference. We present a metric
named SLIDE (SLIding Document Evaluator), which operates on blocks of
sentences. SLIDE leverages a moving window that slides over each document in
the test set, feeding each chunk of sentences into an unmodified, off-the-shelf
quality estimation model. We find that SLIDE obtains significantly higher
pairwise system accuracy than its sentence-level baseline, in some cases even
eliminating the gap with reference-base metrics. This suggests that source
context may provide the same information as a human reference in disambiguating
source ambiguities. This finding is especially pertinent for reference-free
document-level evaluation, wherein SLIDE could provide higher-quality pairwise
system assessments while only requiring document boundary annotations.
