  Subtitles appear on screen as short pieces of text, segmented based on formal
constraints (length) and syntactic/semantic criteria. Subtitle segmentation can
be evaluated with sequence segmentation metrics against a human reference.
However, standard segmentation metrics cannot be applied when systems generate
outputs different than the reference, e.g. with end-to-end subtitling systems.
In this paper, we study ways to conduct reference-based evaluations of
segmentation accuracy irrespective of the textual content. We first conduct a
systematic analysis of existing metrics for evaluating subtitle segmentation.
We then introduce $Sigma$, a new Subtitle Segmentation Score derived from an
approximate upper-bound of BLEU on segmentation boundaries, which allows us to
disentangle the effect of good segmentation from text quality. To compare
$Sigma$ with existing metrics, we further propose a boundary projection method
from imperfect hypotheses to the true reference. Results show that all metrics
are able to reward high quality output but for similar outputs system ranking
depends on each metric's sensitivity to error type. Our thorough analyses
suggest $Sigma$ is a promising segmentation candidate but its reliability over
other segmentation metrics remains to be validated through correlations with
human judgements.
