  Text-to-Speech synthesis systems are generally evaluated using Mean Opinion
Score (MOS) tests, where listeners score samples of synthetic speech on a
Likert scale. A major drawback of MOS tests is that they only offer a general
measure of overall quality-i.e., the naturalness of an utterance-and so cannot
tell us where exactly synthesis errors occur. This can make evaluation of the
appropriateness of prosodic variation within utterances inconclusive. To
address this, we propose a novel evaluation method based on the Rapid Prosody
Transcription paradigm. This allows listeners to mark the locations of errors
in an utterance in real-time, providing a probabilistic representation of the
perceptual errors that occur in the synthetic signal. We conduct experiments
that confirm that the fine-grained evaluation can be mapped to system rankings
of standard MOS tests, but the error marking gives a much more comprehensive
assessment of synthesized prosody. In particular, for standard audiobook test
set samples, we see that error marks consistently cluster around words at major
prosodic boundaries indicated by punctuation. However, for question-answer
based stimuli, where we control information structure, we see differences
emerge in the ability of neural TTS systems to generate context-appropriate
prosodic prominence.
