  Text-based voice editing (TBVE) uses synthetic output from text-to-speech
(TTS) systems to replace words in an original recording. Recent work has used
neural models to produce edited speech that is similar to the original speech
in terms of clarity, speaker identity, and prosody. However, one limitation of
prior work is the usage of finetuning to optimise performance: this requires
further model training on data from the target speaker, which is a costly
process that may incorporate potentially sensitive data into server-side
models. In contrast, this work focuses on the zero-shot approach which avoids
finetuning altogether, and instead uses pretrained speaker verification
embeddings together with a jointly trained reference encoder to encode
utterance-level information that helps capture aspects such as speaker identity
and prosody. Subjective listening tests find that both utterance embeddings and
a reference encoder improve the continuity of speaker identity and prosody
between the edited synthetic speech and unedited original recording in the
zero-shot setting.
