  Prosody Transfer (PT) is a technique that aims to use the prosody from a
source audio as a reference while synthesising speech. Fine-grained PT aims at
capturing prosodic aspects like rhythm, emphasis, melody, duration, and
loudness, from a source audio at a very granular level and transferring them
when synthesising speech in a different target speaker's voice. Current
approaches for fine-grained PT suffer from source speaker leakage, where the
synthesised speech has the voice identity of the source speaker as opposed to
the target speaker. In order to mitigate this issue, they compromise on the
quality of PT. In this paper, we propose CopyCat, a novel, many-to-many PT
system that is robust to source speaker leakage, without using parallel data.
We achieve this through a novel reference encoder architecture capable of
capturing temporal prosodic representations which are robust to source speaker
leakage. We compare CopyCat against a state-of-the-art fine-grained PT model
through various subjective evaluations, where we show a relative improvement of
$47\%$ in the quality of prosody transfer and $14\%$ in preserving the target
speaker identity, while still maintaining the same naturalness.
