  We propose a neural text-to-speech (TTS) model that can imitate a new
speaker's voice using only a small amount of speech sample. We demonstrate
voice imitation using only a 6-seconds long speech sample without any other
information such as transcripts. Our model also enables voice imitation
instantly without additional training of the model. We implemented the voice
imitating TTS model by combining a speaker embedder network with a
state-of-the-art TTS model, Tacotron. The speaker embedder network takes a new
speaker's speech sample and returns a speaker embedding. The speaker embedding
with a target sentence are fed to Tacotron, and speech is generated with the
new speaker's voice. We show that the speaker embeddings extracted by the
speaker embedder network can represent the latent structure in different
voices. The generated speech samples from our model have comparable voice
quality to the ones from existing multi-speaker TTS models.
