  High-fidelity speech can be synthesized by end-to-end text-to-speech models
in recent years. However, accessing and controlling speech attributes such as
speaker identity, prosody, and emotion in a text-to-speech system remains a
challenge. This paper presents a system involving feedback constraint for
multispeaker speech synthesis. We manage to enhance the knowledge transfer from
the speaker verification to the speech synthesis by engaging the speaker
verification network. The constraint is taken by an added loss related to the
speaker identity, which is centralized to improve the speaker similarity
between the synthesized speech and its natural reference audio. The model is
trained and evaluated on publicly available datasets. Experimental results,
including visualization on speaker embedding space, show significant
improvement in terms of speaker identity cloning in the spectrogram level.
Synthesized samples are available online for listening.
(https://caizexin.github.io/mlspk-syn-samples/index.html)
