  End-to-end neural TTS training has shown improved performance in speech style
transfer. However, the improvement is still limited by the training data in
both target styles and speakers. Inadequate style transfer performance occurs
when the trained TTS tries to transfer the speech to a target style from a new
speaker with an unknown, arbitrary style. In this paper, we propose a new
approach to style transfer for both seen and unseen styles, with disjoint,
multi-style datasets, i.e., datasets of different styles are recorded, each
individual style is by one speaker with multiple utterances. To encode the
style information, we adopt an inverse autoregressive flow (IAF) structure to
improve the variational inference. The whole system is optimized to minimize a
weighed sum of four different loss functions: 1) a reconstruction loss to
measure the distortions in both source and target reconstructions; 2) an
adversarial loss to "fool" a well-trained discriminator; 3) a style distortion
loss to measure the expected style loss after the transfer; 4) a cycle
consistency loss to preserve the speaker identity of the source after the
transfer. Experiments demonstrate, both objectively and subjectively, the
effectiveness of the proposed approach for seen and unseen style transfer
tasks. The performance of the new approach is better and more robust than those
of four baseline systems of the prior art.
