  End-to-end neural TTS has shown improved performance in speech style
transfer. However, the improvement is still limited by the available training
data in both target styles and speakers. Additionally, degenerated performance
is observed when the trained TTS tries to transfer the speech to a target style
from a new speaker with an unknown, arbitrary style. In this paper, we propose
a new approach to seen and unseen style transfer training on disjoint,
multi-style datasets, i.e., datasets of different styles are recorded, one
individual style by one speaker in multiple utterances. An inverse
autoregressive flow (IAF) technique is first introduced to improve the
variational inference for learning an expressive style representation. A
speaker encoder network is then developed for learning a discriminative speaker
embedding, which is jointly trained with the rest neural TTS modules. The
proposed approach of seen and unseen style transfer is effectively trained with
six specifically-designed objectives: reconstruction loss, adversarial loss,
style distortion loss, cycle consistency loss, style classification loss, and
speaker classification loss. Experiments demonstrate, both objectively and
subjectively, the effectiveness of the proposed approach for seen and unseen
style transfer tasks. The performance of our approach is superior to and more
robust than those of four other reference systems of prior art.
