  Recent advances in style transfer text-to-speech (TTS) have improved the
expressiveness of synthesized speech. Despite these advancements, encoding
stylistic information from diverse and unseen reference speech remains
challenging. This paper introduces StyleMoE, an approach that divides the
embedding space, modeled by the style encoder, into tractable subsets handled
by style experts. The proposed method replaces the style encoder in a TTS
system with a Mixture of Experts (MoE) layer. By utilizing a gating network to
route reference speeches to different style experts, each expert specializes in
aspects of the style space during optimization. Our experiments objectively and
subjectively demonstrate the effectiveness of our proposed method in increasing
the coverage of the style space for diverse and unseen styles. This approach
can enhance the performance of existing state-of-the-art style transfer TTS
models, marking the first study of MoE in style transfer TTS to our knowledge.
