  Synthesizing speech across different accents while preserving the speaker
identity is essential for various real-world customer applications. However,
the individual and accurate modeling of accents and speakers in a
text-to-speech (TTS) system is challenging due to the complexity of accent
variations and the intrinsic entanglement between the accent and speaker
identity. In this paper, we present a novel approach for multi-speaker
multi-accent TTS synthesis, which aims to synthesize voices of multiple
speakers, each with various accents. Our proposed approach employs a
multi-scale accent modeling strategy to address accent variations at different
levels. Specifically, we introduce both global (utterance level) and local
(phoneme level) accent modeling, supervised by individual accent classifiers to
capture the overall variation within accented utterances and fine-grained
variations between phonemes, respectively. To control accents and speakers
separately, speaker-independent accent modeling is necessary, which is achieved
by adversarial training with speaker classifiers to disentangle speaker
identity within the multi-scale accent modeling. Consequently, we obtain
speaker-independent and accent-discriminative multi-scale embeddings as
comprehensive accent features. Additionally, we propose a local accent
prediction model that allows to generate accented speech directly from phoneme
inputs. Extensive experiments are conducted on an accented English speech
corpus. Both objective and subjective evaluations show the superiority of our
proposed system compared to baselines systems. Detailed component analysis
demonstrates the effectiveness of global and local accent modeling, and speaker
disentanglement on multi-speaker multi-accent speech synthesis.
