  Representing speech as discretized units has numerous benefits in supporting
downstream spoken language processing tasks. However, the approach has been
less explored in speech synthesis of tonal languages like Mandarin Chinese. Our
preliminary experiments on Chinese speech synthesis reveal the issue of "tone
shift", where a synthesized speech utterance contains correct base syllables
but incorrect tones. To address the issue, we propose the ToneUnit framework,
which leverages annotated data with tone labels as CTC supervision to learn
tone-aware discrete speech units for Mandarin Chinese speech. Our findings
indicate that the discrete units acquired through the TonUnit resolve the "tone
shift" issue in synthesized Chinese speech and yield favorable results in
English synthesis. Moreover, the experimental results suggest that finite
scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit
can work effectively even with minimal annotated data.
