  Incremental text-to-speech, also known as streaming TTS, has been
increasingly applied to online speech applications that require ultra-low
response latency to provide an optimal user experience. However, most of the
existing speech synthesis pipelines deployed on GPU are still non-incremental,
which uncovers limitations in high-concurrency scenarios, especially when the
pipeline is built with end-to-end neural network models. To address this issue,
we present a highly efficient approach to perform real-time incremental TTS on
GPUs with Instant Request Pooling and Module-wise Dynamic Batching.
Experimental results demonstrate that the proposed method is capable of
producing high-quality speech with a first-chunk latency lower than 80ms under
100 QPS on a single NVIDIA A10 GPU and significantly outperforms the
non-incremental twin in both concurrency and latency. Our work reveals the
effectiveness of high-performance incremental TTS on GPUs.
