  Speech-to-speech translation (S2ST) converts input speech to speech in
another language. A challenge of delivering S2ST in real time is the
accumulated delay between the translation and speech synthesis modules. While
recently incremental text-to-speech (iTTS) models have shown large quality
improvements, they typically require additional future text inputs to reach
optimal performance. In this work, we minimize the initial waiting time of iTTS
by adapting the upstream speech translator to generate high-quality pseudo
lookahead for the speech synthesizer. After mitigating the initial delay, we
demonstrate that the duration of synthesized speech also plays a crucial role
on latency. We formalize this as a latency metric and then present a simple yet
effective duration-scaling approach for latency reduction. Our approaches
consistently reduce latency by 0.2-0.5 second without sacrificing speech
translation quality.
