  Simultaneous sequence generation is a pivotal task for real-time scenarios,
such as streaming speech recognition, simultaneous machine translation and
simultaneous speech translation, where the target sequence is generated while
receiving the source sequence. The crux of achieving high-quality generation
with low latency lies in identifying the optimal moments for generating,
accomplished by learning a mapping between the source and target sequences.
However, existing methods often rely on task-specific heuristics for different
sequence types, limiting the model's capacity to adaptively learn the
source-target mapping and hindering the exploration of multi-task learning for
various simultaneous tasks. In this paper, we propose a unified
segment-to-segment framework (Seg2Seg) for simultaneous sequence generation,
which learns the mapping in an adaptive and unified manner. During the process
of simultaneous generation, the model alternates between waiting for a source
segment and generating a target segment, making the segment serve as the
natural bridge between the source and target. To accomplish this, Seg2Seg
introduces a latent segment as the pivot between source to target and explores
all potential source-target mappings via the proposed expectation training,
thereby learning the optimal moments for generating. Experiments on multiple
simultaneous generation tasks demonstrate that Seg2Seg achieves
state-of-the-art performance and exhibits better generality across various
tasks.
