  End-to-end speech translation (ST) for conversation recordings involves
several under-explored challenges such as speaker diarization (SD) without
accurate word time stamps and handling of overlapping speech in a streaming
fashion. In this work, we propose DiariST, the first streaming ST and SD
solution. It is built upon a neural transducer-based streaming ST system and
integrates token-level serialized output training and t-vector, which were
originally developed for multi-talker speech recognition. Due to the absence of
evaluation benchmarks in this area, we develop a new evaluation dataset,
DiariST-AliMeeting, by translating the reference Chinese transcriptions of the
AliMeeting corpus into English. We also propose new metrics, called
speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality
while taking SD accuracy into account. Our system achieves a strong ST and SD
capability compared to offline systems based on Whisper, while performing
streaming inference for overlapping speech. To facilitate the research in this
new direction, we release the evaluation data, the offline baseline systems,
and the evaluation code.
