  We adapt the well-known beam-search algorithm for machine translation to
operate in a cascaded real-time speech translation system. This proved to be
more complex than initially anticipated, due to four key challenges: (1)
real-time processing of intermediate and final transcriptions with incomplete
words from ASR, (2) emitting intermediate and final translations with minimal
user perceived latency, (3) handling beam search hypotheses that have unequal
length and different model state, and (4) handling sentence boundaries.
Previous work in the field of simultaneous machine translation only implemented
greedy decoding. We present a beam-search realization that handles all of the
above, providing guidance through the minefield of challenges. Our approach
increases the BLEU score by 1 point compared to greedy search, reduces the CPU
time by up to 40% and character flicker rate by 20+% compared to a baseline
heuristic that just retranslates input repeatedly.
