  Simultaneous machine translation (SimulMT) models start translation before
the end of the source sentence, making the translation monotonically aligned
with the source sentence. However, the general full-sentence translation test
set is acquired by offline translation of the entire source sentence, which is
not designed for SimulMT evaluation, making us rethink whether this will
underestimate the performance of SimulMT models. In this paper, we manually
annotate a monotonic test set based on the MuST-C English-Chinese test set,
denoted as SiMuST-C. Our human evaluation confirms the acceptability of our
annotated test set. Evaluations on three different SimulMT models verify that
the underestimation problem can be alleviated on our test set. Further
experiments show that finetuning on an automatically extracted monotonic
training set improves SimulMT models by up to 3 BLEU points.
