  Simultaneous interpretation (SI), the translation of one language to another
in real time, starts translation before the original speech has finished. Its
evaluation needs to consider both latency and quality. This trade-off is
challenging especially for distant word order language pairs such as English
and Japanese. To handle this word order gap, interpreters maintain the word
order of the source language as much as possible to keep up with original
language to minimize its latency while maintaining its quality, whereas in
translation reordering happens to keep fluency in the target language. This
means outputs synchronized with the source language are desirable based on the
real SI situation, and it's a key for further progress in computational SI and
simultaneous machine translation (SiMT). In this work, we propose an automatic
evaluation metric for SI and SiMT focusing on word order synchronization. Our
evaluation metric is based on rank correlation coefficients, leveraging
cross-lingual pre-trained language models. Our experimental results on
NAIST-SIC-Aligned and JNPC showed our metrics' effectiveness to measure word
order synchronization between source and target language.
