  Assessing the performance of interpreting services is a complex task, given
the nuanced nature of spoken language translation, the strategies that
interpreters apply, and the diverse expectations of users. The complexity of
this task become even more pronounced when automated evaluation methods are
applied. This is particularly true because interpreted texts exhibit less
linearity between the source and target languages due to the strategies
employed by the interpreter.
  This study aims to assess the reliability of automatic metrics in evaluating
simultaneous interpretations by analyzing their correlation with human
evaluations. We focus on a particular feature of interpretation quality, namely
translation accuracy or faithfulness. As a benchmark we use human assessments
performed by language experts, and evaluate how well sentence embeddings and
Large Language Models correlate with them. We quantify semantic similarity
between the source and translated texts without relying on a reference
translation. The results suggest GPT models, particularly GPT-3.5 with direct
prompting, demonstrate the strongest correlation with human judgment in terms
of semantic similarity between source and target texts, even when evaluating
short textual segments. Additionally, the study reveals that the size of the
context window has a notable impact on this correlation.
