  We introduce a new, extensive multidimensional quality metrics (MQM)
annotated dataset covering 11 language pairs in the biomedical domain. We use
this dataset to investigate whether machine translation (MT) metrics which are
fine-tuned on human-generated MT quality judgements are robust to domain shifts
between training and inference. We find that fine-tuned metrics exhibit a
substantial performance drop in the unseen domain scenario relative to metrics
that rely on the surface form, as well as pre-trained metrics which are not
fine-tuned on MT quality judgments.
