  State-of-the-art trainable machine translation evaluation metrics like xCOMET
achieve high correlation with human judgment but rely on large encoders (up to
10.7B parameters), making them computationally expensive and inaccessible to
researchers with limited resources. To address this issue, we investigate
whether the knowledge stored in these large encoders can be compressed while
maintaining quality. We employ distillation, quantization, and pruning
techniques to create efficient xCOMET alternatives and introduce a novel data
collection pipeline for efficient black-box distillation. Our experiments show
that, using quantization, xCOMET can be compressed up to three times with no
quality degradation. Additionally, through distillation, we create an
xCOMET-lite metric, which has only 2.6% of xCOMET-XXL parameters, but retains
92.1% of its quality. Besides, it surpasses strong small-scale metrics like
COMET-22 and BLEURT-20 on the WMT22 metrics challenge dataset by 6.4%, despite
using 50% fewer parameters. All code, dataset, and models are available online.
