  Sentence-level Quality estimation (QE) of machine translation is
traditionally formulated as a regression task, and the performance of QE models
is typically measured by Pearson correlation with human labels. Recent QE
models have achieved previously-unseen levels of correlation with human
judgments, but they rely on large multilingual contextualized language models
that are computationally expensive and make them infeasible for real-world
applications. In this work, we evaluate several model compression techniques
for QE and find that, despite their popularity in other NLP tasks, they lead to
poor performance in this regression setting. We observe that a full model
parameterization is required to achieve SoTA results in a regression task.
However, we argue that the level of expressiveness of a model in a continuous
range is unnecessary given the downstream applications of QE, and show that
reframing QE as a classification problem and evaluating QE models using
classification metrics would better reflect their actual performance in
real-world applications.
