  The study of the applicability of the BERTScore metric was conducted to
translation quality assessment at the sentence level for English -> Russian
direction. Experiments were performed with a pre-trained Multilingual BERT as
well as with a pair of Monolingual BERT models. To align monolingual
embeddings, an orthogonal transformation based on anchor tokens was used. It
was demonstrated that such transformation helps to prevent mismatching issue
and shown that this approach gives better results than using embeddings of the
Multilingual model. To improve the token matching process it is proposed to
combine all incomplete WorkPiece tokens into meaningful words and use simple
averaging of corresponding vectors and to calculate BERTScore based on anchor
tokens only. Such modifications allowed us to achieve a better correlation of
the model predictions with human judgments. In addition to evaluating machine
translation, several versions of human translation were evaluated as well, the
problems of this approach were listed.
