  In this paper, we explore various multilingual and Russian pre-trained
transformer-based models for the Dialogue Evaluation 2021 shared task on
headline selection. Our experiments show that the combined approach is superior
to individual multilingual and monolingual models. We present an analysis of a
number of ways to obtain sentence embeddings and learn a ranking model on top
of them. We achieve the result of 87.28% and 86.60% accuracy for the public and
private test sets respectively.
