  The UMLS Metathesaurus integrates more than 200 biomedical source
vocabularies. During the Metathesaurus construction process, synonymous terms
are clustered into concepts by human editors, assisted by lexical similarity
algorithms. This process is error-prone and time-consuming. Recently, a deep
learning model (LexLM) has been developed for the UMLS Vocabulary Alignment
(UVA) task. This work introduces UBERT, a BERT-based language model, pretrained
on UMLS terms via a supervised Synonymy Prediction (SP) task replacing the
original Next Sentence Prediction (NSP) task. The effectiveness of UBERT for
UMLS Metathesaurus construction process is evaluated using the UMLS Vocabulary
Alignment (UVA) task. We show that UBERT outperforms the LexLM, as well as
biomedical BERT-based models. Key to the performance of UBERT are the synonymy
prediction task specifically developed for UBERT, the tight alignment of
training data to the UVA task, and the similarity of the models used for
pretrained UBERT.
