  The newly emerged transformer technology has a tremendous impact on NLP
research. In the general English domain, transformer-based models have achieved
state-of-the-art performances on various NLP benchmarks. In the clinical
domain, researchers also have investigated transformer models for clinical
applications. The goal of this study is to systematically explore three widely
used transformer-based models (i.e., BERT, RoBERTa, and XLNet) for clinical
relation extraction and develop an open-source package with clinical
pre-trained transformer-based models to facilitate information extraction in
the clinical domain. We developed a series of clinical RE models based on three
transformer architectures, namely BERT, RoBERTa, and XLNet. We evaluated these
models using 2 publicly available datasets from 2018 MADE1.0 and 2018 n2c2
challenges. We compared two classification strategies (binary vs. multi-class
classification) and investigated two approaches to generate candidate relations
in different experimental settings. In this study, we compared three
transformer-based (BERT, RoBERTa, and XLNet) models for relation extraction. We
demonstrated that the RoBERTa-clinical RE model achieved the best performance
on the 2018 MADE1.0 dataset with an F1-score of 0.8958. On the 2018 n2c2
dataset, the XLNet-clinical model achieved the best F1-score of 0.9610. Our
results indicated that the binary classification strategy consistently
outperformed the multi-class classification strategy for clinical relation
extraction. Our methods and models are publicly available at
https://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction.
We believe this work will improve current practice on clinical relation
extraction and other related NLP tasks in the biomedical domain.
