  Inspired by the success of the General Language Understanding Evaluation
benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE)
benchmark to facilitate research in the development of pre-training language
representations in the biomedicine domain. The benchmark consists of five tasks
with ten datasets that cover both biomedical and clinical texts with different
dataset sizes and difficulties. We also evaluate several baselines based on
BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and
MIMIC-III clinical notes achieves the best results. We make the datasets,
pre-trained models, and codes publicly available at
https://github.com/ncbi-nlp/BLUE_Benchmark.
