  The number of biomedical literature on new biomedical concepts is rapidly
increasing, which necessitates a reliable biomedical named entity recognition
(BioNER) model for identifying new and unseen entity mentions. However, it is
questionable whether existing models can effectively handle them. In this work,
we systematically analyze the three types of recognition abilities of BioNER
models: memorization, synonym generalization, and concept generalization. We
find that although current best models achieve state-of-the-art performance on
benchmarks based on overall performance, they have limitations in identifying
synonyms and new biomedical concepts, indicating they are overestimated in
terms of their generalization abilities. We also investigate failure cases of
models and identify several difficulties in recognizing unseen mentions in
biomedical literature as follows: (1) models tend to exploit dataset biases,
which hinders the models' abilities to generalize, and (2) several biomedical
names have novel morphological patterns with weak name regularity, and models
fail to recognize them. We apply a statistics-based debiasing method to our
problem as a simple remedy and show the improvement in generalization to unseen
mentions. We hope that our analyses and findings would be able to facilitate
further research into the generalization capabilities of NER models in a domain
where their reliability is of utmost importance.
