  Both generic and domain-specific BERT models are widely used for natural
language processing (NLP) tasks. In this paper we investigate the vulnerability
of BERT models to variation in input data for Named Entity Recognition (NER)
through adversarial attack. Experimental results show that BERT models are
vulnerable to variation in the entity context with 20.2 to 45.0% of entities
predicted completely wrong and another 29.3 to 53.3% of entities predicted
wrong partially. BERT models seem most vulnerable to changes in the local
context of entities and often a single change is sufficient to fool the model.
The domain-specific BERT model trained from scratch (SciBERT) is more
vulnerable than the original BERT model or the domain-specific model that
retains the BERT vocabulary (BioBERT). We also find that BERT models are
particularly vulnerable to emergent entities. Our results chart the
vulnerabilities of BERT models for NER and emphasize the importance of further
research into uncovering and reducing these weaknesses.
