  Named Entity Recognition systems achieve remarkable performance on domains
such as English news. It is natural to ask: What are these models actually
learning to achieve this? Are they merely memorizing the names themselves? Or
are they capable of interpreting the text and inferring the correct entity type
from the linguistic context? We examine these questions by contrasting the
performance of several variants of LSTM-CRF architectures for named entity
recognition, with some provided only representations of the context as
features. We also perform similar experiments for BERT. We find that context
representations do contribute to system performance, but that the main factor
driving high performance is learning the name tokens themselves. We enlist
human annotators to evaluate the feasibility of inferring entity types from the
context alone and find that, while people are not able to infer the entity type
either for the majority of the errors made by the context-only system, there is
some room for improvement. A system should be able to recognize any name in a
predictive context correctly and our experiments indicate that current systems
may be further improved by such capability.
