  While neural network-based models have achieved impressive performance on a
large body of NLP tasks, the generalization behavior of different models
remains poorly understood: Does this excellent performance imply a perfect
generalization model, or are there still some limitations? In this paper, we
take the NER task as a testbed to analyze the generalization behavior of
existing models from different perspectives and characterize the differences of
their generalization abilities through the lens of our proposed measures, which
guides us to better design models and training methods. Experiments with
in-depth analyses diagnose the bottleneck of existing neural NER models in
terms of breakdown performance analysis, annotation errors, dataset bias, and
category relationships, which suggest directions for improvement. We have
released the datasets: (ReCoNLL, PLONER) for the future research at our project
page: http://pfliu.com/InterpretNER/. As a by-product of this paper, we have
open-sourced a project that involves a comprehensive summary of recent NER
papers and classifies them into different research topics:
https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers.
