  Data annotation plays a crucial role in ensuring your named entity
recognition (NER) projects are trained with the right information to learn
from. Producing the most accurate labels is a challenge due to the complexity
involved with annotation. Label inconsistency between multiple subsets of data
annotation (e.g., training set and test set, or multiple training subsets) is
an indicator of label mistakes. In this work, we present an empirical method to
explore the relationship between label (in-)consistency and NER model
performance. It can be used to validate the label consistency (or catches the
inconsistency) in multiple sets of NER data annotation. In experiments, our
method identified the label inconsistency of test data in SCIERC and CoNLL03
datasets (with 26.7% and 5.4% label mistakes). It validated the consistency in
the corrected version of both datasets.
