  Data accuracy is essential for scientific research and policy development.
The National Violent Death Reporting System (NVDRS) data is widely used for
discovering the patterns and causes of death. Recent studies suggested the
annotation inconsistencies within the NVDRS and the potential impact on
erroneous suicide-cause attributions. We present an empirical Natural Language
Processing (NLP) approach to detect annotation inconsistencies and adopt a
cross-validation-like paradigm to identify problematic instances. We analyzed
267,804 suicide death incidents between 2003 and 2020 from the NVDRS. Our
results showed that incorporating the target state's data into training the
suicide-crisis classifier brought an increase of 5.4% to the F-1 score on the
target state's test set and a decrease of 1.1% on other states' test set. To
conclude, we demonstrated the annotation inconsistencies in NVDRS's death
investigation notes, identified problematic instances, evaluated the
effectiveness of correcting problematic instances, and eventually proposed an
NLP improvement solution.
