  Clinical coding is currently a labour-intensive, error-prone, but critical
administrative process whereby hospital patient episodes are manually assigned
codes by qualified staff from large, standardised taxonomic hierarchies of
codes. Automating clinical coding has a long history in NLP research and has
recently seen novel developments setting new state of the art results. A
popular dataset used in this task is MIMIC-III, a large intensive care database
that includes clinical free text notes and associated codes. We argue for the
reconsideration of the validity MIMIC-III's assigned codes that are often
treated as gold-standard, especially when MIMIC-III has not undergone secondary
validation. This work presents an open-source, reproducible experimental
methodology for assessing the validity of codes derived from EHR discharge
summaries. We exemplify the methodology with MIMIC-III discharge summaries and
show the most frequently assigned codes in MIMIC-III are under-coded up to 35%.
