  Medical coding is the task of assigning medical codes to clinical free-text
documentation. Healthcare professionals manually assign such codes to track
patient diagnoses and treatments. Automated medical coding can considerably
alleviate this administrative burden. In this paper, we reproduce, compare, and
analyze state-of-the-art automated medical coding machine learning models. We
show that several models underperform due to weak configurations, poorly
sampled train-test splits, and insufficient evaluation. In previous work, the
macro F1 score has been calculated sub-optimally, and our correction doubles
it. We contribute a revised model comparison using stratified sampling and
identical experimental setups, including hyperparameters and decision boundary
tuning. We analyze prediction errors to validate and falsify assumptions of
previous works. The analysis confirms that all models struggle with rare codes,
while long documents only have a negligible impact. Finally, we present the
first comprehensive results on the newly released MIMIC-IV dataset using the
reproduced models. We release our code, model parameters, and new MIMIC-III and
MIMIC-IV training and evaluation pipelines to accommodate fair future
comparisons.
