  Background: Encouraged by the success of pretrained Transformer models in
many natural language processing tasks, their use for International
Classification of Diseases (ICD) coding tasks is now actively being explored.
In this study, we investigate three types of Transformer-based models, aiming
to address the extreme label set and long text classification challenges that
are posed by automated ICD coding tasks. Methods: The Transformer-based model
PLM-ICD achieved the current state-of-the-art (SOTA) performance on the ICD
coding benchmark dataset MIMIC-III. It was chosen as our baseline model to be
further optimised. XR-Transformer, the new SOTA model in the general extreme
multi-label text classification domain, and XR-LAT, a novel adaptation of the
XR-Transformer model, were also trained on the MIMIC-III dataset. XR-LAT is a
recursively trained model chain on a predefined hierarchical code tree with
label-wise attention, knowledge transferring and dynamic negative sampling
mechanisms. Results: Our optimised PLM-ICD model, which was trained with longer
total and chunk sequence lengths, significantly outperformed the current SOTA
PLM-ICD model, and achieved the highest micro-F1 score of 60.8%. The
XR-Transformer model, although SOTA in the general domain, did not perform well
across all metrics. The best XR-LAT based model obtained results that were
competitive with the current SOTA PLM-ICD model, including improving the
macro-AUC by 2.1%. Conclusion: Our optimised PLM-ICD model is the new SOTA
model for automated ICD coding on the MIMIC-III dataset, while our novel XR-LAT
model performs competitively with the previous SOTA PLM-ICD model.
