  Automatically classifying electronic health records (EHRs) into diagnostic
codes has been challenging to the NLP community. State-of-the-art methods
treated this problem as a multilabel classification problem and proposed
various architectures to model this problem. However, these systems did not
leverage the superb performance of pretrained language models, which achieved
superb performance on natural language understanding tasks. Prior work has
shown that pretrained language models underperformed on this task with the
regular finetuning scheme. Therefore, this paper aims at analyzing the causes
of the underperformance and developing a framework for automatic ICD coding
with pretrained language models. We spotted three main issues through the
experiments: 1) large label space, 2) long input sequences, and 3) domain
mismatch between pretraining and fine-tuning. We propose PLMICD, a framework
that tackles the challenges with various strategies. The experimental results
show that our proposed framework can overcome the challenges and achieves
state-of-the-art performance in terms of multiple metrics on the benchmark
MIMIC data. The source code is available at https://github.com/MiuLab/PLM-ICD
