  Acute kidney injury (AKI) is a common clinical syndrome characterized by a
sudden episode of kidney failure or kidney damage within a few hours or a few
days. Accurate early prediction of AKI for patients in ICU who are more likely
than others to have AKI can enable timely interventions, and reduce the
complications of AKI. Much of the clinical information relevant to AKI is
captured in clinical notes that are largely unstructured text and requires
advanced natural language processing (NLP) for useful information extraction.
On the other hand, pre-trained contextual language models such as Bidirectional
Encoder Representations from Transformers (BERT) have improved performances for
many NLP tasks in general domain recently. However, few have explored BERT on
disease-specific medical domain tasks such as AKI early prediction. In this
paper, we try to apply BERT to specific diseases and present an AKI
domain-specific pre-trained language model based on BERT (AKI-BERT) that could
be used to mine the clinical notes for early prediction of AKI. AKI-BERT is a
BERT model pre-trained on the clinical notes of patients having risks for AKI.
Our experiments on Medical Information Mart for Intensive Care III (MIMIC-III)
dataset demonstrate that AKI-BERT can yield performance improvements for early
AKI prediction, thus expanding the utility of the BERT model from general
clinical domain to disease-specific domain.
