  The advancement of natural language processing (NLP) systems in healthcare
hinges on language model ability to interpret the intricate information
contained within clinical notes. This process often requires integrating
information from various time points in a patient's medical history. However,
most earlier clinical language models were pretrained with a context length
limited to roughly one clinical document. In this study, We introduce
ClinicalMamba, a specialized version of the Mamba language model, pretrained on
a vast corpus of longitudinal clinical notes to address the unique linguistic
characteristics and information processing needs of the medical domain.
ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a
superior performance in modeling clinical language across extended text lengths
compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba
achieves notable benchmarks in speed and accuracy, outperforming existing
clinical language models and general domain large models like GPT-4 in
longitudinal clinical notes information extraction tasks.
