  Electronic health records represent a holistic overview of patients'
trajectories. Their increasing availability has fueled new hopes to leverage
them and develop accurate risk prediction models for a wide range of diseases.
Given the complex interrelationships of medical records and patient outcomes,
deep learning models have shown clear merits in achieving this goal. However, a
key limitation of these models remains their capacity in processing long
sequences. Capturing the whole history of medical encounters is expected to
lead to more accurate predictions, but the inclusion of records collected for
decades and from multiple resources can inevitably exceed the receptive field
of the existing deep learning architectures. This can result in missing
crucial, long-term dependencies. To address this gap, we present Hi-BEHRT, a
hierarchical Transformer-based model that can significantly expand the
receptive field of Transformers and extract associations from much longer
sequences. Using a multimodal large-scale linked longitudinal electronic health
records, the Hi-BEHRT exceeds the state-of-the-art BEHRT 1% to 5% for area
under the receiver operating characteristic (AUROC) curve and 3% to 6% for area
under the precision recall (AUPRC) curve on average, and 3% to 6% (AUROC) and
3% to 11% (AUPRC) for patients with long medical history for 5-year heart
failure, diabetes, chronic kidney disease, and stroke risk prediction.
Additionally, because pretraining for hierarchical Transformer is not
well-established, we provide an effective end-to-end contrastive pre-training
strategy for Hi-BEHRT using EHR, improving its transferability on predicting
clinical events with relatively small training dataset.
