  Decoder-only transformers are the backbone of the popular generative
pre-trained transformer (GPT) series of large language models. In this work, we
apply the same framework to periodic heart time-series data to create two
pre-trained general purpose cardiac models, namely PPG-PT and ECG-PT. We
demonstrate that both such pre-trained models are fully interpretable. This is
achieved firstly through aggregate attention maps which show that the model
focuses on similar points in previous cardiac cycles in order to make
predictions and gradually broadens its attention in deeper layers. Next, tokens
with the same value, that occur at different distinct points in the ECG and PPG
cycle, form separate clusters in high dimensional space based on their phase as
they propagate through the transformer blocks. Finally, we highlight that
individual attention heads respond to specific physiologically relevent
features, such as the dicrotic notch in PPG and the P-wave in ECG. It is also
demonstrated that these pre-trained models can be easily fine-tuned for tasks
such as classification of atrial fibrillation. In this specific example, the
fine-tuning took 11 minutes of computer time, and achieved a
leave-one-subject-out AUCs of 0.99 and 0.93 for ECG and PPG respectively.
Importantly, these fine-tuned models are also fully explainable, with attention
shifting to regions in the context that are strongly indicative of atrial
fibrillation.
