  We present a novel multimodal deep learning framework for cardiac
resynchronisation therapy (CRT) response prediction from 2D echocardiography
and cardiac magnetic resonance (CMR) data. The proposed method first uses the
`nnU-Net' segmentation model to extract segmentations of the heart over the
full cardiac cycle from the two modalities. Next, a multimodal deep learning
classifier is used for CRT response prediction, which combines the latent
spaces of the segmentation models of the two modalities. At inference time,
this framework can be used with 2D echocardiography data only, whilst taking
advantage of the implicit relationship between CMR and echocardiography
features learnt from the model. We evaluate our pipeline on a cohort of 50 CRT
patients for whom paired echocardiography/CMR data were available, and results
show that the proposed multimodal classifier results in a statistically
significant improvement in accuracy compared to the baseline approach that uses
only 2D echocardiography data. The combination of multimodal data enables CRT
response to be predicted with 77.38% accuracy (83.33% sensitivity and 71.43%
specificity), which is comparable with the current state-of-the-art in machine
learning-based CRT response prediction. Our work represents the first
multimodal deep learning approach for CRT response prediction.
