  Clinical outcome prediction plays an important role in stroke patient
management. From a machine learning point-of-view, one of the main challenges
is dealing with heterogeneous data at patient admission, i.e. the image data
which are multidimensional and the clinical data which are scalars. In this
paper, a multimodal convolutional neural network - long short-term memory
(CNN-LSTM) based ensemble model is proposed. For each MR image module, a
dedicated network provides preliminary prediction of the clinical outcome using
the modified Rankin scale (mRS). The final mRS score is obtained by merging the
preliminary probabilities of each module dedicated to a specific type of MR
image weighted by the clinical metadata, here age or the National Institutes of
Health Stroke Scale (NIHSS). The experimental results demonstrate that the
proposed model surpasses the baselines and offers an original way to
automatically encode the spatio-temporal context of MR images in a deep
learning architecture. The highest AUC (0.77) was achieved for the proposed
model with NIHSS.
