  One third of stroke survivors have language difficulties. Emerging evidence
suggests that their likelihood of recovery depends mainly on the damage to
language centers. Thus previous research for predicting language recovery
post-stroke has focused on identifying damaged regions of the brain. In this
paper, we introduce a novel method where we only make use of stitched
2-dimensional cross-sections of raw MRI scans in a deep convolutional neural
network setup to predict language recovery post-stroke. Our results show: a)
the proposed model that only uses MRI scans has comparable performance to
models that are dependent on lesion specific information; b) the features
learned by our model are complementary to the lesion specific information and
the combination of both appear to outperform previously reported results in
similar settings. We further analyse the CNN model for understanding regions in
brain that are responsible for arriving at these predictions using gradient
based saliency maps. Our findings are in line with previous lesion studies.
