  Functional Magnetic Resonance Imaging (fMRI) captures the temporal dynamics
of neural activity as a function of spatial location in the brain. Thus, fMRI
scans are represented as 4-Dimensional (3-space + 1-time) tensors. And it is
widely believed that the spatio-temporal patterns in fMRI manifests as
behaviour and clinical symptoms. Because of the high dimensionality ($\sim$ 1
Million) of fMRI, and the added constraints of limited cardinality of data
sets, extracting such patterns are challenging. A standard approach to overcome
these hurdles is to reduce the dimensionality of the data by either summarizing
activation over time or space at the expense of possible loss of useful
information. Here, we introduce an end-to-end algorithm capable of extracting
spatiotemporal features from the full 4-D data using 3-D CNNs and 3-D
Convolutional LSTMs. We evaluate our proposed model on the publicly available
ABIDE dataset to demonstrate the capability of our model to classify Autism
Spectrum Disorder (ASD) from resting-state fMRI data. Our results show that the
proposed model achieves state of the art results on single sites with F1-scores
of 0.78 and 0.7 on NYU and UM sites, respectively.
