  We propose a novel integrated framework that jointly models complementary
information from resting-state functional MRI (rs-fMRI) connectivity and
diffusion tensor imaging (DTI) tractography to extract biomarkers of brain
connectivity predictive of behavior. Our framework couples a generative model
of the connectomics data with a deep network that predicts behavioral scores.
The generative component is a structurally-regularized Dynamic Dictionary
Learning (sr-DDL) model that decomposes the dynamic rs-fMRI correlation
matrices into a collection of shared basis networks and time varying
subject-specific loadings. We use the DTI tractography to regularize this
matrix factorization and learn anatomically informed functional connectivity
profiles. The deep component of our framework is an LSTM-ANN block, which uses
the temporal evolution of the subject-specific sr-DDL loadings to predict
multidimensional clinical characterizations. Our joint optimization strategy
collectively estimates the basis networks, the subject-specific time-varying
loadings, and the neural network weights. We validate our framework on a
dataset of neurotypical individuals from the Human Connectome Project (HCP)
database to map to cognition and on a separate multi-score prediction task on
individuals diagnosed with Autism Spectrum Disorder (ASD) in a five-fold cross
validation setting. Our hybrid model outperforms several state-of-the-art
approaches at clinical outcome prediction and learns interpretable multimodal
neural signatures of brain organization.
