  Mental disorders present challenges in diagnosis and treatment due to their
complex and heterogeneous nature. Electroencephalogram (EEG) has shown promise
as a potential biomarker for these disorders. However, existing methods for
analyzing EEG signals have limitations in addressing heterogeneity and
capturing complex brain activity patterns between regions. This paper proposes
a novel random effects state-space model (RESSM) for analyzing large-scale
multi-channel resting-state EEG signals, accounting for the heterogeneity of
brain connectivities between groups and individual subjects. We incorporate
multi-level random effects for temporal dynamical and spatial mapping matrices
and address nonstationarity so that the brain connectivity patterns can vary
over time. The model is fitted under a Bayesian hierarchical model framework
coupled with a Gibbs sampler. Compared to previous mixed-effects state-space
models, we directly model high-dimensional random effects matrices without
structural constraints and tackle the challenge of identifiability. Through
extensive simulation studies, we demonstrate that our approach yields valid
estimation and inference. We apply RESSM to a multi-site clinical trial of
Major Depressive Disorder (MDD). Our analysis uncovers significant differences
in resting-state brain temporal dynamics among MDD patients compared to healthy
individuals. In addition, we show the subject-level EEG features derived from
RESSM exhibit a superior predictive value for the heterogeneous treatment
effect compared to the EEG frequency band power, suggesting the potential of
EEG as a valuable biomarker for MDD.
