  We developed a novel, interpretable multimodal classification method to
identify symptoms of mood disorders viz. depression, anxiety and anhedonia
using audio, video and text collected from a smartphone application. We used
CNN-based unimodal encoders to learn dynamic embeddings for each modality and
then combined these through a transformer encoder. We applied these methods to
a novel dataset - collected by a smartphone application - on 3002 participants
across up to three recording sessions. Our method demonstrated better
multimodal classification performance compared to existing methods that
employed static embeddings. Lastly, we used SHapley Additive exPlanations
(SHAP) to prioritize important features in our model that could serve as
potential digital markers.
