  Over the years, multimodal mobile sensing has been used extensively for
inferences regarding health and well being, behavior, and context. However, a
significant challenge hindering the widespread deployment of such models in
real world scenarios is the issue of distribution shift. This is the phenomenon
where the distribution of data in the training set differs from the
distribution of data in the real world, the deployment environment. While
extensively explored in computer vision and natural language processing, and
while prior research in mobile sensing briefly addresses this concern, current
work primarily focuses on models dealing with a single modality of data, such
as audio or accelerometer readings, and consequently, there is little research
on unsupervised domain adaptation when dealing with multimodal sensor data. To
address this gap, we did extensive experiments with domain adversarial neural
networks (DANN) showing that they can effectively handle distribution shifts in
multimodal sensor data. Moreover, we proposed a novel improvement over DANN,
called M3BAT, unsupervised domain adaptation for multimodal mobile sensing with
multi-branch adversarial training, to account for the multimodality of sensor
data during domain adaptation with multiple branches. Through extensive
experiments conducted on two multimodal mobile sensing datasets, three
inference tasks, and 14 source-target domain pairs, including both regression
and classification, we demonstrate that our approach performs effectively on
unseen domains. Compared to directly deploying a model trained in the source
domain to the target domain, the model shows performance increases up to 12%
AUC (area under the receiver operating characteristics curves) on
classification tasks, and up to 0.13 MAE (mean absolute error) on regression
tasks.
