  We propose MToMnet - a Theory of Mind (ToM) neural network for predicting
beliefs and their dynamics during human social interactions from multimodal
input. ToM is key for effective nonverbal human communication and
collaboration, yet, existing methods for belief modelling have not included
explicit ToM modelling or have typically been limited to one or two modalities.
MToMnet encodes contextual cues (scene videos and object locations) and
integrates them with person-specific cues (human gaze and body language) in a
separate MindNet for each person. Inspired by prior research on social
cognition and computational ToM, we propose three different MToMnet variants:
two involving fusion of latent representations and one involving re-ranking of
classification scores. We evaluate our approach on two challenging real-world
datasets, one focusing on belief prediction, while the other examining belief
dynamics prediction. Our results demonstrate that MToMnet surpasses existing
methods by a large margin while at the same time requiring a significantly
smaller number of parameters. Taken together, our method opens up a highly
promising direction for future work on artificial intelligent systems that can
robustly predict human beliefs from their non-verbal behaviour and, as such,
more effectively collaborate with humans.
