  Detecting mental states of human users is crucial for the development of
cooperative and intelligent robots, as it enables the robot to understand the
user's intentions and desires. Despite their importance, it is difficult to
obtain a large amount of high quality data for training automatic recognition
algorithms as the time and effort required to collect and label such data is
prohibitively high. In this paper we present a multimodal machine learning
approach for detecting dis-/agreement and confusion states in a human-robot
interaction environment, using just a small amount of manually annotated data.
We collect a data set by conducting a human-robot interaction study and develop
a novel preprocessing pipeline for our machine learning approach. By combining
semi-supervised and supervised architectures, we are able to achieve an average
F1-score of 81.1\% for dis-/agreement detection with a small amount of labeled
data and a large unlabeled data set, while simultaneously increasing the
robustness of the model compared to the supervised approach.
