  Machine learning models are commonly tested in-distribution (same dataset);
performance almost always drops in out-of-distribution settings. For HRI
research, the goal is often to develop generalized models. This makes domain
generalization - retaining performance in different settings - a critical
issue. In this study, we present a concise analysis of domain generalization in
failure detection models trained on human facial expressions. Using two
distinct datasets of humans reacting to videos where error occurs, one from a
controlled lab setting and another collected online, we trained deep learning
models on each dataset. When testing these models on the alternate dataset, we
observed a significant performance drop. We reflect on the causes for the
observed model behavior and leave recommendations. This work emphasizes the
need for HRI research focusing on improving model robustness and real-life
applicability.
