  During collaborative learning, confusion and conflict emerge naturally.
However, persistent confusion or conflict have the potential to generate
frustration and significantly impede learners' performance. Early automatic
detection of confusion and conflict would allow us to support early
interventions which can in turn improve students' experience with and outcomes
from collaborative learning. Despite the extensive studies modeling confusion
during solo learning, there is a need for further work in collaborative
learning. This paper presents a multimodal machine-learning framework that
automatically detects confusion and conflict during collaborative learning. We
used data from 38 elementary school learners who collaborated on a series of
programming tasks in classrooms. We trained deep multimodal learning models to
detect confusion and conflict using features that were automatically extracted
from learners' collaborative dialogues, including (1) language-derived features
including TF-IDF, lexical semantics, and sentiment, (2) audio-derived features
including acoustic-prosodic features, and (3) video-derived features including
eye gaze, head pose, and facial expressions. Our results show that multimodal
models that combine semantics, pitch, and facial expressions detected confusion
and conflict with the highest accuracy, outperforming all unimodal models. We
also found that prosodic cues are more predictive of conflict, and facial cues
are more predictive of confusion. This study contributes to the automated
modeling of collaborative learning processes and the development of real-time
adaptive support to enhance learners' collaborative learning experience in
classroom contexts.
