  Gestures are inherent to human interaction and often complement speech in
face-to-face communication, forming a multimodal communication system. An
important task in gesture analysis is detecting a gesture's beginning and end.
Research on automatic gesture detection has primarily focused on visual and
kinematic information to detect a limited set of isolated or silent gestures
with low variability, neglecting the integration of speech and vision signals
to detect gestures that co-occur with speech. This work addresses this gap by
focusing on co-speech gesture detection, emphasising the synchrony between
speech and co-speech hand gestures. We address three main challenges: the
variability of gesture forms, the temporal misalignment between gesture and
speech onsets, and differences in sampling rate between modalities. We
investigate extended speech time windows and employ separate backbone models
for each modality to address the temporal misalignment and sampling rate
differences. We utilize Transformer encoders in cross-modal and early fusion
techniques to effectively align and integrate speech and skeletal sequences.
The study results show that combining visual and speech information
significantly enhances gesture detection performance. Our findings indicate
that expanding the speech buffer beyond visual time segments improves
performance and that multimodal integration using cross-modal and early fusion
techniques outperforms baseline methods using unimodal and late fusion methods.
Additionally, we find a correlation between the models' gesture prediction
confidence and low-level speech frequency features potentially associated with
gestures. Overall, the study provides a better understanding and detection
methods for co-speech gestures, facilitating the analysis of multimodal
communication.
