  In this article, we introduce a novel problem of audio-visual autism behavior
recognition, which includes social behavior recognition, an essential aspect
previously omitted in AI-assisted autism screening research. We define the task
at hand as one that is audio-visual autism behavior recognition, which uses
audio and visual cues, including any speech present in the audio, to recognize
autism-related behaviors. To facilitate this new research direction, we
collected an audio-visual autism spectrum dataset (AV-ASD), currently the
largest video dataset for autism screening using a behavioral approach. It
covers an extensive range of autism-associated behaviors, including those
related to social communication and interaction. To pave the way for further
research on this new problem, we intensively explored leveraging foundation
models and multimodal large language models across different modalities. Our
experiments on the AV-ASD dataset demonstrate that integrating audio, visual,
and speech modalities significantly enhances the performance in autism behavior
recognition. Additionally, we explored the use of a post-hoc to ad-hoc pipeline
in a multimodal large language model to investigate its potential to augment
the model's explanatory capability during autism behavior recognition. We will
release our dataset, code, and pre-trained models.
