  Turn-taking has played an essential role in structuring the regulation of a
conversation. The task of identifying the main speaker (who is properly taking
his/her turn of speaking) and the interrupters (who are interrupting or
reacting to the main speaker's utterances) remains a challenging task. Although
some prior methods have partially addressed this task, there still remain some
limitations. Firstly, a direct association of Audio and Visual features may
limit the correlations to be extracted due to different modalities. Secondly,
the relationship across temporal segments helping to maintain the consistency
of localization, separation, and conversation contexts is not effectively
exploited. Finally, the interactions between speakers that usually contain the
tracking and anticipatory decisions about the transition to a new speaker are
usually ignored. Therefore, this work introduces a new Audio-Visual Transformer
approach to the problem of localization and highlighting the main speaker in
both audio and visual channels of a multi-speaker conversation video in the
wild. The proposed method exploits different types of correlations presented in
both visual and audio signals. The temporal audio-visual relationships across
spatial-temporal space are anticipated and optimized via the self-attention
mechanism in a Transformerstructure. Moreover, a newly collected dataset is
introduced for the main speaker detection. To the best of our knowledge, it is
one of the first studies that is able to automatically localize and highlight
the main speaker in both visual and audio channels in multi-speaker
conversation videos.
