  Many continuous sign language recognition (CSLR) studies adopt
transformer-based architectures for sequence modeling due to their powerful
capacity for capturing global contexts. Nevertheless, vanilla self-attention,
which serves as the core module of the transformer, calculates a weighted
average over all time steps; therefore, the local temporal semantics of sign
videos may not be fully exploited. In this study, we introduce a novel module
in sign language recognition studies, called intra-inter gloss attention
module, to leverage the relationships among frames within glosses and the
semantic and grammatical dependencies between glosses in the video. In the
intra-gloss attention module, the video is divided into equally sized chunks
and a self-attention mechanism is applied within each chunk. This localized
self-attention significantly reduces complexity and eliminates noise introduced
by considering non-relative frames. In the inter-gloss attention module, we
first aggregate the chunk-level features within each gloss chunk by average
pooling along the temporal dimension. Subsequently, multi-head self-attention
is applied to all chunk-level features. Given the non-significance of the
signer-environment interaction, we utilize segmentation to remove the
background of the videos. This enables the proposed model to direct its focus
toward the signer. Experimental results on the PHOENIX-2014 benchmark dataset
demonstrate that our method can effectively extract sign language features in
an end-to-end manner without any prior knowledge, improve the accuracy of CSLR,
and achieve the word error rate (WER) of 20.4 on the test set which is a
competitive result compare to the state-of-the-art which uses additional
supervisions.
