  The goal of sign language recognition (SLR) is to help those who are hard of
hearing or deaf overcome the communication barrier. Most existing approaches
can be typically divided into two lines, i.e., Skeleton-based and RGB-based
methods, but both the two lines of methods have their limitations.
Skeleton-based methods do not consider facial expressions, while RGB-based
approaches usually ignore the fine-grained hand structure. To overcome both
limitations, we propose a new framework called Spatial-temporal Part-aware
network~(StepNet), based on RGB parts. As its name suggests, it is made up of
two modules: Part-level Spatial Modeling and Part-level Temporal Modeling.
Part-level Spatial Modeling, in particular, automatically captures the
appearance-based properties, such as hands and faces, in the feature space
without the use of any keypoint-level annotations. On the other hand,
Part-level Temporal Modeling implicitly mines the long-short term context to
capture the relevant attributes over time. Extensive experiments demonstrate
that our StepNet, thanks to spatial-temporal modules, achieves competitive
Top-1 Per-instance accuracy on three commonly-used SLR benchmarks, i.e., 56.89%
on WLASL, 77.2% on NMFs-CSL, and 77.1% on BOBSL. Additionally, the proposed
method is compatible with the optical flow input and can produce superior
performance if fused. For those who are hard of hearing, we hope that our work
can act as a preliminary step.
