  Vision-based Continuous Sign Language Recognition (CSLR) aims to recognize
unsegmented signs from image streams. Overfitting is one of the most critical
problems in CSLR training, and previous works show that the iterative training
scheme can partially solve this problem while also costing more training time.
In this study, we revisit the iterative training scheme in recent CSLR works
and realize that sufficient training of the feature extractor is critical to
solving the overfitting problem. Therefore, we propose a Visual Alignment
Constraint (VAC) to enhance the feature extractor with alignment supervision.
Specifically, the proposed VAC comprises two auxiliary losses: one focuses on
visual features only, and the other enforces prediction alignment between the
feature extractor and the alignment module. Moreover, we propose two metrics to
reflect overfitting by measuring the prediction inconsistency between the
feature extractor and the alignment module. Experimental results on two
challenging CSLR datasets show that the proposed VAC makes CSLR networks
end-to-end trainable and achieves competitive performance.
