  Labeling is the cornerstone of supervised machine learning, which has been
exploited in a plethora of various applications, with sign language recognition
being one of them. However, such algorithms must be fed with a huge amount of
consistently labeled data during the training process to elaborate a
well-generalizing model. In addition, there is a great need for an automated
solution that works with any nationally diversified sign language. Although
there are language-agnostic transcription systems, such as the Hamburg Sign
Language Notation System (HamNoSys) that describe the signer's initial position
and body movement instead of the glosses' meanings, there are still issues with
providing accurate and reliable labels for every real-world use case. In this
context, the industry relies heavily on manual attribution and labeling of the
available video data. In this work, we tackle this issue and thoroughly analyze
the HamNoSys labels provided by various maintainers of open sign language
corpora in five sign languages, in order to examine the challenges encountered
in labeling video data. We also investigate the consistency and objectivity of
HamNoSys-based labels for the purpose of training machine learning models. Our
findings provide valuable insights into the limitations of the current labeling
methods and pave the way for future research on developing more accurate and
efficient solutions for sign language recognition.
