  Unsupervised learning of the Dawid-Skene (D&S) model from noisy, incomplete
and crowdsourced annotations has been a long-standing challenge, and is a
critical step towards reliably labeling massive data. A recent work takes a
coupled nonnegative matrix factorization (CNMF) perspective, and shows
appealing features: It ensures the identifiability of the D\&S model and enjoys
low sample complexity, as only the estimates of the co-occurrences of annotator
labels are involved. However, the identifiability holds only when certain
somewhat restrictive conditions are met in the context of crowdsourcing.
Optimizing the CNMF criterion is also costly -- and convergence assurances are
elusive. This work recasts the pairwise co-occurrence based D&S model learning
problem as a symmetric NMF (SymNMF) problem -- which offers enhanced
identifiability relative to CNMF. In practice, the SymNMF model is often
(largely) incomplete, due to the lack of co-labeled items by some annotators.
Two lightweight algorithms are proposed for co-occurrence imputation. Then, a
low-complexity shifted rectified linear unit (ReLU)-empowered SymNMF algorithm
is proposed to identify the D&S model. Various performance characterizations
(e.g., missing co-occurrence recoverability, stability, and convergence) and
evaluations are also presented.
