  Large-scale datasets are important for the development of deep learning
models. Such datasets usually require a heavy workload of annotations, which
are extremely time-consuming and expensive. To accelerate the annotation
procedure, multiple annotators may be employed to label different subsets of
the data. However, the inconsistency and bias among different annotators are
harmful to the model training, especially for qualitative and subjective
tasks.To address this challenge, in this paper, we propose a novel contrastive
regression framework to address the disjoint annotations problem, where each
sample is labeled by only one annotator and multiple annotators work on
disjoint subsets of the data. To take account of both the intra-annotator
consistency and inter-annotator inconsistency, two strategies are
employed.Firstly, a contrastive-based loss is applied to learn the relative
ranking among different samples of the same annotator, with the assumption that
the ranking of samples from the same annotator is unanimous. Secondly, we apply
the gradient reversal layer to learn robust representations that are invariant
to different annotators. Experiments on the facial expression prediction task,
as well as the image quality assessment task, verify the effectiveness of our
proposed framework.
