  Medical images are generally labeled by multiple experts before the final
ground-truth labels are determined. Consensus or disagreement among experts
regarding individual images reflects the gradeability and difficulty levels of
the image. However, when being used for model training, only the final
ground-truth label is utilized, while the critical information contained in the
raw multi-rater gradings regarding the image being an easy/hard case is
discarded. In this paper, we aim to take advantage of the raw multi-rater
gradings to improve the deep learning model performance for the glaucoma
classification task. Specifically, a multi-branch model structure is proposed
to predict the most sensitive, most specifical and a balanced fused result for
the input images. In order to encourage the sensitivity branch and specificity
branch to generate consistent results for consensus labels and opposite results
for disagreement labels, a consensus loss is proposed to constrain the output
of the two branches. Meanwhile, the consistency/inconsistency between the
prediction results of the two branches implies the image being an easy/hard
case, which is further utilized to encourage the balanced fusion branch to
concentrate more on the hard cases. Compared with models trained only with the
final ground-truth labels, the proposed method using multi-rater consensus
information has achieved superior performance, and it is also able to estimate
the difficulty levels of individual input images when making the prediction.
