  Real-world image classification tasks tend to be complex, where expert
labellers are sometimes unsure about the classes present in the images, leading
to the issue of learning with noisy labels (LNL). The ill-posedness of the LNL
task requires the adoption of strong assumptions or the use of multiple noisy
labels per training image, resulting in accurate models that work well in
isolation but fail to optimise human-AI collaborative classification (HAI-CC).
Unlike such LNL methods, HAI-CC aims to leverage the synergies between human
expertise and AI capabilities but requires clean training labels, limiting its
real-world applicability. This paper addresses this gap by introducing the
innovative Learning to Complement with Multiple Humans (LECOMH) approach.
LECOMH is designed to learn from noisy labels without depending on clean
labels, simultaneously maximising collaborative accuracy while minimising the
cost of human collaboration, measured by the number of human expert annotations
required per image. Additionally, new benchmarks featuring multiple noisy
labels for both training and testing are proposed to evaluate HAI-CC methods.
Through quantitative comparisons on these benchmarks, LECOMH consistently
outperforms competitive HAI-CC approaches, human labellers, multi-rater
learning, and noisy-label learning methods across various datasets, offering a
promising solution for addressing real-world image classification challenges.
