  Complementary-label learning (CLL) is a weakly-supervised learning paradigm
that aims to train a multi-class classifier using only complementary labels,
which indicate classes to which an instance does not belong. Despite numerous
algorithmic proposals for CLL, their practical applicability remains unverified
for two reasons. Firstly, these algorithms often rely on assumptions about the
generation of complementary labels, and it is not clear how far the assumptions
are from reality. Secondly, their evaluation has been limited to synthetic
datasets. To gain insights into the real-world performance of CLL algorithms,
we developed a protocol to collect complementary labels from human annotators.
Our efforts resulted in the creation of four datasets: CLCIFAR10, CLCIFAR20,
CLMicroImageNet10, and CLMicroImageNet20, derived from well-known
classification datasets CIFAR10, CIFAR100, and TinyImageNet200. These datasets
represent the very first real-world CLL datasets. Through extensive benchmark
experiments, we discovered a notable decrease in performance when transitioning
from synthetic datasets to real-world datasets. We investigated the key factors
contributing to the decrease with a thorough dataset-level ablation study. Our
analyses highlight annotation noise as the most influential factor in the
real-world datasets. In addition, we discover that the biased-nature of
human-annotated complementary labels and the difficulty to validate with only
complementary labels are two outstanding barriers to practical CLL. These
findings suggest that the community focus more research efforts on developing
CLL algorithms and validation schemes that are robust to noisy and biased
complementary-label distributions.
