  Multi-label image recognition is a fundamental yet practical task because
real-world images inherently possess multiple semantic labels. However, it is
difficult to collect large-scale multi-label annotations due to the complexity
of both the input images and output label spaces. To reduce the annotation
cost, we propose a structured semantic transfer (SST) framework that enables
training multi-label recognition models with partial labels, i.e., merely some
labels are known while other labels are missing (also called unknown labels)
per image. The framework consists of two complementary transfer modules that
explore within-image and cross-image semantic correlations to transfer
knowledge of known labels to generate pseudo labels for unknown labels.
Specifically, an intra-image semantic transfer module learns image-specific
label co-occurrence matrix and maps the known labels to complement unknown
labels based on this matrix. Meanwhile, a cross-image transfer module learns
category-specific feature similarities and helps complement unknown labels with
high similarities. Finally, both known and generated labels are used to train
the multi-label recognition models. Extensive experiments on the Microsoft
COCO, Visual Genome and Pascal VOC datasets show that the proposed SST
framework obtains superior performance over current state-of-the-art
algorithms. Codes are available at https://github.com/HCPLab-SYSU/HCP-MLR-PL.
