  A comprehensive understanding of human-object interaction (HOI) requires
detecting not only a small portion of predefined HOI concepts (or categories)
but also other reasonable HOI concepts, while current approaches usually fail
to explore a huge portion of unknown HOI concepts (i.e., unknown but reasonable
combinations of verbs and objects). In this paper, 1) we introduce a novel and
challenging task for a comprehensive HOI understanding, which is termed as HOI
Concept Discovery; and 2) we devise a self-compositional learning framework (or
SCL) for HOI concept discovery. Specifically, we maintain an online updated
concept confidence matrix during training: 1) we assign pseudo-labels for all
composite HOI instances according to the concept confidence matrix for
self-training; and 2) we update the concept confidence matrix using the
predictions of all composite HOI instances. Therefore, the proposed method
enables the learning on both known and unknown HOI concepts. We perform
extensive experiments on several popular HOI datasets to demonstrate the
effectiveness of the proposed method for HOI concept discovery, object
affordance recognition and HOI detection. For example, the proposed
self-compositional learning framework significantly improves the performance of
1) HOI concept discovery by over 10% on HICO-DET and over 3% on V-COCO,
respectively; 2) object affordance recognition by over 9% mAP on MS-COCO and
HICO-DET; and 3) rare-first and non-rare-first unknown HOI detection relatively
over 30% and 20%, respectively. Code is publicly available at
https://github.com/zhihou7/HOI-CL.
