  Human-Object Interaction (HOI) detection is a challenging computer vision
task that requires visual models to address the complex interactive
relationship between humans and objects and predict HOI triplets. Despite the
challenges posed by the numerous interaction combinations, they also offer
opportunities for multimodal learning of visual texts. In this paper, we
present a systematic and unified framework (RmLR) that enhances HOI detection
by incorporating structured text knowledge. Firstly, we qualitatively and
quantitatively analyze the loss of interaction information in the two-stage HOI
detector and propose a re-mining strategy to generate more comprehensive visual
representation.Secondly, we design more fine-grained sentence- and word-level
alignment and knowledge transfer strategies to effectively address the
many-to-many matching problem between multiple interactions and multiple
texts.These strategies alleviate the matching confusion problem that arises
when multiple interactions occur simultaneously, thereby improving the
effectiveness of the alignment process. Finally, HOI reasoning by visual
features augmented with textual knowledge substantially improves the
understanding of interactions. Experimental results illustrate the
effectiveness of our approach, where state-of-the-art performance is achieved
on public benchmarks. We further analyze the effects of different components of
our approach to provide insights into its efficacy.
