  Human-Object Interaction (HOI) Detection is an important problem to
understand how humans interact with objects. In this paper, we explore
Interactiveness Knowledge which indicates whether human and object interact
with each other or not. We found that interactiveness knowledge can be learned
across HOI datasets, regardless of HOI category settings. Our core idea is to
exploit an Interactiveness Network to learn the general interactiveness
knowledge from multiple HOI datasets and perform Non-Interaction Suppression
before HOI classification in inference. On account of the generalization of
interactiveness, interactiveness network is a transferable knowledge learner
and can be cooperated with any HOI detection models to achieve desirable
results. We extensively evaluate the proposed method on HICO-DET and V-COCO
datasets. Our framework outperforms state-of-the-art HOI detection results by a
great margin, verifying its efficacy and flexibility. Code is available at
https://github.com/DirtyHarryLYL/Transferable-Interactiveness-Network.
