  Human-Object Interaction (HOI) detection, which localizes and infers
relationships between human and objects, plays an important role in scene
understanding. Although two-stage HOI detectors have advantages of high
efficiency in training and inference, they suffer from lower performance than
one-stage methods due to the old backbone networks and the lack of
considerations for the HOI perception process of humans in the interaction
classifiers. In this paper, we propose Vision Transformer based
Pose-Conditioned Self-Loop Graph (ViPLO) to resolve these problems. First, we
propose a novel feature extraction method suitable for the Vision Transformer
backbone, called masking with overlapped area (MOA) module. The MOA module
utilizes the overlapped area between each patch and the given region in the
attention function, which addresses the quantization problem when using the
Vision Transformer backbone. In addition, we design a graph with a
pose-conditioned self-loop structure, which updates the human node encoding
with local features of human joints. This allows the classifier to focus on
specific human joints to effectively identify the type of interaction, which is
motivated by the human perception process for HOI. As a result, ViPLO achieves
the state-of-the-art results on two public benchmarks, especially obtaining a
+2.07 mAP performance gain on the HICO-DET dataset. The source codes are
available at https://github.com/Jeeseung-Park/ViPLO.
