  Human Object Interaction (HOI) detection aims to localize and infer the
relationships between a human and an object. Arguably, training supervised
models for this task from scratch presents challenges due to the performance
drop over rare classes and the high computational cost and time required to
handle long-tailed distributions of HOIs in complex HOI scenes in realistic
settings. This observation motivates us to design an HOI detector that can be
trained even with long-tailed labeled data and can leverage existing knowledge
from pre-trained models. Inspired by the powerful generalization ability of the
large Vision-Language Models (VLM) on classification and retrieval tasks, we
propose an efficient Adaptive HOI Detector with Concept-guided Memory (ADA-CM).
ADA-CM has two operating modes. The first mode makes it tunable without
learning new parameters in a training-free paradigm. Its second mode
incorporates an instance-aware adapter mechanism that can further efficiently
boost performance if updating a lightweight set of parameters can be afforded.
Our proposed method achieves competitive results with state-of-the-art on the
HICO-DET and V-COCO datasets with much less training time. Code can be found at
https://github.com/ltttpku/ADA-CM.
