  The task of Human-Object Interaction (HOI) detection targets fine-grained
visual parsing of humans interacting with their environment, enabling a broad
range of applications. Prior work has demonstrated the benefits of effective
architecture design and integration of relevant cues for more accurate HOI
detection. However, the design of an appropriate pre-training strategy for this
task remains underexplored by existing approaches. To address this gap, we
propose Relational Language-Image Pre-training (RLIP), a strategy for
contrastive pre-training that leverages both entity and relation descriptions.
To make effective use of such pre-training, we make three technical
contributions: (1) a new Parallel entity detection and Sequential relation
inference (ParSe) architecture that enables the use of both entity and relation
descriptions during holistically optimized pre-training; (2) a synthetic data
generation framework, Label Sequence Extension, that expands the scale of
language data available within each minibatch; (3) mechanisms to account for
ambiguity, Relation Quality Labels and Relation Pseudo-Labels, to mitigate the
influence of ambiguous/noisy samples in the pre-training data. Through
extensive experiments, we demonstrate the benefits of these contributions,
collectively termed RLIP-ParSe, for improved zero-shot, few-shot and
fine-tuning HOI detection performance as well as increased robustness to
learning from noisy annotations. Code will be available at
https://github.com/JacobYuan7/RLIP.
