  A common problem in human-object interaction (HOI) detection task is that
numerous HOI classes have only a small number of labeled examples, resulting in
training sets with a long-tailed distribution. The lack of positive labels can
lead to low classification accuracy for these classes. Towards addressing this
issue, we observe that there exist natural correlations and anti-correlations
among human-object interactions. In this paper, we model the correlations as
action co-occurrence matrices and present techniques to learn these priors and
leverage them for more effective training, especially in rare classes. The
utility of our approach is demonstrated experimentally, where the performance
of our approach exceeds the state-of-the-art methods on both of the two leading
HOI detection benchmark datasets, HICO-Det and V-COCO.
