  Scene graph generation (SGG) aims to detect objects and predict their
pairwise relationships within an image. Current SGG methods typically utilize
graph neural networks (GNNs) to acquire context information between
objects/relationships. Despite their effectiveness, however, current SGG
methods only assume scene graph homophily while ignoring heterophily.
Accordingly, in this paper, we propose a novel Heterophily Learning Network
(HL-Net) to comprehensively explore the homophily and heterophily between
objects/relationships in scene graphs. More specifically, HL-Net comprises the
following 1) an adaptive reweighting transformer module, which adaptively
integrates the information from different layers to exploit both the
heterophily and homophily in objects; 2) a relationship feature propagation
module that efficiently explores the connections between relationships by
considering heterophily in order to refine the relationship representation; 3)
a heterophily-aware message-passing scheme to further distinguish the
heterophily and homophily between objects/relationships, thereby facilitating
improved message passing in graphs. We conducted extensive experiments on two
public datasets: Visual Genome (VG) and Open Images (OI). The experimental
results demonstrate the superiority of our proposed HL-Net over existing
state-of-the-art approaches. In more detail, HL-Net outperforms the second-best
competitors by 2.1$\%$ on the VG dataset for scene graph classification and
1.2$\%$ on the IO dataset for the final score. Code is available at
https://github.com/siml3/HL-Net.
