  Scene graph generation aims to produce structured representations for images,
which requires to understand the relations between objects. Due to the
continuous nature of deep neural networks, the prediction of scene graphs is
divided into object detection and relation classification. However, the
independent relation classes cannot separate the visual features well. Although
some methods organize the visual features into graph structures and use message
passing to learn contextual information, they still suffer from drastic
intra-class variations and unbalanced data distributions. One important factor
is that they learn an unstructured output space that ignores the inherent
structures of scene graphs. Accordingly, in this paper, we propose a Higher
Order Structure Embedded Network (HOSE-Net) to mitigate this issue. First, we
propose a novel structure-aware embedding-to-classifier(SEC) module to
incorporate both local and global structural information of relationships into
the output space. Specifically, a set of context embeddings are learned via
local graph based message passing and then mapped to a global structure based
classification space. Second, since learning too many context-specific
classification subspaces can suffer from data sparsity issues, we propose a
hierarchical semantic aggregation(HSA) module to reduces the number of
subspaces by introducing higher order structural information. HSA is also a
fast and flexible tool to automatically search a semantic object hierarchy
based on relational knowledge graphs. Extensive experiments show that the
proposed HOSE-Net achieves the state-of-the-art performance on two popular
benchmarks of Visual Genome and VRD.
