  Generating scene graph to describe all the relations inside an image gains
increasing interests these years. However, most of the previous methods use
complicated structures with slow inference speed or rely on the external data,
which limits the usage of the model in real-life scenarios. To improve the
efficiency of scene graph generation, we propose a subgraph-based connection
graph to concisely represent the scene graph during the inference. A bottom-up
clustering method is first used to factorize the entire scene graph into
subgraphs, where each subgraph contains several objects and a subset of their
relationships. By replacing the numerous relationship representations of the
scene graph with fewer subgraph and object features, the computation in the
intermediate stage is significantly reduced. In addition, spatial information
is maintained by the subgraph features, which is leveraged by our proposed
Spatial-weighted Message Passing~(SMP) structure and Spatial-sensitive Relation
Inference~(SRI) module to facilitate the relationship recognition. On the
recent Visual Relationship Detection and Visual Genome datasets, our method
outperforms the state-of-the-art method in both accuracy and speed.
