  Abstract--- Exploiting the spatial structure in scene images is a key
research direction for scene recognition. Due to the large intra-class
structural diversity, building and modeling flexible structural layout to adapt
various image characteristics is a challenge. Existing structural modeling
methods in scene recognition either focus on predefined grids or rely on
learned prototypes, which all have limited representative ability. In this
paper, we propose Prototype-agnostic Scene Layout (PaSL) construction method to
build the spatial structure for each image without conforming to any prototype.
Our PaSL can flexibly capture the diverse spatial characteristic of scene
images and have considerable generalization capability. Given a PaSL, we build
Layout Graph Network (LGN) where regions in PaSL are defined as nodes and two
kinds of independent relations between regions are encoded as edges. The LGN
aims to incorporate two topological structures (formed in spatial and semantic
similarity dimensions) into image representations through graph convolution.
Extensive experiments show that our approach achieves state-of-the-art results
on widely recognized MIT67 and SUN397 datasets without multi-model or
multi-scale fusion. Moreover, we also conduct the experiments on one of the
largest scale datasets, Places365. The results demonstrate the proposed method
can be well generalized and obtains competitive performance.
