  Panoramic image enables deeper understanding and more holistic perception of
$360^\circ$ surrounding environment, which can naturally encode enriched scene
context information compared to standard perspective image. Previous work has
made lots of effort to solve the scene understanding task in a bottom-up form,
thus each sub-task is processed separately and few correlations are explored in
this procedure. In this paper, we propose a novel method using depth prior for
holistic indoor scene understanding which recovers the objects' shapes,
oriented bounding boxes and the 3D room layout simultaneously from a single
panorama. In order to fully utilize the rich context information, we design a
transformer-based context module to predict the representation and relationship
among each component of the scene. In addition, we introduce a real-world
dataset for scene understanding, including photo-realistic panoramas,
high-fidelity depth images, accurately annotated room layouts, and oriented
object bounding boxes and shapes. Experiments on the synthetic and real-world
datasets demonstrate that our method outperforms previous panoramic scene
understanding methods in terms of both layout estimation and 3D object
detection.
