  This paper presents a novel attention-based neural network for structured
reconstruction, which takes a 2D raster image as an input and reconstructs a
planar graph depicting an underlying geometric structure. The approach detects
corners and classifies edge candidates between corners in an end-to-end manner.
Our contribution is a holistic edge classification architecture, which 1)
initializes the feature of an edge candidate by a trigonometric positional
encoding of its end-points; 2) fuses image feature to each edge candidate by
deformable attention; 3) employs two weight-sharing Transformer decoders to
learn holistic structural patterns over the graph edge candidates; and 4) is
trained with a masked learning strategy. The corner detector is a variant of
the edge classification architecture, adapted to operate on pixels as corner
candidates. We conduct experiments on two structured reconstruction tasks:
outdoor building architecture and indoor floorplan planar graph reconstruction.
Extensive qualitative and quantitative evaluations demonstrate the superiority
of our approach over the state of the art. Code and pre-trained models are
available at https://heat-structured-reconstruction.github.io.
