  We present a new formulation for structured information extraction (SIE) from
visually rich documents. It aims to address the limitations of existing IOB
tagging or graph-based formulations, which are either overly reliant on the
correct ordering of input text or struggle with decoding a complex graph.
Instead, motivated by anchor-based object detectors in vision, we represent an
entity as an anchor word and a bounding box, and represent entity linking as
the association between anchor words. This is more robust to text ordering, and
maintains a compact graph for entity linking. The formulation motivates us to
introduce 1) a DOCument TRansformer (DocTr) that aims at detecting and
associating entity bounding boxes in visually rich documents, and 2) a simple
pre-training strategy that helps learn entity detection in the context of
language. Evaluations on three SIE benchmarks show the effectiveness of the
proposed formulation, and the overall approach outperforms existing solutions.
