  Tables are widely used with various structures to organize and present data.
Recent attempts on table understanding mainly focus on relational tables, yet
overlook to other common table structures. In this paper, we propose TUTA, a
unified pre-training architecture for understanding generally structured
tables. Noticing that understanding a table requires spatial, hierarchical, and
semantic information, we enhance transformers with three novel structure-aware
mechanisms. First, we devise a unified tree-based structure, called a
bi-dimensional coordinate tree, to describe both the spatial and hierarchical
information of generally structured tables. Upon this, we propose tree-based
attention and position embedding to better capture the spatial and hierarchical
information. Moreover, we devise three progressive pre-training objectives to
enable representations at the token, cell, and table levels. We pre-train TUTA
on a wide range of unlabeled web and spreadsheet tables and fine-tune it on two
critical tasks in the field of table structure understanding: cell type
classification and table type classification. Experiments show that TUTA is
highly effective, achieving state-of-the-art on five widely-studied datasets.
