  Documents are often used for knowledge sharing and preservation in business
and science, within which are tables that capture most of the critical data.
Unfortunately, most documents are stored and distributed as PDF or scanned
images, which fail to preserve logical table structure. Recent vision-based
deep learning approaches have been proposed to address this gap, but most still
cannot achieve state-of-the-art results. We present Global Table Extractor
(GTE), a vision-guided systematic framework for joint table detection and cell
structured recognition, which could be built on top of any object detection
model. With GTE-Table, we invent a new penalty based on the natural cell
containment constraint of tables to train our table network aided by cell
location predictions. GTE-Cell is a new hierarchical cell detection network
that leverages table styles. Further, we design a method to automatically label
table and cell structure in existing documents to cheaply create a large corpus
of training and test data. We use this to enhance PubTabNet with cell labels
and create FinTabNet, real-world and complex scientific and financial datasets
with detailed table structure annotations to help train and test structure
recognition. Our framework surpasses previous state-of-the-art results on the
ICDAR 2013 and ICDAR 2019 table competition in both table detection and cell
structure recognition with a significant 5.8% improvement in the full table
extraction system. Further experiments demonstrate a greater than 45%
improvement in cell structure recognition when compared to a vanilla RetinaNet
object detection model in our new out-of-domain FinTabNet.
