  Discrete reasoning over table-text documents (e.g., financial reports) gains
increasing attention in recent two years. Existing works mostly simplify this
challenge by manually selecting and transforming document pages to structured
tables and paragraphs, hindering their practical application. In this work, we
explore a more realistic problem setting in the form of TAT-DQA, i.e. to answer
the question over a visually-rich table-text document. Specifically, we propose
a novel Doc2SoarGraph framework with enhanced discrete reasoning capability by
harnessing the differences and correlations among different elements (e.g.,
quantities, dates) of the given question and document with Semantic-oriented
hierarchical Graph structures. We conduct extensive experiments on TAT-DQA
dataset, and the results show that our proposed framework outperforms the best
baseline model by 17.73% and 16.91% in terms of Exact Match (EM) and F1 score
respectively on the test set, achieving the new state-of-the-art.
