  Recent advancements in NLP have witnessed the groundbreaking impact of
pretrained models, yielding impressive outcomes across various tasks. This
study seeks to extend the power of pretraining methodologies to facilitating
the prediction over tables in data science, a domain traditionally overlooked,
yet inherently challenging due to the plethora of table schemas intrinsic to
different tasks. The primary research questions underpinning this work revolve
around the establishment of a universal pretraining protocol for tables with
varied structures, the generalizability and transferability of learned
knowledge across tasks, the adaptation to diverse downstream applications, and
the incorporation of incremental columns over time. In response to these
challenges, we introduce UniTabE, a straightforward yet effective method
designed to process tables in a uniform manner, devoid of constraints imposed
by specific table structures. UniTabE's core concept relies on representing
each basic table element with a module, termed TabUnit. This is subsequently
followed by a Transformer encoder to refine the representation. Moreover, our
model is designed to facilitate pretraining and finetuning through the
utilization of free-form prompts. In order to implement the pretraining phase,
we curated an expansive tabular dataset comprising approximately 13B samples,
meticulously gathered from the Kaggle platform. This research primarily centers
on classification and regression tasks involving tabular data, and conducts
rigorous experimental testing and analyses to validate the effectiveness of our
methodology. The experimental results demonstrate UniTabE's superior
performance against several baselines across massive benchmarks. This,
therefore, underscores UniTabE's potential to significantly enhance the
semantic representation of tabular data, thereby marking a significant stride
for tabular data analysis.
