  Tabular data from different tables exhibit significant diversity due to
varied definitions and types of features, as well as complex inter-feature and
feature-target relationships. Cross-dataset pretraining, which learns reusable
patterns from upstream data to support downstream tasks, have shown notable
success in various fields. Yet, when applied to tabular data prediction, this
paradigm faces challenges due to the limited reusable patterns among diverse
tabular datasets (tables) and the general scarcity of tabular data available
for fine-tuning. In this study, we fill this gap by introducing a cross-table
pretrained Transformer, XTFormer, for versatile downstream tabular prediction
tasks. Our methodology insight is pretraining XTFormer to establish a
"meta-function" space that encompasses all potential feature-target mappings.
In pre-training, a variety of potential mappings are extracted from
pre-training tabular datasets and are embedded into the "meta-function" space,
and suited mappings are extracted from the "meta-function" space for downstream
tasks by a specified coordinate positioning approach. Experiments show that, in
190 downstream tabular prediction tasks, our cross-table pretrained XTFormer
wins both XGBoost and Catboost on 137 (72%) tasks, and surpasses representative
deep learning models FT-Transformer and the tabular pre-training approach XTab
on 144 (76%) and 162 (85%) tasks.
