  Tabular data is foundational to predictive modeling in various crucial
industries, including healthcare, finance, retail, sustainability, etc. Despite
the progress made in specialized models, there is an increasing demand for
universal models that can transfer knowledge, generalize from limited data, and
follow human instructions. These are challenges that current tabular deep
learning approaches have not fully tackled. Here we introduce Generative
Tabular Learning (GTL), a novel framework that integrates the advanced
functionalities of large language models (LLMs)-such as prompt-based zero-shot
generalization and in-context learning-into tabular deep learning. GTL
capitalizes on the pre-training of LLMs on diverse tabular data, enhancing
their understanding of domain-specific knowledge, numerical sequences, and
statistical dependencies critical for accurate predictions. Our empirical study
spans 384 public datasets, rigorously analyzing GTL's convergence and scaling
behaviors and assessing the impact of varied data templates. The GTL-enhanced
LLaMA-2 model demonstrates superior zero-shot and in-context learning
capabilities across numerous classification and regression tasks. Notably, it
achieves this without fine-tuning, outperforming traditional methods and
rivaling state-of-the-art models like GPT-4 in certain cases. Through GTL, we
not only foster a deeper integration of LLMs' sophisticated abilities into
tabular data comprehension and application but also offer a new training
resource and a test bed for LLMs to enhance their ability to comprehend tabular
data. To facilitate reproducible research, we release our code, data, and model
checkpoints at https://github.com/microsoft/Industrial-Foundation-Models.
