  Tabular data is the foundation of the information age and has been
extensively studied. Recent studies show that neural-based models are effective
in learning contextual representation for tabular data. The learning of an
effective contextual representation requires meaningful features and a large
amount of data. However, current methods often fail to properly learn a
contextual representation from the features without semantic information. In
addition, it's intractable to enlarge the training set through mixed tabular
datasets due to the difference between datasets. To address these problems, we
propose a novel framework PTab, using the Pre-trained language model to model
Tabular data. PTab learns a contextual representation of tabular data through a
three-stage processing: Modality Transformation(MT), Masked-Language
Fine-tuning(MF), and Classification Fine-tuning(CF). We initialize our model
with a pre-trained Model (PTM) which contains semantic information learned from
the large-scale language data. Consequently, contextual representation can be
learned effectively during the fine-tuning stages. In addition, we can
naturally mix the textualized tabular data to enlarge the training set to
further improve representation learning. We evaluate PTab on eight popular
tabular classification datasets. Experimental results show that our method has
achieved a better average AUC score in supervised settings compared to the
state-of-the-art baselines(e.g. XGBoost), and outperforms counterpart methods
under semi-supervised settings. We present visualization results that show PTab
has well instance-based interpretability.
