  Tabular data pervades the landscape of the World Wide Web, playing a
foundational role in the digital architecture that underpins online
information. Given the recent influence of large-scale pretrained models like
ChatGPT and SAM across various domains, exploring the application of
pretraining techniques for mining tabular data on the web has emerged as a
highly promising research direction. Indeed, there have been some recent works
around this topic where most (if not all) of them are limited in the scope of a
fixed-schema/single table. Due to the scale of the dataset and the parameter
size of the prior models, we believe that we have not reached the ''BERT
moment'' for the ubiquitous tabular data. The development on this line
significantly lags behind the counterpart research domains such as natural
language processing. In this work, we first identify the crucial challenges
behind tabular data pretraining, particularly overcoming the cross-table
hurdle. As a pioneering endeavor, this work mainly (i)-contributes a
high-quality real-world tabular dataset, (ii)-proposes an innovative, generic,
and efficient cross-table pretraining framework, dubbed as CM2, where the core
to it comprises a semantic-aware tabular neural network that uniformly encodes
heterogeneous tables without much restriction and (iii)-introduces a novel
pretraining objective -- prompt Masked Table Modeling (pMTM) -- inspired by NLP
but intricately tailored to scalable pretraining on tables. Our extensive
experiments demonstrate CM2's state-of-the-art performance and validate that
cross-table pretraining can enhance various downstream tasks.
