  Despite groundbreaking success in image and text learning, deep learning has
not achieved significant improvements against traditional machine learning (ML)
when it comes to tabular data. This performance gap underscores the need for
data-centric treatment and benchmarking of learning algorithms. Recently,
attention and contrastive learning breakthroughs have shifted computer vision
and natural language processing paradigms. However, the effectiveness of these
advanced deep models on tabular data is sparsely studied using a few data sets
with very large sample sizes, reporting mixed findings after benchmarking
against a limited number of baselines. We argue that the heterogeneity of
tabular data sets and selective baselines in the literature can bias the
benchmarking outcomes. This article extensively evaluates state-of-the-art
attention and contrastive learning methods on a wide selection of 28 tabular
data sets (14 easy and 14 hard-to-classify) against traditional deep and
machine learning. Our data-centric benchmarking demonstrates when traditional
ML is preferred over deep learning and vice versa because no best learning
method exists for all tabular data sets. Combining between-sample and
between-feature attentions conquers the invincible traditional ML on tabular
data sets by a significant margin but fails on high dimensional data, where
contrastive learning takes a robust lead. While a hybrid attention-contrastive
learning strategy mostly wins on hard-to-classify data sets, traditional
methods are frequently superior on easy-to-classify data sets with presumably
simpler decision boundaries. To the best of our knowledge, this is the first
benchmarking paper with statistical analyses of attention and contrastive
learning performances on a diverse selection of tabular data sets against
traditional deep and machine learning baselines to facilitate further advances
in this field.
