  Deep Learning has revolutionized the field of AI and led to remarkable
achievements in applications involving image and text data. Unfortunately,
there is inconclusive evidence on the merits of neural networks for structured
tabular data. In this paper, we introduce a large-scale empirical study
comparing neural networks against gradient-boosted decision trees on tabular
data, but also transformer-based architectures against traditional multi-layer
perceptrons (MLP) with residual connections. In contrast to prior work, our
empirical findings indicate that neural networks are competitive against
decision trees. Furthermore, we assess that transformer-based architectures do
not outperform simpler variants of traditional MLP architectures on tabular
datasets. As a result, this paper helps the research and practitioner
communities make informed choices on deploying neural networks on future
tabular data applications.
