  Until recently, the question of the effective inductive bias of deep models
on tabular data has remained unanswered. This paper investigates the hypothesis
that arithmetic feature interaction is necessary for deep tabular learning. To
test this point, we create a synthetic tabular dataset with a mild feature
interaction assumption and examine a modified transformer architecture enabling
arithmetical feature interactions, referred to as AMFormer. Results show that
AMFormer outperforms strong counterparts in fine-grained tabular data modeling,
data efficiency in training, and generalization. This is attributed to its
parallel additive and multiplicative attention operators and prompt-based
optimization, which facilitate the separation of tabular samples in an extended
space with arithmetically-engineered features. Our extensive experiments on
real-world data also validate the consistent effectiveness, efficiency, and
rationale of AMFormer, suggesting it has established a strong inductive bias
for deep learning on tabular data. Code is available at
https://github.com/aigc-apps/AMFormer.
