  Despite the prevalence and significance of tabular data across numerous
industries and fields, it has been relatively underexplored in the realm of
deep learning. Even today, neural networks are often overshadowed by techniques
such as gradient boosted decision trees (GBDT). However, recent models are
beginning to close this gap, outperforming GBDT in various setups and garnering
increased attention in the field. Inspired by this development, we introduce a
novel stochastic deep learning model specifically designed for tabular data.
The foundation of this model is a Transformer-based architecture, carefully
adapted to cater to the unique properties of tabular data through strategic
architectural modifications and leveraging two forms of stochastic competition.
First, we employ stochastic "Local Winner Takes All" units to promote
generalization capacity through stochasticity and sparsity. Second, we
introduce a novel embedding layer that selects among alternative linear
embedding layers through a mechanism of stochastic competition. The
effectiveness of the model is validated on a variety of widely-used, publicly
available datasets. We demonstrate that, through the incorporation of these
elements, our model yields high performance and marks a significant advancement
in the application of deep learning to tabular data.
