  We perform a comparative analysis of transformer-based models designed for
modeling tabular data, specifically on an industry-scale dataset. While earlier
studies demonstrated promising outcomes on smaller public or synthetic
datasets, the effectiveness did not extend to larger industry-scale datasets.
The challenges identified include handling high-dimensional data, the necessity
for efficient pre-processing of categorical and numerical features, and
addressing substantial computational requirements.
  To overcome the identified challenges, the study conducts an extensive
examination of various transformer-based models using both synthetic datasets
and the default prediction Kaggle dataset (2022) from American Express. The
paper presents crucial insights into optimal data pre-processing, compares
pre-training and direct supervised learning methods, discusses strategies for
managing categorical and numerical features, and highlights trade-offs between
computational resources and performance. Focusing on temporal financial data
modeling, the research aims to facilitate the systematic development and
deployment of transformer-based models in real-world scenarios, emphasizing
scalability.
