  The recently introduced TabPFN pretrains an In-Context Learning (ICL)
transformer on synthetic data to perform tabular data classification. As
synthetic data does not share features or labels with real-world data, the
underlying mechanism that contributes to the success of this method remains
unclear. This study provides an explanation by demonstrating that
ICL-transformers acquire the ability to create complex decision boundaries
during pretraining. To validate our claim, we develop a novel forest dataset
generator which creates datasets that are unrealistic, but have complex
decision boundaries. Our experiments confirm the effectiveness of
ICL-transformers pretrained on this data. Furthermore, we create TabForestPFN,
the ICL-transformer pretrained on both the original TabPFN synthetic dataset
generator and our forest dataset generator. By fine-tuning this model, we reach
the current state-of-the-art on tabular data classification. Code is available
at https://github.com/FelixdenBreejen/TabForestPFN.
