  While tabular classification has traditionally relied on from-scratch
training, a recent breakthrough called prior-data fitted networks (PFNs)
challenges this approach. Similar to large language models, PFNs make use of
pretraining and in-context learning to achieve strong performance on new tasks
in a single forward pass. However, current PFNs have limitations that prohibit
their widespread adoption. Notably, TabPFN achieves very strong performance on
small tabular datasets but is not designed to make predictions for datasets of
size larger than 1000. In this work, we overcome these limitations and
substantially improve the performance of PFNs by developing context
optimization techniques for PFNs. Specifically, we propose TuneTables, a novel
prompt-tuning strategy that compresses large datasets into a smaller learned
context. TuneTables scales TabPFN to be competitive with state-of-the-art
tabular classification methods on larger datasets, while having a substantially
lower inference time than TabPFN. Furthermore, we show that TuneTables can be
used as an interpretability tool and can even be used to mitigate biases by
optimizing a fairness objective.
