  Recent benchmarks found In-Context Learning (ICL) outperforms both deep
learning and tree-based algorithms on small tabular datasets. However, on
larger datasets, ICL for tabular learning cannot run without severely
compromising performance, due to its quadratic space and time complexity w.r.t.
dataset size. We propose MIXTUREPFN, which both extends nearest-neighbor
sampling to the state-of-the-art ICL for tabular learning model and uses
bootstrapping to finetune said model on the inference-time dataset. MIXTUREPFN
is the Condorcet winner across 36 diverse tabular datasets against 19 strong
deep learning and tree-based baselines, achieving the highest mean rank among
Top-10 aforementioned algorithms with statistical significance.
