  For classification and regression on tabular data, the dominance of
gradient-boosted decision trees (GBDTs) has recently been challenged by often
much slower deep learning methods with extensive hyperparameter tuning. We
address this discrepancy by introducing (a) RealMLP, an improved multilayer
perceptron (MLP), and (b) improved default parameters for GBDTs and RealMLP. We
tune RealMLP and the default parameters on a meta-train benchmark with 71
classification and 47 regression datasets and compare them to
hyperparameter-optimized versions on a disjoint meta-test benchmark with 48
classification and 42 regression datasets, as well as the GBDT-friendly
benchmark by Grinsztajn et al. (2022). Our benchmark results show that RealMLP
offers a better time-accuracy tradeoff than other neural nets and is
competitive with GBDTs. Moreover, a combination of RealMLP and GBDTs with
improved default parameters can achieve excellent results on medium-sized
tabular datasets (1K--500K samples) without hyperparameter tuning.
