  Academic tabular benchmarks often contain small sets of curated features. In
contrast, data scientists typically collect as many features as possible into
their datasets, and even engineer new features from existing ones. To prevent
overfitting in subsequent downstream modeling, practitioners commonly use
automated feature selection methods that identify a reduced subset of
informative features. Existing benchmarks for tabular feature selection
consider classical downstream models, toy synthetic datasets, or do not
evaluate feature selectors on the basis of downstream performance. Motivated by
the increasing popularity of tabular deep learning, we construct a challenging
feature selection benchmark evaluated on downstream neural networks including
transformers, using real datasets and multiple methods for generating
extraneous features. We also propose an input-gradient-based analogue of Lasso
for neural networks that outperforms classical feature selection methods on
challenging problems such as selecting from corrupted or second-order features.
