  Tabular biomedical data is often high-dimensional but with a very small
number of samples. Although recent work showed that well-regularised simple
neural networks could outperform more sophisticated architectures on tabular
data, they are still prone to overfitting on tiny datasets with many
potentially irrelevant features. To combat these issues, we propose Weight
Predictor Network with Feature Selection (WPFS) for learning neural networks
from high-dimensional and small sample data by reducing the number of learnable
parameters and simultaneously performing feature selection. In addition to the
classification network, WPFS uses two small auxiliary networks that together
output the weights of the first layer of the classification model. We evaluate
on nine real-world biomedical datasets and demonstrate that WPFS outperforms
other standard as well as more recent methods typically applied to tabular
data. Furthermore, we investigate the proposed feature selection mechanism and
show that it improves performance while providing useful insights into the
learning task.
