  Neural networks have seen limited use in prediction for high-dimensional data
with small sample sizes, because they tend to overfit and require tuning many
more hyperparameters than existing off-the-shelf machine learning methods. With
small modifications to the network architecture and training procedure, we show
that dense neural networks can be a practical data analysis tool in these
settings. The proposed method, Ensemble by Averaging Sparse-Input Hierarchical
networks (EASIER-net), appropriately prunes the network structure by tuning
only two L1-penalty parameters, one that controls the input sparsity and
another that controls the number of hidden layers and nodes. The method selects
variables from the true support if the irrelevant covariates are only weakly
correlated with the response; otherwise, it exhibits a grouping effect, where
strongly correlated covariates are selected at similar rates. On a collection
of real-world datasets with different sizes, EASIER-net selected network
architectures in a data-adaptive manner and achieved higher prediction accuracy
than off-the-shelf methods on average.
