  With the growing adoption of deep learning models in different real-world
domains, including computational biology, it is often necessary to understand
which data features are essential for the model's decision. Despite extensive
recent efforts to define different feature importance metrics for deep learning
models, we identified that inherent stochasticity in the design and training of
deep learning models makes commonly used feature importance scores unstable.
This results in varied explanations or selections of different features across
different runs of the model. We demonstrate how the signal strength of features
and correlation among features directly contribute to this instability. To
address this instability, we explore the ensembling of feature importance
scores of models across different epochs and find that this simple approach can
substantially address this issue. For example, we consider knockoff inference
as they allow feature selection with statistical guarantees. We discover
considerable variability in selected features in different epochs of deep
learning training, and the best selection of features doesn't necessarily occur
at the lowest validation loss, the conventional approach to determine the best
model. As such, we present a framework to combine the feature importance of
trained models across different hyperparameter settings and epochs, and instead
of selecting features from one best model, we perform an ensemble of feature
importance scores from numerous good models. Across the range of experiments in
simulated and various real-world datasets, we demonstrate that the proposed
framework consistently improves the power of feature selection.
