  Black-box neural network models are widely used in industry and science, yet
are hard to understand and interpret. Recently, the attention mechanism was
introduced, offering insights into the inner workings of neural language
models. This paper explores the use of attention-based neural networks
mechanism for estimating feature importance, as means for explaining the models
learned from propositional (tabular) data. Feature importance estimates,
assessed by the proposed Self-Attention Network (SAN) architecture, are
compared with the established ReliefF, Mutual Information and Random
Forest-based estimates, which are widely used in practice for model
interpretation. For the first time we conduct scale-free comparisons of feature
importance estimates across algorithms on ten real and synthetic data sets to
study the similarities and differences of the resulting feature importance
estimates, showing that SANs identify similar high-ranked features as the other
methods. We demonstrate that SANs identify feature interactions which in some
cases yield better predictive performance than the baselines, suggesting that
attention extends beyond interactions of just a few key features and detects
larger feature subsets relevant for the considered learning task.
