  Machine-learning models have been recently used for detecting malicious
Android applications, reporting impressive performances on benchmark datasets,
even when trained only on features statically extracted from the application,
such as system calls and permissions. However, recent findings have highlighted
the fragility of such in-vitro evaluations with benchmark datasets, showing
that very few changes to the content of Android malware may suffice to evade
detection. How can we thus trust that a malware detector performing well on
benchmark data will continue to do so when deployed in an operating
environment? To mitigate this issue, the most popular Android malware detectors
use linear, explainable machine-learning models to easily identify the most
influential features contributing to each decision. In this work, we generalize
this approach to any black-box machine- learning model, by leveraging a
gradient-based approach to identify the most influential local features. This
enables using nonlinear models to potentially increase accuracy without
sacrificing interpretability of decisions. Our approach also highlights the
global characteristics learned by the model to discriminate between benign and
malware applications. Finally, as shown by our empirical analysis on a popular
Android malware detection task, it also helps identifying potential
vulnerabilities of linear and nonlinear models against adversarial
manipulations.
