  Increasing interest in securing the Android ecosystem has spawned numerous
efforts to assist app developers in building secure apps. These efforts have
resulted in tools and techniques capable of detecting vulnerabilities (and
malicious behaviors) in apps. However, there has been no evaluation of the
effectiveness of these tools and techniques in detecting known vulnerabilities.
The absence of such evaluations puts app developers at a disadvantage when
choosing security analysis tools to secure their apps. In this regard, we
evaluated the effectiveness of vulnerability detection tools for Android apps.
We reviewed 64 tools and empirically evaluated 14 vulnerability detection tools
(incidentally along with five malicious behavior detection tools) against 42
known unique vulnerabilities captured by Ghera benchmarks, which are composed
of both vulnerable and secure apps. Of the 24 observations from the evaluation,
the main observation is existing vulnerability detection tools for Android apps
are very limited in their ability to detect known vulnerabilities -- all of the
evaluated tools together could only detect 30 of the 42 known unique
vulnerabilities. More effort is required if security analysis tools are to help
developers build secure apps. We hope the observations from this evaluation
will help app developers choose appropriate security analysis tools and
persuade tool developers and researchers to identify and address limitations in
their tools and techniques. We also hope this evaluation will catalyze or spark
a conversation in the software engineering and security communities to require
a more rigorous and explicit evaluation of security analysis tools and
techniques.
