  In recent years, researchers have developed a number of tools to conduct
taint analysis of Android applications. While all the respective papers aim at
providing a thorough empirical evaluation, comparability is hindered by varying
or unclear evaluation targets. Sometimes, the apps used for evaluation are not
precisely described. In other cases, authors use an established benchmark but
cover it only partially. In yet other cases, the evaluations differ in terms of
the data leaks searched for, or lack a ground truth to compare against. All
those limitations make it impossible to truly compare the tools based on those
published evaluations.
  We thus present ReproDroid, a framework allowing the accurate comparison of
Android taint analysis tools. ReproDroid supports researchers in inferring the
ground truth for data leaks in apps, in automatically applying tools to
benchmarks, and in evaluating the obtained results. We use ReproDroid to
comparatively evaluate on equal grounds the six prominent taint analysis tools
Amandroid, DIALDroid, DidFail, DroidSafe, FlowDroid and IccTA. The results are
largely positive although four tools violate some promises concerning features
and accuracy. Finally, we contribute to the area of unbiased benchmarking with
a new and improved version of the open test suite DroidBench.
