  Quality assurance (QA) tools are receiving more and more attention and are
widely used by developers. Given the wide range of solutions for QA technology,
it is still a question of evaluating QA tools. Most existing research is
limited in the following ways: (i) They compare tools without considering
scanning rules analysis. (ii) They disagree on the effectiveness of tools due
to the study methodology and benchmark dataset. (iii) They do not separately
analyze the role of the warnings. (iv) There is no large-scale study on the
analysis of time performance. To address these problems, in the paper, we
systematically select 6 free or open-source tools for a comprehensive study
from a list of 148 existing Java QA tools. To carry out a comprehensive study
and evaluate tools in multi-level dimensions, we first mapped the scanning
rules to the CWE and analyze the coverage and granularity of the scanning
rules. Then we conducted an experiment on 5 benchmarks, including 1,425 bugs,
to investigate the effectiveness of these tools. Furthermore, we took
substantial effort to investigate the effectiveness of warnings by comparing
the real labeled bugs with the warnings and investigating their role in bug
detection. Finally, we assessed these tools' time performance on 1,049
projects. The useful findings based on our comprehensive study can help
developers improve their tools and provide users with suggestions for selecting
QA tools.
