  Background: Modern software uses many third-party libraries and frameworks as
dependencies. Known vulnerabilities in these dependencies are a potential
security risk. Software composition analysis (SCA) tools, therefore, are being
increasingly adopted by practitioners to keep track of vulnerable dependencies.
Aim: The goal of this study is to understand the difference in vulnerability
reporting by various SCA tools. Understanding if and how existing SCA tools
differ in their analysis may help security practitioners to choose the right
tooling and identify future research needs. Method: We present an in-depth case
study by comparing the analysis reports of 9 industry-leading SCA tools on a
large web application, OpenMRS, composed of Maven (Java) and npm (JavaScript)
projects. Results: We find that the tools vary in their vulnerability
reporting. The count of reported vulnerable dependencies ranges from 17 to 332
for Maven and from 32 to 239 for npm projects across the studied tools.
Similarly, the count of unique known vulnerabilities reported by the tools
ranges from 36 to 313 for Maven and from 45 to 234 for npm projects. Our manual
analysis of the tools' results suggest that accuracy of the vulnerability
database is a key differentiator for SCA tools. Conclusion: We recommend that
practitioners should not rely on any single tool at the present, as that can
result in missing known vulnerabilities. We point out two research directions
in the SCA space: i) establishing frameworks and metrics to identify false
positives for dependency vulnerabilities; and ii) building automation
technologies for continuous monitoring of vulnerability data from open source
package ecosystems.
