  Several studies showed that misuses of cryptographic APIs are common in
real-world code (e.g., Apache projects and Android apps). There exist several
open-sourced and commercial security tools that automatically screen Java
programs to detect misuses. To compare their accuracy and security guarantees,
we develop two comprehensive benchmarks named CryptoAPI-Bench and
ApacheCryptoAPI-Bench. CryptoAPI-Bench consists of 181 unit test cases that
cover basic cases, as well as complex cases, including interprocedural, field
sensitive, multiple class test cases, and path sensitive data flow of misuse
cases. The benchmark also includes correct cases for testing false-positive
rates. The ApacheCryptoAPI-Bench consists of 121 cryptographic cases from 10
Apache projects. We evaluate four tools, namely, SpotBugs, CryptoGuard, CrySL,
and Coverity using both benchmarks. We present their performance and
comparative analysis. The ApacheCryptoAPI-Bench also examines the scalability
of the tools. Our benchmarks are useful for advancing state-of-the-art
solutions in the space of misuse detection.
