  In recent years, the importance of smart contract security has been
heightened by the increasing number of attacks against them. To address this
issue, a multitude of static application security testing (SAST) tools have
been proposed for detecting vulnerabilities in smart contracts. However,
objectively comparing these tools to determine their effectiveness remains
challenging. Existing studies often fall short due to the taxonomies and
benchmarks only covering a coarse and potentially outdated set of vulnerability
types, which leads to evaluations that are not entirely comprehensive and may
display bias.
  In this paper, we fill this gap by proposing an up-to-date and fine-grained
taxonomy that includes 45 unique vulnerability types for smart contracts.
Taking it as a baseline, we develop an extensive benchmark that covers 40
distinct types and includes a diverse range of code characteristics,
vulnerability patterns, and application scenarios. Based on them, we evaluated
8 SAST tools using this benchmark, which comprises 788 smart contract files and
10,394 vulnerabilities. Our results reveal that the existing SAST tools fail to
detect around 50% of vulnerabilities in our benchmark and suffer from high
false positives, with precision not surpassing 10%. We also discover that by
combining the results of multiple tools, the false negative rate can be reduced
effectively, at the expense of flagging 36.77 percentage points more functions.
Nevertheless, many vulnerabilities, especially those beyond Access Control and
Reentrancy vulnerabilities, remain undetected. We finally highlight the
valuable insights from our study, hoping to provide guidance on tool
development, enhancement, evaluation, and selection for developers,
researchers, and practitioners.
