  Ethereum smart contracts, which are autonomous decentralized applications on
the blockchain that manage assets often exceeding millions of dollars, have
become primary targets for cyberattacks. In 2023 alone, such vulnerabilities
led to substantial financial losses exceeding a billion of US dollars. To
counter these threats, various tools have been developed by academic and
commercial entities to detect and mitigate vulnerabilities in smart contracts.
Our study investigates the gap between the effectiveness of existing security
scanners and the vulnerabilities that still persist in practice. We compiled
four distinct datasets for this analysis. The first dataset comprises 77,219
source codes extracted directly from the blockchain, while the second includes
over 4 million bytecodes obtained from Ethereum Mainnet and testnets. The other
two datasets consist of nearly 14,000 manually annotated smart contracts and
373 smart contracts verified through audits, providing a foundation for a
rigorous ground truth analysis on bytecode and source code. Using the unlabeled
datasets, we conducted a comprehensive quantitative evaluation of 17
vulnerability scanners, revealing considerable discrepancies in their findings.
Our analysis of the ground truth datasets indicated poor performance across all
the tools we tested. This study unveils the reasons for poor performance and
underscores that the current state of the art for smart contract security falls
short in effectively addressing open problems, highlighting that the challenge
of effectively detecting vulnerabilities remains a significant and unresolved
issue.
