  The Smart Contract Weakness Classification Registry (SWC Registry) is a
widely recognized list of smart contract weaknesses specific to the Ethereum
platform. Despite the SWC Registry not being updated with new entries since
2020, the sustained development of smart contract analysis tools for detecting
SWC-listed weaknesses highlights their ongoing significance in the field.
However, evaluating these tools has proven challenging due to the absence of a
large, unbiased, real-world dataset. To address this problem, we aim to build a
large-scale SWC weakness dataset from real-world DApp projects. We recruited 22
participants and spent 44 person-months analyzing 1,199 open source audit
reports from 29 security teams. In total, we identified 9,154 weaknesses and
developed two distinct datasets, i.e., DAPPSCAN-SOURCE and DAPPSCAN-BYTECODE.
The DAPPSCAN-SOURCE dataset comprises 39,904 Solidity files, featuring 1,618
SWC weaknesses sourced from 682 real-world DApp projects. However, the Solidity
files in this dataset may not be directly compilable for further analysis. To
facilitate automated analysis, we developed a tool capable of automatically
identifying dependency relationships within DApp projects and completing
missing public libraries. Using this tool, we created DAPPSCAN-BYTECODE
dataset, which consists of 6,665 compiled smart contract with 888 SWC
weaknesses. Based on DAPPSCAN-BYTECODE, we conducted an empirical study to
evaluate the performance of state-of-the-art smart contract weakness detection
tools. The evaluation results revealed sub-par performance for these tools in
terms of both effectiveness and success detection rate, indicating that future
development should prioritize real-world datasets over simplistic toy
contracts.
