  Over the last few years, there has been substantial research on automated
analysis, testing, and debugging of Ethereum smart contracts. However, it is
not trivial to compare and reproduce that research. To address this, we present
an empirical evaluation of 9 state-of-the-art automated analysis tools using
two new datasets: i) a dataset of 69 annotated vulnerable smart contracts that
can be used to evaluate the precision of analysis tools; and ii) a dataset with
all the smart contracts in the Ethereum Blockchain that have Solidity source
code available on Etherscan (a total of 47,518 contracts). The datasets are
part of SmartBugs, a new extendable execution framework that we created to
facilitate the integration and comparison between multiple analysis tools and
the analysis of Ethereum smart contracts. We used SmartBugs to execute the 9
automated analysis tools on the two datasets. In total, we ran 428,337 analyses
that took approximately 564 days and 3 hours, being the largest experimental
setup to date both in the number of tools and in execution time. We found that
only 42% of the vulnerabilities from our annotated dataset are detected by all
the tools, with the tool Mythril having the higher accuracy (27%). When
considering the largest dataset, we observed that 97% of contracts are tagged
as vulnerable, thus suggesting a considerable number of false positives.
Indeed, only a small number of vulnerabilities (and of only two categories)
were detected simultaneously by four or more tools.
