  Blockchain programs (also known as smart contracts) manage valuable assets
like cryptocurrencies and tokens, and implement protocols in domains like
decentralized finance (DeFi) and supply-chain management. These types of
applications require a high level of security that is hard to achieve due to
the transparency of public blockchains. Numerous tools support developers and
auditors in the task of detecting weaknesses. As a young technology,
blockchains and utilities evolve fast, making it challenging for tools and
developers to keep up with the pace.
  In this work, we study the robustness of code analysis tools and the
evolution of weakness detection on a dataset representing six years of
blockchain activity. We focus on Ethereum as the crypto ecosystem with the
largest number of developers and deployed programs. We investigate the behavior
of single tools as well as the agreement of several tools addressing similar
weaknesses.
  Our study is the first that is based on the entire body of deployed bytecode
on Ethereum's main chain. We achieve this coverage by considering bytecodes as
equivalent if they share the same skeleton. The skeleton of a bytecode is
obtained by omitting functionally irrelevant parts. This reduces the 48 million
contracts deployed on Ethereum up to January 2022 to 248328 contracts with
distinct skeletons. For bulk execution, we utilize the open-source framework
SmartBugs that facilitates the analysis of Solidity smart contracts, and
enhance it to accept also bytecode as the only input. Moreover, we integrate
six further tools for bytecode analysis. The execution of the 12 tools included
in our study on the dataset took 30 CPU years. While the tools report a total
of 1307486 potential weaknesses, we observe a decrease in reported weaknesses
over time, as well as a degradation of tools to varying degrees.
