  Security attacks targeting smart contracts have been on the rise, which have
led to financial loss and erosion of trust. Therefore, it is important to
enable developers to discover security vulnerabilities in smart contracts
before deployment. A number of static analysis tools have been developed for
finding security bugs in smart contracts. However, despite the numerous
bug-finding tools, there is no systematic approach to evaluate the proposed
tools and gauge their effectiveness. This paper proposes SolidiFI, an automated
and systematic approach for evaluating smart contract static analysis tools.
SolidiFI is based on injecting bugs (i.e., code defects) into all potential
locations in a smart contract to introduce targeted security vulnerabilities.
SolidiFI then checks the generated buggy contract using the static analysis
tools, and identifies the bugs that the tools are unable to detect
(false-negatives) along with identifying the bugs reported as false-positives.
SolidiFI is used to evaluate six widely-used static analysis tools, namely,
Oyente, Securify, Mythril, SmartCheck, Manticore and Slither, using a set of 50
contracts injected by 9369 distinct bugs. It finds several instances of bugs
that are not detected by the evaluated tools despite their claims of being able
to detect such bugs, and all the tools report many false positives
