  A critical component of any blockchain or distributed ledger technology (DLT)
platform is the consensus algorithm. Blockchain consensus algorithms are the
primary vehicle for the nodes within a blockchain network to reach an
agreement. In recent years, many blockchain consensus algorithms have been
proposed mainly for private and permissioned blockchain networks. However, the
performance of these algorithms and their reliability in hostile environments
or the presence of byzantine and other network failures are not well
understood. In addition, the testing and validation of blockchain applications
come with many technical challenges. In this paper, we apply chaos engineering
and testing to understand the performance of consensus algorithms in the
presence of different loads, byzantine failure and other communication failure
scenarios. We apply chaos engineering to evaluate the performance of three
different consensus algorithms (PBFT, Clique, Raft) and their respective
blockchain platforms. We measure the blockchain network's throughput, latency,
and success rate while executing chaos and load tests. We develop lightweight
blockchain applications to execute our test in a semi-production environment.
Our results show that using chaos engineering helps understand how different
consensus algorithms perform in a hostile or unreliable environment and the
limitations of blockchain platforms. Our work demonstrates the benefits of
using chaos engineering in testing complex distributed systems such as
blockchain networks.
