  Large-scale, fault-tolerant, distributed systems are the backbone for many
critical software services. Since they must execute correctly in a possibly
adversarial environment with arbitrary communication delays and failures, the
underlying algorithms are intricate. In particular, achieving consistency and
data retention relies on intricate consensus (state machine replication)
protocols. Ensuring the reliability of implementations of such protocols
remains a significant challenge because of the enormous number of exceptional
conditions that may arise in production. We propose a methodology and a tool
called Netrix for testing such implementations that aims to exploit
programmer's knowledge to improve coverage, enables robust bug reproduction,
and can be used in regression testing across different versions of an
implementation. As evaluation, we apply our tool to a popular proof of stake
blockchain protocol, Tendermint, which relies on a Byzantine consensus
algorithm, a benign consensus algorithm, Raft, and BFT-Smart. We were able to
identify 4 deviations of the Tendermint implementation from the protocol
specification and check their absence on an updated implementation.
Additionally, we were able to reproduce 4 previously known bugs in Raft.
