  Production distributed systems are challenging to formally verify, in
particular when they are based on distributed protocols that are not rigorously
described or fully understood. In this paper, we derive models and properties
for two core distributed protocols used in eventually consistent production
key-value stores such as Riak and Cassandra. We propose a novel modeling called
certified program models, where complete distributed systems are captured as
programs written in traditional systems languages such as concurrent C.
Specifically, we model the read-repair and hinted-handoff recovery protocols as
concurrent C programs, test them for conformance with real systems, and then
verify that they guarantee eventual consistency, modeling precisely the
specification as well as the failure assumptions under which the results hold.
