  Large-scale data centers and cloud computing have turned system configuration
into a challenging problem. Several widely-publicized outages have been blamed
not on software bugs, but on configuration bugs. To cope, thousands of
organizations use system configuration languages to manage their computing
infrastructure. Of these, Puppet is the most widely used with thousands of
paying customers and many more open-source users. The heart of Puppet is a
domain-specific language that describes the state of a system. Puppet already
performs some basic static checks, but they only prevent a narrow range of
errors. Furthermore, testing is ineffective because many errors are only
triggered under specific machine states that are difficult to predict and
reproduce. With several examples, we show that a key problem with Puppet is
that configurations can be non-deterministic.
  This paper presents Rehearsal, a verification tool for Puppet configurations.
Rehearsal implements a sound, complete, and scalable determinacy analysis for
Puppet. To develop it, we (1) present a formal semantics for Puppet, (2) use
several analyses to shrink our models to a tractable size, and (3) frame
determinism-checking as decidable formulas for an SMT solver. Rehearsal then
leverages the determinacy analysis to check other important properties, such as
idempotency. Finally, we apply Rehearsal to several real-world Puppet
configurations.
