  Modern cloud infrastructure is powered by cluster management systems such as
Kubernetes and Docker Swarm. While these systems seek to minimize users'
operational burden, the complex, dynamic, and non-deterministic nature of these
systems makes them hard to reason about, potentially leading to failures
ranging from performance degradation to outages. We present Kivi, the first
system for verifying controllers and their configurations in cluster management
systems. Kivi focuses on the popular system Kubernetes, and models its
controllers and events into processes whereby their interleavings are
exhaustively checked via model checking. Central to handling autoscaling and
large-scale deployments is our design that seeks to find violations in a
smaller and reduced topology. We also develop several model optimizations in
Kivi to scale to large clusters. We show that Kivi is effective and accurate in
finding issues in realistic and complex scenarios and showcase two new issues
in Kubernetes controller source code.
