  It is hard to operate and debug systems like OpenStack that integrate many
independently developed modules with multiple levels of abstractions. A major
challenge is to navigate through the complex dependencies and relationships of
the states in different modules or subsystems, to ensure the correctness and
consistency of these states. We present a system that captures the runtime
states and events from the entire OpenStack-Ceph stack, and automatically
organizes these data into a graph that we call system operation state graph
(SOSG).With SOSG we can use intuitive graph traversal techniques to solve
problems like reasoning about the state of a virtual machine. Also, using
graph-based anomaly detection, we can automatically discover hidden problems in
OpenStack. We have a scalable implementation of SOSG, and evaluate the approach
on a 125-node production OpenStack cluster, finding a number of interesting
problems.
