  This paper presents the results of a research study related to software
system failures, with the goal of understanding how we might better evolve,
maintain and support software systems in production. We have qualitatively
analyzed thirty incidents: fifteen collected through in depth interviews with
engineers, and fifteen sampled from publicly published incident reports
(generally produced as part of postmortem reviews). Our analysis focused on
understanding and categorizing how failures occurred, and how they were
detected, investigated and mitigated. We also captured analytic insights
related to the current state of the practice and associated challenges in the
form of 11 key observations. For example, we observed that failures can cascade
through a system leading to major outages; and that often engineers do not
understand the scaling limits of systems they are supporting until those limits
are exceeded. We argue that the challenges we have identified can lead to
improvements to how systems are engineered and supported.
