  Large scale cloud services use Key Performance Indicators (KPIs) for tracking
and monitoring performance. They usually have Service Level Objectives (SLOs)
baked into the customer agreements which are tied to these KPIs. Dependency
failures, code bugs, infrastructure failures, and other problems can cause
performance regressions. It is critical to minimize the time and manual effort
in diagnosing and triaging such issues to reduce customer impact. Large volume
of logs and mixed type of attributes (categorical, continuous) in the logs
makes diagnosis of regressions non-trivial.
  In this paper, we present the design, implementation and experience from
building and deploying DeCaf, a system for automated diagnosis and triaging of
KPI issues using service logs. It uses machine learning along with pattern
mining to help service owners automatically root cause and triage performance
issues. We present the learnings and results from case studies on two large
scale cloud services in Microsoft where DeCaf successfully diagnosed 10 known
and 31 unknown issues. DeCaf also automatically triages the identified issues
by leveraging historical data. Our key insights are that for any such diagnosis
tool to be effective in practice, it should a) scale to large volumes of
service logs and attributes, b) support different types of KPIs and ranking
functions, c) be integrated into the DevOps processes.
