  We present Kaleidoscope an innovative system that supports live forensics for
application performance problems caused by either individual component failures
or resource contention issues in large-scale distributed storage systems. The
design of Kaleidoscope is driven by our study of I/O failures observed in a
peta-scale storage system anonymized as PetaStore. Kaleidoscope is built on
three key features: 1) using temporal and spatial differential observability
for end-to-end performance monitoring of I/O requests, 2) modeling the health
of storage components as a stochastic process using domain-guided functions
that accounts for path redundancy and uncertainty in measurements, and, 3)
observing differences in reliability and performance metrics between similar
types of healthy and unhealthy components to attribute the most likely root
causes. We deployed Kaleidoscope on PetaStore and our evaluation shows that
Kaleidoscope can run live forensics at 5-minute intervals and pinpoint the root
causes of 95.8% of real-world performance issues, with negligible monitoring
overhead.
