  Modern data centers that provide Internet-scale services are stadium-size
structures housing tens of thousands of heterogeneous devices (server clusters,
networking equipment, power and cooling infrastructures) that must operate
continuously and reliably. As part of their operation, these devices produce
large amounts of data in the form of event and error logs that are essential
not only for identifying problems but also for improving data center efficiency
and management. These activities employ data analytics and often exploit hidden
statistical patterns and correlations among different factors present in the
data. Uncovering these patterns and correlations is challenging due to the
sheer volume of data to be analyzed. This paper presents BiDAl, a prototype
"log-data analysis framework" that incorporates various Big Data technologies
to simplify the analysis of data traces from large clusters. BiDAl is written
in Java with a modular and extensible architecture so that different storage
backends (currently, HDFS and SQLite are supported), as well as different
analysis languages (current implementation supports SQL, R and Hadoop
MapReduce) can be easily selected as appropriate. We present the design of
BiDAl and describe our experience using it to analyze several public traces of
Google data clusters for building a simulation model capable of reproducing
observed behavior.
