  With tremendous growing interests in Big Data systems, analyzing and
facilitating their performance improvement become increasingly important.
Although there have much research efforts for improving Big Data systems
performance, efficiently analysing and diagnosing performance bottlenecks over
these massively distributed systems remain a major challenge. In this paper, we
propose a spatio-temporal correlation analysis approach based on stage
characteristic and distribution characteristic of Big Data applications, which
can associate the multi-level performance data fine-grained. On the basis of
correlation data, we define some priori rules, select features and vectorize
the corresponding datasets for different performance bottlenecks, such as,
workload imbalance, data skew, abnormal node and outlier metrics. And then, we
utilize the data and model driven algorithms for bottlenecks detection and
diagnosis. In addition, we design and develop a lightweight, extensible tool
HybridTune, and validate the diagnosis effectiveness of our tool with
BigDataBench on several benchmark experiments in which the outperform
state-of-the-art methods. Our experiments show that the accuracy of
abnormal/outlier detection we obtained reaches about 80%. At last, we report
several Spark and Hadoop use cases, which are demonstrated how HybridTune
supports users to carry out the performance analysis and diagnosis efficiently
on the Spark and Hadoop applications, and our experiences demonstrate
HybridTune can help users find the performance bottlenecks and provide
optimization recommendations.
