  The simple program and multiple data (SPMD) programming model is widely used
for both high performance computing and Cloud computing. In this paper, we
design and implement an innovative system, AutoAnalyzer, that automates the
process of debugging performance problems of SPMD-style parallel programs,
including data collection, performance behavior analysis, locating bottlenecks,
and uncovering their root causes. AutoAnalyzer is unique in terms of two
features: first, without any apriori knowledge, it automatically locates
bottlenecks and uncovers their root causes for performance optimization;
second, it is lightweight in terms of the size of performance data to be
collected and analyzed. Our contributions are three-fold: first, we propose two
effective clustering algorithms to investigate the existence of performance
bottlenecks that cause process behavior dissimilarity or code region behavior
disparity, respectively; meanwhile, we present two searching algorithms to
locate bottlenecks; second, on a basis of the rough set theory, we propose an
innovative approach to automatically uncovering root causes of bottlenecks;
third, on the cluster systems with two different configurations, we use two
production applications, written in Fortran 77, and one open source
code-MPIBZIP2 (http://compression.ca/mpibzip2/), written in C++, to verify the
effectiveness and correctness of our methods. For three applications, we also
propose an experimental approach to investigating the effects of different
metrics on locating bottlenecks.
