  The Message Passing Interface (MPI) is the most commonly used application
programming interface for process communication on current large-scale parallel
systems. Due to the scale and complexity of modern parallel architectures, it
is becoming increasingly difficult to optimize MPI libraries, as many factors
can influence the communication performance. To assist MPI developers and
users, we propose an automatic way to check whether MPI libraries respect
self-consistent performance guidelines for collective communication operations.
We introduce the PGMPI framework to detect violations of performance guidelines
through benchmarking. Our experimental results show that PGMPI can pinpoint
undesired and often unexpected performance degradations of collective MPI
operations. We demonstrate how to overcome performance issues of several
libraries by adapting the algorithmic implementations of their respective
collective MPI calls.
