  Comprehending the performance bottlenecks at the core of the intricate
hardware-software interactions exhibited by highly parallel programs on HPC
clusters is crucial. This paper sheds light on the issue of automatically
asynchronous MPI communication in memory-bound parallel programs on multicore
clusters and how it can be facilitated. For instance, slowing down MPI
processes by deliberate injection of delays can improve performance if certain
conditions are met. This leads to the counter-intuitive conclusion that noise,
independent of its source, is not always detrimental but can be leveraged for
performance improvements. We employ phase-space graphs as a new tool to
visualize parallel program dynamics. They are useful in spotting certain
patterns in parallel execution that will easily go unnoticed with traditional
tracing tools. We investigate five different microbenchmarks and applications
on different supercomputer platforms: an MPI-augmented STREAM Triad, two
implementations of Lattice-Boltzmann fluid solvers, and the LULESH and HPCG
proxy applications.
