  The increasing complexity of the software/hardware stack of modern
supercomputers results in explosion of parameters. The performance analysis
becomes a truly experimental science, even more challenging in the presence of
massive irregularity and data dependency. We analyze how the existing body of
research handles the experimental aspect in the context of distributed graph
algorithms (DGAs). We distinguish algorithm-level contributions, often
prioritized by authors, from runtime-level concerns that are harder to place.
We show that the runtime is such an integral part of DGAs that experimental
results are difficult to interpret and extrapolate without understanding the
properties of the runtime used. We argue that in order to gain understanding
about the impact of runtimes, more information needs to be gathered. To begin
this process, we provide an initial set of recommendations for describing DGA
results based on our analysis of the current state of the field.
