  Performance variability has been acknowledged as a problem for over a decade
by cloud practitioners and performance engineers. Yet, our survey of top
systems conferences reveals that the research community regularly disregards
variability when running experiments in the cloud. Focusing on networks, we
assess the impact of variability on cloud-based big-data workloads by gathering
traces from mainstream commercial clouds and private research clouds. Our data
collection consists of millions of datapoints gathered while transferring over
9 petabytes of data. We characterize the network variability present in our
data and show that, even though commercial cloud providers implement mechanisms
for quality-of-service enforcement, variability still occurs, and is even
exacerbated by such mechanisms and service provider policies. We show how
big-data workloads suffer from significant slowdowns and lack predictability
and replicability, even when state-of-the-art experimentation techniques are
used. We provide guidelines for practitioners to reduce the volatility of big
data performance, making experiments more repeatable.
