  Microbenchmarking is a widely used form of performance testing in Java
software. A microbenchmark repeatedly executes a small chunk of code while
collecting measurements related to its performance. Due to Java Virtual Machine
optimizations, microbenchmarks are usually subject to severe performance
fluctuations in the first phase of their execution (also known as warmup). For
this reason, software developers typically discard measurements of this phase
and focus their analysis when benchmarks reach a steady state of performance.
Developers estimate the end of the warmup phase based on their expertise, and
configure their benchmarks accordingly. Unfortunately, this approach is based
on two strong assumptions: (i) benchmarks always reach a steady state of
performance and (ii) developers accurately estimate warmup. In this paper, we
show that Java microbenchmarks do not always reach a steady state, and often
developers fail to accurately estimate the end of the warmup phase. We found
that a considerable portion of studied benchmarks do not hit the steady state,
and warmup estimates provided by software developers are often inaccurate (with
a large error). This has significant implications both in terms of results
quality and time-effort. Furthermore, we found that dynamic reconfiguration
significantly improves warmup estimation accuracy, but still it induces
suboptimal warmup estimates and relevant side-effects. We envision this paper
as a starting point for supporting the introduction of more sophisticated
automated techniques that can ensure results quality in a timely fashion.
