  Background: Publication bias is the failure to publish the results of a study
based on the direction or strength of the study findings. The existence of
publication bias is firmly established in areas like medical research. Recent
research suggests the existence of publication bias in Software Engineering.
Aims: Finding out whether experiments published in the International Workshop
on Empirical Software Engineering and Measurement (ESEM) are affected by
publication bias. Method: We review experiments published in ESEM. We also
survey with experimental researchers to triangulate our findings. Results: ESEM
experiments do not define hypotheses and frequently perform multiple testing.
One-tailed tests have a slightly higher rate of achieving statistically
significant results. We could not find other practices associated with
publication bias. Conclusions: Our results provide a more encouraging
perspective of SE research than previous research: (1) ESEM publications do not
seem to be strongly affected by biases and (2) we identify some practices that
could be associated with p-hacking, but it is more likely that they are related
to the conduction of exploratory research.
