  Background. In recent years, cyber security security user studies have been
appraised in meta-research, mostly focusing on the completeness of their
statistical inferences and the fidelity of their statistical reporting.
However, estimates of the field's distribution of statistical power and its
publication bias have not received much attention. Aim. In this study, we aim
to estimate the effect sizes and their standard errors present as well as the
implications on statistical power and publication bias. Method. We built upon a
published systematic literature review of $146$ user studies in cyber security
(2006--2016). We took into account $431$ statistical inferences including $t$-,
$\chi^2$-, $r$-, one-way $F$-tests, and $Z$-tests. In addition, we coded the
corresponding total sample sizes, group sizes and test families. Given these
data, we established the observed effect sizes and evaluated the overall
publication bias. We further computed the statistical power vis-{\`a}-vis of
parametrized population thresholds to gain unbiased estimates of the power
distribution. Results. We obtained a distribution of effect sizes and their
conversion into comparable log odds ratios together with their standard errors.
We, further, gained funnel-plot estimates of the publication bias present in
the sample as well as insights into the power distribution and its
consequences. Conclusions. Through the lenses of power and publication bias, we
shed light on the statistical reliability of the studies in the field. The
upshot of this introspection is practical recommendations on conducting and
evaluating studies to advance the field.
