  Background. In recent years, cyber security user studies have been
scrutinized for their reporting completeness, statistical reporting fidelity,
statistical reliability and biases. It remains an open question what strength
of evidence positive reports of such studies actually yield. We focus on the
extent to which positive reports indicate relation true in reality, that is, a
probabilistic assessment.
  Aim. This study aims at establishing the overall strength of evidence in
cyber security user studies, with the dimensions -- Positive Predictive Value
(PPV) and its complement False Positive Risk (FPR), -- Likelihood Ratio (LR),
and -- Reverse-Bayesian Prior (RBP) for a fixed tolerated False Positive Risk.
  Method. Based on $431$ coded statistical inferences in $146$ cyber security
user studies from a published SLR covering the years 2006-2016, we first
compute a simulation of the a posteriori false positive risk based on assumed
prior and bias thresholds. Second, we establish the observed likelihood ratios
for positive reports. Third, we compute the reverse Bayesian argument on the
observed positive reports by computing the prior required for a fixed a
posteriori false positive rate.
  Results. We obtain a comprehensive analysis of the strength of evidence
including an account of appropriate multiple comparison corrections. The
simulations show that even in face of well-controlled conditions and high prior
likelihoods, only few studies achieve good a posteriori probabilities.
  Conclusions. Our work shows that the strength of evidence of the field is
weak and that most positive reports are likely false. From this, we learn what
to watch out for in studies to advance the knowledge of the field.
