  Rosenthal's (1979) Fail-Safe-Number (FSN) is probably one of the best known
statistics in the context of meta-analysis aimed to estimate the number of
unpublished studies in meta-analyses required to bring the meta-analytic mean
effect size down to a statistically insignificant level. Already before
Scargle's (2000) and Schonemann & Scargle's (2008) fundamental critique on the
claimed stability of the basic rationale of the FSN approach, objections
focusing on the basic assumption of the FSN which treats the number of studies
as unbiased with averaging null were expressed throughout the history of the
FSN by different authors (Elashoff, 1978; Iyengar & Greenhouse, 1988a; 1988b;
see also Scargle, 2000). In particular, Elashoff's objection appears to be
important because it was the very first critique pointing directly to the
central problem of the FSN: "R & R claim that the number of studies hidden in
the drawers would have to be 65,000 to achieve a mean effect size of zero when
combined with the 345 studies reviewed here. But surely, if we allowed the
hidden studies to be negative, on the average no more than 345 hidden studies
would be necessary to obtain a zero mean effect size" (p. 392). Thus, users of
meta-analysis could have been aware right from the beginning that something was
wrong with the statistical reasoning of the FSN. In particular, from an applied
research perspective, it is therefore of interest whether any of the
fundamental objections on the FSN are reflected in standard handbooks on
meta-analysis as well as -and of course even more importantly- in meta-analytic
studies itself.
