  Null hypothesis statistical significance tests (NHST) are widely used in
quantitative research in the empirical sciences including scientometrics.
Nevertheless, since their introduction nearly a century ago significance tests
have been controversial. Many researchers are not aware of the numerous
criticisms raised against NHST. As practiced, NHST has been characterized as a
null ritual that is overused and too often misapplied and misinterpreted. NHST
is in fact a patchwork of two fundamentally different classical statistical
testing models, often blended with some wishful quasi-Bayesian interpretations.
This is undoubtedly a major reason why NHST is very often misunderstood. But
NHST also has intrinsic logical problems and the epistemic range of the
information provided by such tests is much more limited than most researchers
recognize. In this article we introduce to the scientometric community the
theoretical origins of NHST, which is mostly absent from standard statistical
textbooks, and we discuss some of the most prevalent problems relating to the
practice of NHST and trace these problems back to the mixup of the two
different theoretical origins. Finally, we illustrate some of the
misunderstandings with examples from the scientometric literature and bring
forward some modest recommendations for a more sound practice in quantitative
data analysis.
