  This is the second part of a small-scale explorative study in an effort to
assess reproducibility issues specific to scientometrics research. This effort
is motivated by the desire to generate empirical data to inform debates about
reproducibility in scientometrics. Rather than attempt to reproduce studies, we
explore how we might assess "in principle" reproducibility based on a critical
review of the content of published papers. While the first part of the study
(Waltman et al. 2018) focuses on direct reproducibility - that is the ability
to reproduce the specific evidence produced by an original study using the same
data, methods, and procedures, this second part is dedicated to conceptual
reproducibility - that is the robustness of knowledge claims towards
verification by an alternative approach using different data, methods and
procedures. The study is exploratory: it investigates only a very limited
number of publications and serves us to develop instruments for identifying
potential reproducibility issues of published studies: These are a
categorization of study types and a taxonomy of threats to reproducibility. We
work with a select sample of five publications in scientometrics covering a
variation of study types of theoretical, methodological, and empirical nature.
Based on observations made during our exploratory review, we conclude with open
questions on how to approach and assess the status of conceptual
reproducibility in scientometrics intended for discussion at the special track
on "Reproducibility in Scientometrics" at STI2018 in Leiden.
