  When different researchers study the same research question using the same
dataset they may obtain different and potentially even conflicting results.
This is because there is often substantial flexibility in researchers'
analytical choices, an issue also referred to as ''researcher degrees of
freedom''. Combined with selective reporting of the smallest p-value or largest
effect, researcher degrees of freedom may lead to an increased rate of false
positive and overoptimistic results. In this paper, we address this issue by
formalizing the multiplicity of analysis strategies as a multiple testing
problem. As the test statistics of different analysis strategies are usually
highly dependent, a naive approach such as the Bonferroni correction is
inappropriate because it leads to an unacceptable loss of power. Instead, we
propose using the ''minP'' adjustment method, which takes potential test
dependencies into account and approximates the underlying null distribution of
the minimal p-value through a permutation-based procedure. This procedure is
known to achieve more power than simpler approaches while ensuring a weak
control of the family-wise error rate. We illustrate our approach for
addressing researcher degrees of freedom by applying it to a study on the
impact of perioperative paO2 on post-operative complications after
neurosurgery. A total of 48 analysis strategies are considered and adjusted
using the minP procedure. This approach allows to selectively report the result
of the analysis strategy yielding the most convincing evidence, while
controlling the type 1 error -- and thus the risk of publishing false positive
results that may not be replicable.
