  Van Miltenburg et al. (2021) suggest NLP research should adopt
preregistration to prevent fishing expeditions and to promote publication of
negative results. At face value, this is a very reasonable suggestion,
seemingly solving many methodological problems with NLP research. We discuss
pros and cons -- some old, some new: a) Preregistration is challenged by the
practice of retrieving hypotheses after the results are known; b)
preregistration may bias NLP toward confirmatory research; c) preregistration
must allow for reclassification of research as exploratory; d) preregistration
may increase publication bias; e) preregistration may increase flag-planting;
f) preregistration may increase p-hacking; and finally, g) preregistration may
make us less risk tolerant. We cast our discussion as a dialogue, presenting
both sides of the debate.
