  Good (Frequentist) statistical practice requires that statistical tests be
performed in order to determine if the phenomenon being observed could
plausibly occur by chance if the null hypothesis is false. Good practice also
requires that a test is not performed if the study is underpowered: if the
number of observations is not sufficiently large to be able to reliably detect
the effect one hypothesizes, even if the effect exists. Running underpowered
studies runs the risk of false negative results. This creates tension in the
guidelines and expectations for computer science education conferences: while
things are clear for studies with a large number of observations, researchers
should in fact not compute p-values and perform statistical tests if the number
of observations is too small. The issue is particularly live in CSed venues,
since class sizes where those issues are salient are common. We outline the
considerations for when to compute and when not to compute p-values in
different settings encountered by computer science education researchers. We
survey the author and reviewer guidelines in different computer science
education conferences (ICER, SIGCSE TS, ITiCSE, EAAI, CompEd, Koli Calling). We
present summary data and make several preliminary observations about reviewer
guidelines: guidelines vary from conference to conference; guidelines allow for
qualitative studies, and, in some cases, experience reports, but guidelines do
not generally explicitly indicate that a paper should have at least one of (1)
an appropriately-powered statistical analysis or (2) rich qualitative
descriptions. We present preliminary ideas for addressing the tension in the
guidelines between small-n and large-n studies
