  An unfortunate feature of traditional hypothesis testing is the necessity to
pre-specify a significance level $\alpha$ to bound the `size' of the test: its
probability to falsely reject the hypothesis. Indeed, a data-dependent
selection of $\alpha$ would generally distort the size, possibly making it
larger than the selected level $\alpha$. We develop post-hoc hypothesis
testing, which guarantees that there is no such size distortion in expectation,
even if the level $\alpha$ is arbitrarily selected based on the data. Unlike
regular $p$-values, resulting `post-hoc $p$-values' allow us to `reject at
level $p$' and still provide this guarantee. Moreover, they can easily be
combined since the product of independent post-hoc $p$-values is also a
post-hoc $p$-value. Interestingly, we find that $p$ is a post-hoc $p$-value if
and only if $1/p$ is an $e$-value, a recently introduced measure of evidence.
Post-hoc hypothesis testing eliminates the need for standardized levels such as
$\alpha = 0.05$, which takes away incentives for $p$-hacking and contributes to
solving the file-drawer problem.
