  Permutation tests date back nearly a century to Fisher's randomized
experiments, and remain an immensely popular statistical tool, used for testing
hypotheses of independence between variables and other common inferential
questions. Much of the existing literature has emphasized that, for the
permutation p-value to be valid, one must first pick a subgroup $G$ of
permutations (which could equal the full group) and then recalculate the test
statistic on permuted data using either an exhaustive enumeration of $G$, or a
sample from $G$ drawn uniformly at random. In this work, we demonstrate that
the focus on subgroups and uniform sampling are both unnecessary for validity
-- in fact, a simple random modification of the permutation p-value remains
valid even when using an arbitrary distribution (not necessarily uniform) over
any subset of permutations (not necessarily a subgroup). We provide a unified
theoretical treatment of such generalized permutation tests, recovering all
known results from the literature as special cases. Thus, this work expands the
flexibility of the permutation test toolkit available to the practitioner.
