  Applied researchers are increasingly interested in whether and how treatment
effects vary in randomized evaluations, especially variation not explained by
observed covariates. We propose a model-free approach for testing for the
presence of such unexplained variation. To use this randomization-based
approach, we must address the fact that the average treatment effect, generally
the object of interest in randomized experiments, actually acts as a nuisance
parameter in this setting. We explore potential solutions and advocate for a
method that guarantees valid tests in finite samples despite this nuisance. We
also show how this method readily extends to testing for heterogeneity beyond a
given model, which can be useful for assessing the sufficiency of a given
scientific theory. We finally apply our method to the National Head Start
Impact Study, a large-scale randomized evaluation of a Federal preschool
program, finding that there is indeed significant unexplained treatment effect
variation.
