  Typically, a randomized experiment is designed to test a hypothesis about the
average treatment effect and sometimes hypotheses about treatment effect
variation. The results of such a study may then be used to inform policy and
practice for units not in the study. In this paper, we argue that given this
use, randomized experiments should instead be designed to predict unit-specific
treatment effects in a well-defined population. We then consider how different
sampling processes and models affect the bias, variance, and mean squared
prediction error of these predictions. The results indicate, for example, that
problems of generalizability (differences between samples and populations) can
greatly affect bias both in predictive models and in measures of error in these
models. We also examine when the average treatment effect estimate outperforms
unit-specific treatment effect predictive models and implications of this for
planning studies.
