  The Neyman Allocation is used in many papers on experimental design, which
typically assume that researchers have access to large pilot studies. This may
be unrealistic. To understand the properties of the Neyman Allocation with
small pilots, we study its behavior in an asymptotic framework that takes pilot
size to be fixed even as the size of the main wave tends to infinity. Our
analysis shows that the Neyman Allocation can lead to estimates of the ATE with
higher asymptotic variance than with (non-adaptive) balanced randomization. In
particular, this happens when the outcome variable is relatively homoskedastic
with respect to treatment status or when it exhibits high kurtosis. We provide
a series of empirical examples showing that such situations can arise in
practice. Our results suggest that researchers with small pilots should not use
the Neyman Allocation if they believe that outcomes are homoskedastic or
heavy-tailed. Finally, we examine some potential methods for improving the
finite sample performance of the FNA via simulations.
