  We consider the performance of the difference-in-means estimator in a two-arm
randomized experiment under common experimental endpoints such as continuous
(regression), incidence, proportion and survival. We examine performance under
both equal and unequal allocation to treatment groups and we consider both the
Neyman randomization model and the population model. We show that in the Neyman
model, where the only source of randomness is the treatment manipulation, there
is no free lunch: complete randomization is minimax for the estimator's mean
squared error. In the population model, where each subject experiences response
noise with zero mean, the optimal design is the deterministic perfect-balance
allocation. However, this allocation is generally NP-hard to compute and
moreover, depends on unknown response parameters. When considering the tail
criterion of Kapelner et al. (2021), we show the optimal design is less random
than complete randomization and more random than the deterministic
perfect-balance allocation. We prove that Fisher's blocking design provides the
asymptotically optimal degree of experimental randomness. Theoretical results
are supported by simulations in all considered experimental settings.
