  In stochastic optimization, particularly in evolutionary computation and
reinforcement learning, the optimization of a function $f: \Omega \to
\mathbb{R}$ is often addressed through optimizing a so-called relaxation
$\theta \in \Theta \mapsto \mathbb{E}_\theta(f)$ of $f$, where $\Theta$
resembles the parameters of a family of probability measures on $\Omega$. We
investigate the structure of such relaxations by means of measure theory and
Fourier analysis, enabling us to shed light on the success of many associated
stochastic optimization methods. The main structural traits we derive and that
allow fast and reliable optimization of relaxations are the consistency of
optimal values of $f$, Lipschitzness of gradients, and convexity. We emphasize
settings where $f$ itself is not differentiable or convex, e.g., in the
presence of (stochastic) disturbance.
