  The minimal number of neurons required for a feedforward neural network to
interpolate $n$ generic input-output pairs from $\mathbb{R}^d\times \mathbb{R}$
is $\Theta(\sqrt{n})$. While previous results have shown that
$\Theta(\sqrt{n})$ neurons are sufficient, they have been limited to logistic,
Heaviside, and rectified linear unit (ReLU) as the activation function. Using a
different approach, we prove that $\Theta(\sqrt{n})$ neurons are sufficient as
long as the activation function is real analytic at a point and not a
polynomial there. Thus, the only practical activation functions that our result
does not apply to are piecewise polynomials. Importantly, this means that
activation functions can be freely chosen in a problem-dependent manner without
loss of interpolation power.
