  This paper establishes the (nearly) optimal approximation error
characterization of deep rectified linear unit (ReLU) networks for smooth
functions in terms of both width and depth simultaneously. To that end, we
first prove that multivariate polynomials can be approximated by deep ReLU
networks of width $\mathcal{O}(N)$ and depth $\mathcal{O}(L)$ with an
approximation error $\mathcal{O}(N^{-L})$. Through local Taylor expansions and
their deep ReLU network approximations, we show that deep ReLU networks of
width $\mathcal{O}(N\ln N)$ and depth $\mathcal{O}(L\ln L)$ can approximate
$f\in C^s([0,1]^d)$ with a nearly optimal approximation error
$\mathcal{O}(\|f\|_{C^s([0,1]^d)}N^{-2s/d}L^{-2s/d})$. Our estimate is
non-asymptotic in the sense that it is valid for arbitrary width and depth
specified by $N\in\mathbb{N}^+$ and $L\in\mathbb{N}^+$, respectively.
