  In recent years, functional neural networks have been proposed and studied in
order to approximate nonlinear continuous functionals defined on $L^p([-1,
1]^s)$ for integers $s\ge1$ and $1\le p<\infty$. However, their theoretical
properties are largely unknown beyond universality of approximation or the
existing analysis does not apply to the rectified linear unit (ReLU) activation
function. To fill in this void, we investigate here the approximation power of
functional deep neural networks associated with the ReLU activation function by
constructing a continuous piecewise linear interpolation under a simple
triangulation. In addition, we establish rates of approximation of the proposed
functional deep ReLU networks under mild regularity conditions. Finally, our
study may also shed some light on the understanding of functional data learning
algorithms.
