<p>The universal approximation theorem (<a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="noreferrer">https://en.wikipedia.org/wiki/Universal_approximation_theorem</a>)
informally states that up to several conditions, any function can be approximated by a shallow neural network up to any degree.</p>

<p>I was wondering if there is a version of this theorem that states the same for the gradients of the functions, i.e, for any function there is a (shallow or not shallow) neural network that approximates it and its gradient approximates the gradient of the original function (up to any degree)?</p>

CC BY-SA 3.0
2017-12-30T21:33:35.243Z
A possible approach: given a continuous, differentiable function $f:[0,1]^n \to \mathbb{R}$ and its gradient $\nabla f$, first construct a piecewise linear function $g:[0,1]^n \to \mathbb{R}$ such that $g(x) \approx f(x)$ and $\nabla g(x) \approx \nabla f(x)$ everywhere, then implement or approximate $g(x)$ with a neural network.  At least if $n=1$ I think that with a ReLU activation function a neural network can directly and exactly compute any desired continuous piecewise linear function (not just approximate it).  I don't know if that's helpful.
D.W.
CC BY-SA 3.0
2018-01-01T13:13:46.993Z
Actually, that's a good answer (https://arxiv.org/abs/1611.01491). I forgot to mention that I'm looking for smooth activation functions. So, ReLU is off the table. I actually solved it using http://zmjones.com/static/statistical-learning/hornik-nn-1991.pdf
tomerg
<machine-learning><approximation><function>
Universal approximation theorem of second order
tomerg
# Universal approximation theorem of second order

(Asked by: username_0 on 2017-12-29)

The universal approximation theorem (<https://en.wikipedia.org/wiki/Universal_approximation_theorem>)
informally states that up to several conditions, any function can be approximated by a shallow neural network up to any degree.

I was wondering if there is a version of this theorem that states the same for the gradients of the functions, i.e, for any function there is a (shallow or not shallow) neural network that approximates it and its gradient approximates the gradient of the original function (up to any degree)?

**Tags:** machine-learning, approximation, function

> **Comments:**
> **username_1**: A possible approach: given a continuous, differentiable function $f:[0,1]^n \to \mathbb{R}$ and its gradient $\nabla f$, first construct a piecewise linear function $g:[0,1]^n \to \mathbb{R}$ such that $g(x) \approx f(x)$ and $\nabla g(x) \approx \nabla f(x)$ everywhere, then implement or approximate $g(x)$ with a neural network.  At least if $n=1$ I think that with a ReLU activation function a neural network can directly and exactly compute any desired continuous piecewise linear function (not just approximate it).  I don't know if that's helpful.  
> **username_0**: Actually, that's a good answer (https://arxiv.org/abs/1611.01491). I forgot to mention that I'm looking for smooth activation functions. So, ReLU is off the table. I actually solved it using http://zmjones.com/static/statistical-learning/hornik-nn-1991.pdf  

---

## 0 answers

