<p>The most cited Universal Approximation Theories for multi-layer feedforward neural networks by Cybenko (1989) and Hornik (1991) assume the activation functions of the network to be sigmoidal. However, currently one mainly uses ReLU activation functions (or variants) which are not sigmoidal (not even bounded). </p>

<p>I found a proof of the Approximation Theory summarized in <a href="http://www2.math.technion.ac.il/~pinkus/papers/acta.pdf" rel="nofollow noreferrer">Pinkus: Approximation theory of the MLP model in neural networks (1999)</a>, where he shows the approximation result for continuous activation functions $\sigma \in C(\mathbb{R})$ if $\sigma$ is not polynomial. Unfortunately, he only does this for the case of one hidden layer. I am unable to find a proof that also covers multi-hidden-layer neural networks with continuous activation functions that are not sigmoidal.</p>

<p>Does anyone know of such a result? Or is it easy to extend the result by Pinkus mentioned above to a network with more hidden layers?</p>

CC BY-SA 3.0
2017-06-29T16:11:11.773Z
Doesn't the single layer proof imply the multi-layer one? Just set layers 1 up to n-1 to compute the identity function and use the single-layer proof to approximate the function using the final hidden layer.
user4242
CC BY-SA 3.0
2017-06-29T18:37:53.507Z
I thought that would not be possible if the activation functions in every layer have to be the same. But in the case of the ReLU this might be relatively easy if we want to prove uniformly dense on compacta (in $C(\mathbb{R}^n)$). For each compact $K \subset \mathbb{R}^n$ we can choose the biases such that the weighted input of the neurons in layers $1,\ldots, n-1$ are in the linear part of the ReLU activation function (>0). This settles it for the ReLU, but does not work for general continuous activation functions. Am I correct?
Robo
CC BY-SA 3.0
2017-06-29T19:11:23.377Z
@Robo Doesn't Theorem 2.4 in our paper, https://eccc.weizmann.ac.il/report/2017/098/ give what you want?
Student
CC BY-SA 3.0
2017-06-29T20:19:33.713Z
@Anirbit It does cover it for the case of the ReLU deep neural networks I think. In proving Corollary 2.2, do you use that you can represent any affine linear function (you mean an affine function of the form $A\cdot x + b$?) $l_n$ with a ReLU DNN before you apply the lemma's in the appendix? Thanks! Anyway, this only proves the result for ReLU activation functions and not for general continuous activation functions.
Robo
CC BY-SA 3.0
2017-07-03T02:21:36.457Z
I am a bit confused your question. Corollary 2.2 *is* the proof that every R^n -> R continuous piecewise linear function is representable by a log depth ReLU DNN. Theorem A.2, A.3 and Theorem 2.1 all go into its proof.
Student
CC BY-SA 3.0
2017-07-04T07:59:27.187Z
What I meant is that both theorems in the Appendix already assume functions that can be approximated by ReLU DNN's.
Robo
<machine-learning><lg.learning><ne.neural-evol>
Universal Approximation Theorem for non-sigmoidal activation functions
Robo
# Universal Approximation Theorem for non-sigmoidal activation functions

(Asked by: username_0 on 2017-06-29)

The most cited Universal Approximation Theories for multi-layer feedforward neural networks by Cybenko (1989) and Hornik (1991) assume the activation functions of the network to be sigmoidal. However, currently one mainly uses ReLU activation functions (or variants) which are not sigmoidal (not even bounded).

I found a proof of the Approximation Theory summarized in [Pinkus: Approximation theory of the MLP model in neural networks (1999)](http://www2.math.technion.ac.il/~pinkus/papers/acta.pdf), where he shows the approximation result for continuous activation functions $\sigma \in C(\mathbb{R})$ if $\sigma$ is not polynomial. Unfortunately, he only does this for the case of one hidden layer. I am unable to find a proof that also covers multi-hidden-layer neural networks with continuous activation functions that are not sigmoidal.

Does anyone know of such a result? Or is it easy to extend the result by Pinkus mentioned above to a network with more hidden layers?

**Tags:** machine-learning, lg.learning, ne.neural-evol
> **Comments:**
> **username_1**: Doesn't the single layer proof imply the multi-layer one? Just set layers 1 up to n-1 to compute the identity function and use the single-layer proof to approximate the function using the final hidden layer.  
> **username_0**: I thought that would not be possible if the activation functions in every layer have to be the same. But in the case of the ReLU this might be relatively easy if we want to prove uniformly dense on compacta (in $C(\mathbb{R}^n)$). For each compact $K \subset \mathbb{R}^n$ we can choose the biases such that the weighted input of the neurons in layers $1,\ldots, n-1$ are in the linear part of the ReLU activation function (>0). This settles it for the ReLU, but does not work for general continuous activation functions. Am I correct?  
> **username_2**: @Robo Doesn't Theorem 2.4 in our paper, https://eccc.weizmann.ac.il/report/2017/098/ give what you want?  
> **username_0**: @Anirbit It does cover it for the case of the ReLU deep neural networks I think. In proving Corollary 2.2, do you use that you can represent any affine linear function (you mean an affine function of the form $A\cdot x + b$?) $l_n$ with a ReLU DNN before you apply the lemma's in the appendix? Thanks! Anyway, this only proves the result for ReLU activation functions and not for general continuous activation functions.  
> **username_2**: I am a bit confused your question. Corollary 2.2 *is* the proof that every R^n -> R continuous piecewise linear function is representable by a log depth ReLU DNN. Theorem A.2, A.3 and Theorem 2.1 all go into its proof.  
> **username_0**: What I meant is that both theorems in the Appendix already assume functions that can be approximated by ReLU DNN's.  

---

## 0 answers

