<p>Sigmoid functions have clear probabilistic interpretation, so one can derive optimal learning algorithms for them within the bayesian approach. Moreover, they are differentiable, so the gradient descent can be directly applicable.
However, in general, neural networks are "universal approximators" in almost the same sense as, for example, polynomials. They can capture a limited set of regularities. Different activation functions (like different basis functions in functional approximation) will capture different regularities and will have different biases in approximating other regularities.
Thus, what activation functions are better depends on what underlying regularities are presented in data. Indicator functions can surely be better for some tasks. However, no generally applicable (task-independent) theory advocating the use of indicator functions is possible.</p>

CC BY-SA 3.0
2014-01-21T13:30:17.827Z
thanks! 1) would you know of any teaching material / papers giving the different regularities that various activation functions are good at capturing? 2) by regularities, do you mean properties of the target function, or of the underlying probability distribution, or of the sampling (ie training set)? Eg for sampling, [Jarrett et al](http://cs.nyu.edu/~koray/publis/jarrett-iccv-09.pdf) notice that $\mid$tanh(x)$\mid$ prevents overfitting in their image recognition task, would be nice to know why
Alex Dalyac
CC BY-SA 3.0
2014-01-21T14:06:35.837Z
1) Unfortunately, I have not encountered a detailed analysis of this problem. However, we have one paper on a related subject, but it might not be what you want, since it considers somewhat exotic networks. Nevertheless, here is the 
[link](http://link.springer.com/chapter/10.1007%2F978-3-642-33409-2_8)
I hope, it will not be absolutely useless.
Necro0x0Der
CC BY-SA 3.0
2014-01-21T14:21:16.853Z
2) Both. Learning methods can express some regularities (computable function), and can approximate others. E.g. polinomial basis can represent only polynomial regularities, but you can approximate sine function with it. Data can be produced by a source, which "true" model is arbitrary (stochastic) algorithm. The question is how precisely regularities expressible by a method match regularities in data.
Necro0x0Der
CC BY-SA 3.0
2014-01-21T11:39:16.030Z
2014-01-21T11:39:16.030Z
Necro0x0Der
<p>Does there exist any theory (other than Cybenko's proof of the Universal Approximation Theorem with sigmoids) advocating the use of indicator functions as transfer functions for machine learning with neural networks?</p>

<p>After having read matus's beautiful answer in <a href="https://cstheory.stackexchange.com/questions/17545/universal-approximation-theorem-neural-networks/17630#17630?newreg=bb92f636fd6443e9bc21a23c5924808b">this thread</a> explaining (among other things) Cybenko's proof, I wonder: if it weakens the approximation to use sigmoid transfer functions instead of indicator functions, what are the theoretical reasons for not using indicator functions?</p>

<p>As suggested <a href="https://cstheory.stackexchange.com/questions/20657/generalization-issues-with-practical-suggestions-from-universal-approximation-th">here</a>, perhaps it's because indicator functions have negative implications for generalization. </p>

<p>However, indicator functions are computationally far cheaper to implement than sigmoid functions, and also more closely resemble biological neural networks (ie the brain). Therefore, does there exist any other theory advocating the use of indicator functions as transfer functions for machine learning with neural networks?</p>

CC BY-SA 3.0
2015-02-19T08:55:02.503Z
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes.
Jan Johannsen
<machine-learning><ne.neural-evol>
Using Indicator Functions as Transfer Functions for Neural Networks
Alex Dalyac
# Using Indicator Functions as Transfer Functions for Neural Networks

(Asked by: username_0 on 2014-01-20)

Does there exist any theory (other than Cybenko's proof of the Universal Approximation Theorem with sigmoids) advocating the use of indicator functions as transfer functions for machine learning with neural networks?

After having read matus's beautiful answer in [this thread](https://cstheory.stackexchange.com/questions/17545/universal-approximation-theorem-neural-networks/17630#17630?newreg=bb92f636fd6443e9bc21a23c5924808b) explaining (among other things) Cybenko's proof, I wonder: if it weakens the approximation to use sigmoid transfer functions instead of indicator functions, what are the theoretical reasons for not using indicator functions?

As suggested [here](https://cstheory.stackexchange.com/questions/20657/generalization-issues-with-practical-suggestions-from-universal-approximation-th), perhaps it's because indicator functions have negative implications for generalization.

However, indicator functions are computationally far cheaper to implement than sigmoid functions, and also more closely resemble biological neural networks (ie the brain). Therefore, does there exist any other theory advocating the use of indicator functions as transfer functions for machine learning with neural networks?

**Tags:** machine-learning, ne.neural-evol

> **Comments:**
> **username_1**: While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes.  

---

## 1 answers

### Answer by username_2 on 2014-01-21. Score: 2 | Accepted answer

Sigmoid functions have clear probabilistic interpretation, so one can derive optimal learning algorithms for them within the bayesian approach. Moreover, they are differentiable, so the gradient descent can be directly applicable.
However, in general, neural networks are "universal approximators" in almost the same sense as, for example, polynomials. They can capture a limited set of regularities. Different activation functions (like different basis functions in functional approximation) will capture different regularities and will have different biases in approximating other regularities.
Thus, what activation functions are better depends on what underlying regularities are presented in data. Indicator functions can surely be better for some tasks. However, no generally applicable (task-independent) theory advocating the use of indicator functions is possible.

> **Comments:**
> **username_0**: thanks! 1) would you know of any teaching material / papers giving the different regularities that various activation functions are good at capturing? 2) by regularities, do you mean properties of the target function, or of the underlying probability distribution, or of the sampling (ie training set)? Eg for sampling, [Jarrett et al](http://cs.nyu.edu/~koray/publis/jarrett-iccv-09.pdf) notice that $\mid$tanh(x)$\mid$ prevents overfitting in their image recognition task, would be nice to know why  
> **username_2**: 1) Unfortunately, I have not encountered a detailed analysis of this problem. However, we have one paper on a related subject, but it might not be what you want, since it considers somewhat exotic networks. Nevertheless, here is the  [link](http://link.springer.com/chapter/10.1007%2F978-3-642-33409-2_8) I hope, it will not be absolutely useless.  
> **username_2**: 2) Both. Learning methods can express some regularities (computable function), and can approximate others. E.g. polinomial basis can represent only polynomial regularities, but you can approximate sine function with it. Data can be produced by a source, which "true" model is arbitrary (stochastic) algorithm. The question is how precisely regularities expressible by a method match regularities in data.  

---

