<p>First, I want to clarify a few things. Deep Learning simply refers to any learning on a deep (more than one hidden layer) neural network, where the learning happens (i.e. parameters are adjusted) at all layers in the network. Therefore, deep learning algorithms can span the gamut between quite biologically plausible to very biologically implausible, and can include supervised and unsupervised (and maybe even reinforcement-driven) algorithms. That said, the algorithms that are currently very successful in deep learning (e.g. backprop on a convnet), are not very bio-plausible.</p>

<p>As pointed out in the comments, bio-plausibility is difficult to quantify. What is helpful is pointing out what aspects of an algorithm are or are not bio-plausible, so that's what I'll try to do here.</p>

<p>One mark in favour of RBMs is that the error signals are local. The objective function for an RBM is completely defined by the two layers of the RBM, and this holds even when the RBM is embedded in a larger network.</p>

<p>However, I am reluctant to say that the weight update of an RBM is completely local, at least in their typical formulation. RBMs are typically trained using Contrastive Divergence (CD) [1]. After the forward pass (computing the hidden node activations from the inputs), CD requires recomputing the input node activations from the hiddens, and then recomputing the hidden node activations from these new input node activations. (This can happen many more times depending on the particular variant of CD, but has to happen at least once.) This is called Gibbs sampling, and it is unclear to me how this could happen in biology.</p>

<p>Another mark against RBMs is that they have tied weights: the feedback weights are the transpose of the feedforward weights. Some autoencoders avoid this problem by having untied weights, though this can also make them more difficult to train (the increased flexibility leads to more local minima).</p>

<p>A Deep Belief Network (DBN) is initialized by stacking a bunch of RBMs together. One can think of the first RBM as encoding the inputs (i.e. finding a probability distribution over the inputs), the second RBM as encoding the encoding created by the first, the third RBM encoding this second encoding, and so on. Creating the DBN sometimes involves "untying" the weights, so that there are now separate feedforward and feedback weights.</p>

<p>Once constructed, DBNs can be fine-tuned using a number of algorithms, some more bio-plausible than others. Here, fine tuning simply refers to adjusting all the parameters of the DBN together to minimize some global objective function, rather than using the local objective functions for each RBM (as was done when creating the network). One fine-tuning algorithm is the wake-sleep algorithm [1], an unsupervised learning algorithm that has two distinct phases [2]: A "wake" phase, where the network is run forward, and the feedback weights are adjusted based on these feedforward activations; A sleep phase, where the network is run in reverse (generatively), and the feedforward weights are adjusted based on these feedback activations. This algorithm is reasonably bio-plausible, since the weight updates can be determined locally, and feedforward and feedback weights are separate and are adjusted separately. Some of the papers on the wake-sleep algorithm suggest that it might map onto and explain the wake-sleep cycles of animals, however I do not know of any studies that provide evidence of this.</p>

<p>DBNs can also be fine-tuned using supervised learning algorithms, namely backprop. This of course brings all the bio-plausibility concerns associated with backprop.</p>

<p>So to summarize, and address your overall question, which I interpret as "Are RBMs and DBNs more bio-plausible than deep networks trained with backprop?": It's going to depend on how exactly they're trained. RBMs run synapses backwards just like backprop does, but propagating things backwards through unidirectional synapses is only one of the bio-plausibility concerns associated with backprop. Others include that backprop requires the derivatives of the hidden units at their activation points, and that backprop uses continuous values, not spikes (for a fuller list, see [3]). RBMs avoid these two concerns, however they add additional concerns, like how Gibbs sampling might happen in biology. Which one is "more" or "less" biologically plausible depends on which of these problems might be more easily solved in the brain. With regards to DBNs, I would say that the wake-sleep algorithm is more biologically plausible than backprop, in that the mechanisms in the brain seem more tailored to implement the wake-sleep algorithm than backprop.</p>

<p>That said, could a DBN trained with wake-sleep and a shallow classifier on top achieve the same results as a convnet trained with backprop, on a task like ILSVRC-2012 (ImageNet)? Certainly not with our current computational power and the current machine learning knowledge and tricks. And even though I don't expect a good bio-plausible learning algorithm to scale to large datasets right out of the box, it is important to keep in mind that whatever algorithms we want to attribute to the brain, they're going to have to scale.</p>

<p>[1] <a href="https://www.ncbi.nlm.nih.gov/pubmed/16764513" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/pubmed/16764513</a>
[2] <a href="https://en.wikipedia.org/wiki/Wake-sleep_algorithm" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Wake-sleep_algorithm</a>
[3] <a href="https://arxiv.org/abs/1502.04156v3" rel="nofollow noreferrer">https://arxiv.org/abs/1502.04156v3</a></p>

CC BY-SA 4.0
2019-12-17T12:55:53.897Z
> This is called Gibbs sampling, and it is unclear to me how this could happen in biology. /> Why do you see this as a problem? Many (some would say most) neuronal populations have some measure of recurrent connectivity that would support recurrent activity.
Paul Brodersen
CC BY-SA 4.0
2019-12-17T16:26:43.373Z
Definitely there is recurrent connectivity (I believe a number of studies have even shown more feedback connectivity in visual cortex than feedforward connectivity). One aspect of CD that I didn't elaborate on above is that to do the weight update, you need both the activations from the first pass through the network, and those from after you've done Gibbs sampling. So real neurons would somehow have to incorporate both those pairs of activations. I don't think this is impossible, but to make the case that the brain is using CD, the neural mechanisms for this would need to be explained.
hunse
CC BY-SA 4.0
2019-12-18T10:26:37.490Z
I know that the original RBM papers (in general) stop on the forward pass but training with CD-1 also works if you stop after the first backward pass.
Paul Brodersen
CC BY-SA 3.0
2016-11-21T17:00:27.363Z
2016-11-21T17:27:03.903Z
2016-11-21T17:27:03.903Z
hunse
<p>How <a href="https://cogsci.stackexchange.com/q/8923/4397">biologically plausible</a> are Restricted Bolztmann Machines (RBMs) and their stacked equivalent Deep Belief Networks (DBNs)? I know that Deep Learning (DL) in general isn't considered biologically plausible, since it requires information to be propagated backwards through uni-directional synapses. However, from what I understand, DBNs don't use back-prop and instead uses some sort of unsupervised learning algorithm. Does that make them more biologically plausible? </p>

CC BY-SA 3.0
2016-11-18T10:41:28.810Z
How would you grade the "degree" of biological plausibility? All deep learning is somewhat plausible, since they capture hierarchical structure in feature learning. Generally what people wonder about is whether the learning is biologically plausible. RBM's use local learning rules, rather than backprop.

Taylor, P., et al. (2015). The global landscape of cognition: hierarchical aggregation as an organizational principle of human cortical networks and functions. Scientific Reports, 5(November), 18112.

Bengio et al. (2015) https://arxiv.org/abs/1502.04156
Abraham Nunes
CC BY-SA 3.0
2016-11-21T01:28:21.750Z
I'm measuring the "degree" of biological plausibility based of my answer to [this question](http://cogsci.stackexchange.com/q/8923/4397). So, yes, I was wondering if the learning-rule was local to each neuron and did not require global knowledge or anything else that would be impossible for biological neurons.
Seanny123
CC BY-SA 3.0
2016-11-21T16:14:02.997Z
One strike against RBMs is that they use tied weights: the feedback weights are the transpose of the feedforward weights. Autoencoders (though not all autoencoders) can avoid this problem by having untied weights, though this can also make them more difficult to train (the increased flexibility leads to more local minima).
hunse
<theoretical-neuroscience><neural-network>
Biological plausibility of RBMs
Seanny123
# Biological plausibility of RBMs

(Asked by: username_0 on 2016-11-17)

How [biologically plausible](https://cogsci.stackexchange.com/q/8923/4397) are Restricted Bolztmann Machines (RBMs) and their stacked equivalent Deep Belief Networks (DBNs)? I know that Deep Learning (DL) in general isn't considered biologically plausible, since it requires information to be propagated backwards through uni-directional synapses. However, from what I understand, DBNs don't use back-prop and instead uses some sort of unsupervised learning algorithm. Does that make them more biologically plausible?

**Tags:** theoretical-neuroscience, neural-network

> **Comments:**
> **username_1**: How would you grade the "degree" of biological plausibility? All deep learning is somewhat plausible, since they capture hierarchical structure in feature learning. Generally what people wonder about is whether the learning is biologically plausible. RBM's use local learning rules, rather than backprop.  Taylor, P., et al. (2015). The global landscape of cognition: hierarchical aggregation as an organizational principle of human cortical networks and functions. Scientific Reports, 5(November), 18112.  Bengio et al. (2015) https://arxiv.org/abs/1502.04156  
> **username_0**: I'm measuring the "degree" of biological plausibility based of my answer to [this question](http://cogsci.stackexchange.com/q/8923/4397). So, yes, I was wondering if the learning-rule was local to each neuron and did not require global knowledge or anything else that would be impossible for biological neurons.  
> **username_2**: One strike against RBMs is that they use tied weights: the feedback weights are the transpose of the feedforward weights. Autoencoders (though not all autoencoders) can avoid this problem by having untied weights, though this can also make them more difficult to train (the increased flexibility leads to more local minima).  

---

## 1 answers

### Answer by username_2 on 2016-11-21. Score: 4 | Accepted answer

First, I want to clarify a few things. Deep Learning simply refers to any learning on a deep (more than one hidden layer) neural network, where the learning happens (i.e. parameters are adjusted) at all layers in the network. Therefore, deep learning algorithms can span the gamut between quite biologically plausible to very biologically implausible, and can include supervised and unsupervised (and maybe even reinforcement-driven) algorithms. That said, the algorithms that are currently very successful in deep learning (e.g. backprop on a convnet), are not very bio-plausible.

As pointed out in the comments, bio-plausibility is difficult to quantify. What is helpful is pointing out what aspects of an algorithm are or are not bio-plausible, so that's what I'll try to do here.

One mark in favour of RBMs is that the error signals are local. The objective function for an RBM is completely defined by the two layers of the RBM, and this holds even when the RBM is embedded in a larger network.

However, I am reluctant to say that the weight update of an RBM is completely local, at least in their typical formulation. RBMs are typically trained using Contrastive Divergence (CD) [1]. After the forward pass (computing the hidden node activations from the inputs), CD requires recomputing the input node activations from the hiddens, and then recomputing the hidden node activations from these new input node activations. (This can happen many more times depending on the particular variant of CD, but has to happen at least once.) This is called Gibbs sampling, and it is unclear to me how this could happen in biology.

Another mark against RBMs is that they have tied weights: the feedback weights are the transpose of the feedforward weights. Some autoencoders avoid this problem by having untied weights, though this can also make them more difficult to train (the increased flexibility leads to more local minima).

A Deep Belief Network (DBN) is initialized by stacking a bunch of RBMs together. One can think of the first RBM as encoding the inputs (i.e. finding a probability distribution over the inputs), the second RBM as encoding the encoding created by the first, the third RBM encoding this second encoding, and so on. Creating the DBN sometimes involves "untying" the weights, so that there are now separate feedforward and feedback weights.

Once constructed, DBNs can be fine-tuned using a number of algorithms, some more bio-plausible than others. Here, fine tuning simply refers to adjusting all the parameters of the DBN together to minimize some global objective function, rather than using the local objective functions for each RBM (as was done when creating the network). One fine-tuning algorithm is the wake-sleep algorithm [1], an unsupervised learning algorithm that has two distinct phases [2]: A "wake" phase, where the network is run forward, and the feedback weights are adjusted based on these feedforward activations; A sleep phase, where the network is run in reverse (generatively), and the feedforward weights are adjusted based on these feedback activations. This algorithm is reasonably bio-plausible, since the weight updates can be determined locally, and feedforward and feedback weights are separate and are adjusted separately. Some of the papers on the wake-sleep algorithm suggest that it might map onto and explain the wake-sleep cycles of animals, however I do not know of any studies that provide evidence of this.

DBNs can also be fine-tuned using supervised learning algorithms, namely backprop. This of course brings all the bio-plausibility concerns associated with backprop.

So to summarize, and address your overall question, which I interpret as "Are RBMs and DBNs more bio-plausible than deep networks trained with backprop?": It's going to depend on how exactly they're trained. RBMs run synapses backwards just like backprop does, but propagating things backwards through unidirectional synapses is only one of the bio-plausibility concerns associated with backprop. Others include that backprop requires the derivatives of the hidden units at their activation points, and that backprop uses continuous values, not spikes (for a fuller list, see [3]). RBMs avoid these two concerns, however they add additional concerns, like how Gibbs sampling might happen in biology. Which one is "more" or "less" biologically plausible depends on which of these problems might be more easily solved in the brain. With regards to DBNs, I would say that the wake-sleep algorithm is more biologically plausible than backprop, in that the mechanisms in the brain seem more tailored to implement the wake-sleep algorithm than backprop.

That said, could a DBN trained with wake-sleep and a shallow classifier on top achieve the same results as a convnet trained with backprop, on a task like ILSVRC-2012 (ImageNet)? Certainly not with our current computational power and the current machine learning knowledge and tricks. And even though I don't expect a good bio-plausible learning algorithm to scale to large datasets right out of the box, it is important to keep in mind that whatever algorithms we want to attribute to the brain, they're going to have to scale.

[1] <https://www.ncbi.nlm.nih.gov/pubmed/16764513>
[2] <https://en.wikipedia.org/wiki/Wake-sleep_algorithm>
[3] <https://arxiv.org/abs/1502.04156v3>

> **Comments:**
> **username_3**: > This is called Gibbs sampling, and it is unclear to me how this could happen in biology. /> Why do you see this as a problem? Many (some would say most) neuronal populations have some measure of recurrent connectivity that would support recurrent activity.  
> **username_2**: Definitely there is recurrent connectivity (I believe a number of studies have even shown more feedback connectivity in visual cortex than feedforward connectivity). One aspect of CD that I didn't elaborate on above is that to do the weight update, you need both the activations from the first pass through the network, and those from after you've done Gibbs sampling. So real neurons would somehow have to incorporate both those pairs of activations. I don't think this is impossible, but to make the case that the brain is using CD, the neural mechanisms for this would need to be explained.  
> **username_3**: I know that the original RBM papers (in general) stop on the forward pass but training with CD-1 also works if you stop after the first backward pass.  

---

