<p>I'm not sure if this offers you any help, but I'll give it a shot.</p>

<p>I work with certain programs that allow me to create virtual networks. What these programs have in common is that they fire neurons one by one. So if you have a perceptron with size <code>{input: 1, hidden: 5; output: 1}</code> it will activate the input, then it will activate the hidden neurons chronologically and then the output.</p>

<p>For more complex feed-forward networks, neurons still get activated one by one using <a href="https://en.wikipedia.org/wiki/Topological_sorting" rel="nofollow noreferrer">Topological sorting</a>. </p>

<p><hr>
But in circular neural networks, or networks which have connections going backward, neurons still have to be activated one by one. So say I have the following neural network.</p>

<p><a href="https://i.sstatic.net/RD2WR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RD2WR.png" alt="enter image description here"></a></p>

<p>So the order of activation in this neural network is <code>in, in, a, b, c, d, e, out</code>. But as you might see, there is an error pointing from <strong>d backwards to b</strong>. </p>

<p>So how can <code>b</code> be influended by <code>d</code> when b gets activated first? Because neurons <strong>store</strong> their activation. </p>

<p>So if I run this neural network the first time, the influence from <code>d</code> on <code>b</code> is 0, because <code>d</code> has no activation yet. But after this first activation, <code>d</code> gets a value. <strong>So next time you run the neural network, <code>b</code> will be influenced by the <em>previous</em> value of <code>d</code>!</strong></p>

<p>So arrows pointing backwards are also called memory connections; they make the previous activation values have influence on the next activation. </p>

<p>I hope this is somewhat clear O.O. <a href="https://en.wikipedia.org/wiki/Long_short-term_memory" rel="nofollow noreferrer">LSTM</a> networks use these connections.</p>

CC BY-SA 3.0
2017-03-26T19:01:53.880Z
2017-03-26T19:01:53.880Z
Thomas Wagenaar
<p>The Neural Engineering Framework uses <a href="https://nbviewer.jupyter.org/github/celiasmith/syde556/blob/master/SYDE%20556%20Lecture%205%20Dynamics.ipynb" rel="nofollow noreferrer">Principle 3</a> to describe these "circular" or "recurrent" connections as dynamical systems similar to those used in control theory. However, the only reason it's able to do this is it's focus on spiking neurons and modelling their dendrites as filtering, so I'm not sure how much this applies to your case, but I thought it might still be a helpful hint on a different approach if you're interested.</p>

CC BY-SA 3.0
2017-03-27T03:16:34.367Z
2017-03-27T03:16:34.367Z
Seanny123
<p>Since this question is in the Cognitive Science category, I'm assuming you're asking for ideas supported by neuroscience evidence, and not artificial neural networks. Naturally, this makes things much more theoretical and difficult to get everyone on the same page.</p>

<blockquote>
  <p>...so the C in an input series of A, B, C can be handled differently than the C in input series A, A, C.</p>
</blockquote>

<p>If I understand your example correctly, you are asking what the role of "circular" network connections is in processing series that vary over time, and with respect to the temporal context of the inputs. This is a very clever thing to ask and has many important applications to AI.</p>

<p>Conceptually, the brain does do this by having a mechanism to distinguish between inputs based on context. In your example, you are right in thinking that the C should be handled differently in ABC than it should in AAC. The ability to understand C in a way that is unique to the events leading up to it is a powerful one.</p>

<p>Contextual time series analysis is something that Jeff Hawkin's <a href="https://en.wikipedia.org/wiki/Hierarchical_temporal_memory" rel="nofollow noreferrer">Hierarchical Temporal Memory</a> (HTM) model of intelligence (based loosely on biologically inspired ideas) tries to accomplish, and it may be relevant to you. However, it does not incorporate feedback connections from higher (more abstract) regions to lower ones. Keep in mind that HTM still has its pitfalls in contextual time series analysis and so the actual way the brain does it is much more robust. I offer this merely as an anecdote.</p>

<p>From a biological perspective, you have to understand the word "circular" to mean the feedforward and feedback connections between neurons of different levels, as opposed to lateral connections between neurons within the same level. Neurons within the same level receive connections from the same inhibitory neurons (which are overwhelmingly local connections). You mentioned that neurons connect to themselves (N1 => N1), but this is not a very commonly studied thing in biology and probably has a role in error correction, boosting its own activity, or some other kind of activity modulation.</p>

<p>What seems to be happening in all parts of the brain, and even the retina, is that a level's excitatory neurons will send their feedforward signals to the next higher region's inhibitory and excitatory neurons. The inhibitory neurons will laterally inhibit the least active excitatory neurons within the same level. Some excitatory neurons will then send connections to the next higher region (feedforward) and the same process repeats. Other excitatory neurons will send connections to the lower region that connects to it (can consider these connections feedback), but will generally connect in a way to inhibit the excitatory neurons of the lower region by activating the inhibitory neurons (which work locally, thus only within that region).</p>

<p>With many more details excluded, this leads to a self-stabilizing framework where very little fine-tuning of arbitrary parameters is required (which apparently is a problem that you are well aware of in trying to find that balance). However, a circular network is just one substrate that is in use by the brain to understand context in time series. Unfortunately, circular connections actually have very little to do with contextual time series. Lateral connections seem to have much more to do with that than feedforward and feedback connections. With that said, there is much to be gained in combining the HTM's contextual time series analysis approach with a valid feedforward and feedback connection scheme between levels.</p>

<p>Feedforward connections mainly serve to make input signals much more abstract. Feedback connections serve as a way to fine-tune the lower levels and "tell" those levels which activations it is allowed to have.</p>

<p>If you want to learn more about circular connections and their role, I strongly recommend you study the retina. It is simpler than understanding the cortex and subcortical components, and we know much more about its cognitive processing than we do of the brain. Furthermore, the retina is actually an extension of the brain. It even comes complete with neurons that process signals in complex ways. You find the circular connections of which you speak in there, inhibitory/excitatory neurons, distinct hierarchical levels, and the higher cells even demonstrate characteristics of context-dependent input series on very small timescales.</p>

CC BY-SA 3.0
2017-03-28T04:55:16.467Z
This answer is very informative, but it would be better if it had a conclusion that summed up the answer overall at the end.
J.Todd
CC BY-SA 3.0
2017-03-29T19:15:18.520Z
Unforunately, there is no way to summarize further without going into details. I tried, and discovered I could continue talking on this topic until there is such a detailed picture as to recreate the entire brain (and that is where it would inevitably have to go, regardless of the fact that I couldn't even do that, hah), but that's impractical for the purposes of this website. Could you repost a more specific question describing your current setup and what problems you have with it? Or better yet, send me a message and we can brainstorm.
aiwyn
CC BY-SA 3.0
2017-04-27T12:05:03.997Z
I don't agree with your characterisation of network stability relying on feed-forward and feed-back connections. It's much more likely to rely on the balance between local recurrent excitation and inhibition within a network. The simpler structure of retinal networks probably makes them very different from cortical networks.
Dylan Richard Muir
CC BY-SA 3.0
2017-04-27T12:08:31.457Z
You also write in a very definite way about the role of local (lateral?) and inter-areal cortical connections in processing time series. I don't think what you wrote is backed up experimentally — please provide references if there's more to it than hand-waving by Jeff Hawkins.
Dylan Richard Muir
CC BY-SA 3.0
2017-04-30T22:25:55.357Z
I despise using FF/FB and higher/lower terminology unless I have to. There really aren't such things in biology, so my definitions might just differ from yours. We define those connections based on our faulty and abstract assumptions about the flow of "information". Also, as I already mentioned, the "hand-waving by Jeff Hawkins" was merely brought up anecdotally.
aiwyn
CC BY-SA 3.0
2017-04-30T22:27:32.467Z
My point is just that network stability cannot come from local connections alone because then competition couldn't arise between higher/lower regions. Lower networks would have a strong recruitment bias for neurons and ultimately arbitrary variables (e.g. defining the "size" of each middle network) would be needed to prevent this. Otherwise, no higher regions would be able to self-organize. FF/FB communication between these regions is a way that allows the regions to "decide" the proper size of each region. You just can't get that autonomous behavior from local connections.
aiwyn
CC BY-SA 3.0
2017-05-02T10:53:04.203Z
I disagree with everything you wrote in your comment above. Network stability absolutely can arise from local connections. I'm not sure what you're referring to regarding "competition between higher/lower regions". The size of regions of cortex is most likely defined by patterns of input coming from the thalamus, which is determined genetically (i.e. as a pre-defined value). We have no idea how the functional responses of cortical areas in the visual hierarchy are defined or if they self-organise, so I'm not sure how you can justify writing that.
Dylan Richard Muir
CC BY-SA 3.0
2017-05-04T09:19:03.873Z
Of course they self-organize. Simple watered-down example: how else can you explain the ability of a rodent's cortex to process visual input when the optical pathway is surgically rerouted to the cortical areas that originally processed other kinds of sensory information? And the size of a region is defined by patterns of input, but I disagree that the patterns are determined genetically in the thalamus. A much more plausible solution is that regions (or single cells) organize in a way to minimize information entropy. FYI, my meaning of "higher/lower" is like primary visual cortex's V2/V1.
aiwyn
CC BY-SA 3.0
2017-03-27T14:16:16.987Z
2017-03-27T14:27:48.310Z
2017-03-27T14:27:48.310Z
aiwyn
<p>In contrast to artificial neural networks, which are almost all feed-forward architectures, networks in biological brains are highly recurrent. In the networks of cortex, the majority of synaptic inputs a neuron receives come from nearby neurons in the same area of cortex (<a href="http://doi.org/10.1523/JNEUROSCI.1400-04.2004" rel="noreferrer">Binzegger et al. 2004</a>). These are the "circular" connections that you are referring to.</p>

<p>So recurrent connectivity is the rule, and simple feed-forward (or feed-back) projections between cohorts are the exception. The networks that result are therefore highly dynamic, and respond in complex ways to inputs changing over time. In general there is no one "reason" why recurrent connections exist. But there are guidelines about how connections between excitatory and inhibitory neurons are made.</p>

<p>The patterns of connections which define a network determine the computational properties of that network. At a first level, the strength and connectivity between excitatory and inhibitory neurons control the overall stability of the network (e.g. <a href="http://www.gatsby.ucl.ac.uk/~qhuys/theoretical_neuroscience/Wilson-Cowan73.pdf" rel="noreferrer">Wilson &amp; Cowan 1973</a>; <a href="http://www.jneurosci.org/cgi/pmidlookup?view=long&amp;pmid=9151754" rel="noreferrer">Tsodyks et al. 1997</a>). It's certainly possible to have self-connections and high recurrence, and still keep the network stable.</p>

<p>Secondly, changing the rules by which neurons connect to each other (<a href="http://doi.org/10.1038/nature12015" rel="noreferrer">Ko et al. 2011</a>; <a href="http://doi.org/10.1038/nature14182" rel="noreferrer">Cossell et al. 2015</a>; <a href="http://doi.org/10.1038/nature17192" rel="noreferrer">Lee et al. 2016</a>; <a href="http://doi.org/10.1016/j.neuron.2017.02.035" rel="noreferrer">Wilson et al. 2017</a>) changes the computational properties of the resulting network (<a href="http://doi.org/10.1103/PhysRevE.91.042808" rel="noreferrer">Muir &amp; Mrsic-Flogel 2015</a>).</p>

<p>In general, biological neural networks are highly recurrent dynamical systems which bear only passing relationship to the acyclic networks that we use to think about feed-forward and feed-back connections in the brain.</p>

CC BY-SA 3.0
2017-04-27T12:08:41.367Z
2017-04-29T11:53:55.533Z
2017-04-29T11:53:55.533Z
Dylan Richard Muir
<p>I'm developing small virtual neural networks and for the most part, the networks have remained feed-forward. When I create circular structures, it becomes a challenge to find a balance with the neural refactory period, stopping neurons from firing again for a period of time, giving them high action potential thresholds, then reducing that back down to fire again -it's tough to find a balance that stops infinite feedback loops but allows circular structures to play a role in processing. </p>

<p>I've done some reading on this and my vague idea so far is that circular structures perhaps allow the network to react to information based on its pattern, react to the information over time, so the C in an input series of A, B, C can be handled differently than the C in input series A, A, C. </p>

<p>Is that accurate, basically? What's the role of circular connection structures in a  neural network? I'm asking not only about multi-neuron circular structures like N1 => N2 => N3 => N1 but also about neurons that connect to themselves (N1 => N1) (which from my reading is something that happens if I understand correctly, but I'm unsure whether it achieves a similar affect as a larger scale circular structure).</p>

<p><sub> (=> represents a neural connection) </sub></p>

CC BY-SA 3.0
2017-03-27T11:18:55.830Z
How are the current answer unsatisfactory?
Seanny123
CC BY-SA 3.0
2017-03-27T19:07:05.247Z
@Seanny123 Your answer isn't complete and Thomas's answer is based on simulated networks: I pointed to simulated networks as the context for my question but I'm here asking about neuroscience, not the mechanics of a specific simulator.
J.Todd
CC BY-SA 3.0
2017-03-28T02:39:26.657Z
In your question, could you clarify what a satisfactory answer would contain? Specifically, by expanding on your definition of "accurate" in regards to you current mental model, as well as what level of biological detail you expect for the description of the "role of circular connection structures in a neural network". In other words, I like your question, I'm just not sure how to approach in a satisfactory manner and would appreciate assistance.
Seanny123
<theoretical-neuroscience><neural-network>
What role do circular network structures play in neural networks?
J.Todd
# What role do circular network structures play in neural networks?

(Asked by: username_0 on 2017-03-24)

I'm developing small virtual neural networks and for the most part, the networks have remained feed-forward. When I create circular structures, it becomes a challenge to find a balance with the neural refactory period, stopping neurons from firing again for a period of time, giving them high action potential thresholds, then reducing that back down to fire again -it's tough to find a balance that stops infinite feedback loops but allows circular structures to play a role in processing.

I've done some reading on this and my vague idea so far is that circular structures perhaps allow the network to react to information based on its pattern, react to the information over time, so the C in an input series of A, B, C can be handled differently than the C in input series A, A, C.

Is that accurate, basically? What's the role of circular connection structures in a neural network? I'm asking not only about multi-neuron circular structures like N1 => N2 => N3 => N1 but also about neurons that connect to themselves (N1 => N1) (which from my reading is something that happens if I understand correctly, but I'm unsure whether it achieves a similar affect as a larger scale circular structure).

(=> represents a neural connection)

**Tags:** theoretical-neuroscience, neural-network

> **Comments:**
> **username_1**: How are the current answer unsatisfactory?  
> **username_0**: @username_1 Your answer isn't complete and Thomas's answer is based on simulated networks: I pointed to simulated networks as the context for my question but I'm here asking about neuroscience, not the mechanics of a specific simulator.  
> **username_1**: In your question, could you clarify what a satisfactory answer would contain? Specifically, by expanding on your definition of "accurate" in regards to you current mental model, as well as what level of biological detail you expect for the description of the "role of circular connection structures in a neural network". In other words, I like your question, I'm just not sure how to approach in a satisfactory manner and would appreciate assistance.  

---

## 4 answers

### Answer by username_4 on 2017-04-27. Score: 8 | Accepted answer

In contrast to artificial neural networks, which are almost all feed-forward architectures, networks in biological brains are highly recurrent. In the networks of cortex, the majority of synaptic inputs a neuron receives come from nearby neurons in the same area of cortex ([Binzegger et al. 2004](http://doi.org/10.1523/JNEUROSCI.1400-04.2004)). These are the "circular" connections that you are referring to.

So recurrent connectivity is the rule, and simple feed-forward (or feed-back) projections between cohorts are the exception. The networks that result are therefore highly dynamic, and respond in complex ways to inputs changing over time. In general there is no one "reason" why recurrent connections exist. But there are guidelines about how connections between excitatory and inhibitory neurons are made.

The patterns of connections which define a network determine the computational properties of that network. At a first level, the strength and connectivity between excitatory and inhibitory neurons control the overall stability of the network (e.g. [Wilson & Cowan 1973](http://www.gatsby.ucl.ac.uk/~qhuys/theoretical_neuroscience/Wilson-Cowan73.pdf); [Tsodyks et al. 1997](http://www.jneurosci.org/cgi/pmidlookup?view=long&pmid=9151754)). It's certainly possible to have self-connections and high recurrence, and still keep the network stable.

Secondly, changing the rules by which neurons connect to each other ([Ko et al. 2011](http://doi.org/10.1038/nature12015); [Cossell et al. 2015](http://doi.org/10.1038/nature14182); [Lee et al. 2016](http://doi.org/10.1038/nature17192); [Wilson et al. 2017](http://doi.org/10.1016/j.neuron.2017.02.035)) changes the computational properties of the resulting network ([Muir & Mrsic-Flogel 2015](http://doi.org/10.1103/PhysRevE.91.042808)).

In general, biological neural networks are highly recurrent dynamical systems which bear only passing relationship to the acyclic networks that we use to think about feed-forward and feed-back connections in the brain.


---

### Answer by username_2 on 2017-03-26. Score: 3

I'm not sure if this offers you any help, but I'll give it a shot.

I work with certain programs that allow me to create virtual networks. What these programs have in common is that they fire neurons one by one. So if you have a perceptron with size `{input: 1, hidden: 5; output: 1}` it will activate the input, then it will activate the hidden neurons chronologically and then the output.

For more complex feed-forward networks, neurons still get activated one by one using [Topological sorting](https://en.wikipedia.org/wiki/Topological_sorting).

---

But in circular neural networks, or networks which have connections going backward, neurons still have to be activated one by one. So say I have the following neural network.

[![enter image description here](https://i.sstatic.net/RD2WR.png)](https://i.sstatic.net/RD2WR.png)

So the order of activation in this neural network is `in, in, a, b, c, d, e, out`. But as you might see, there is an error pointing from **d backwards to b**.

So how can `b` be influended by `d` when b gets activated first? Because neurons **store** their activation.

So if I run this neural network the first time, the influence from `d` on `b` is 0, because `d` has no activation yet. But after this first activation, `d` gets a value. **So next time you run the neural network, `b` will be influenced by the *previous* value of `d`!**

So arrows pointing backwards are also called memory connections; they make the previous activation values have influence on the next activation.

I hope this is somewhat clear O.O. [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory) networks use these connections.


---

### Answer by username_3 on 2017-03-27. Score: 1

Since this question is in the Cognitive Science category, I'm assuming you're asking for ideas supported by neuroscience evidence, and not artificial neural networks. Naturally, this makes things much more theoretical and difficult to get everyone on the same page.

> ...so the C in an input series of A, B, C can be handled differently than the C in input series A, A, C.

If I understand your example correctly, you are asking what the role of "circular" network connections is in processing series that vary over time, and with respect to the temporal context of the inputs. This is a very clever thing to ask and has many important applications to AI.

Conceptually, the brain does do this by having a mechanism to distinguish between inputs based on context. In your example, you are right in thinking that the C should be handled differently in ABC than it should in AAC. The ability to understand C in a way that is unique to the events leading up to it is a powerful one.

Contextual time series analysis is something that Jeff Hawkin's [Hierarchical Temporal Memory](https://en.wikipedia.org/wiki/Hierarchical_temporal_memory) (HTM) model of intelligence (based loosely on biologically inspired ideas) tries to accomplish, and it may be relevant to you. However, it does not incorporate feedback connections from higher (more abstract) regions to lower ones. Keep in mind that HTM still has its pitfalls in contextual time series analysis and so the actual way the brain does it is much more robust. I offer this merely as an anecdote.

From a biological perspective, you have to understand the word "circular" to mean the feedforward and feedback connections between neurons of different levels, as opposed to lateral connections between neurons within the same level. Neurons within the same level receive connections from the same inhibitory neurons (which are overwhelmingly local connections). You mentioned that neurons connect to themselves (N1 => N1), but this is not a very commonly studied thing in biology and probably has a role in error correction, boosting its own activity, or some other kind of activity modulation.

What seems to be happening in all parts of the brain, and even the retina, is that a level's excitatory neurons will send their feedforward signals to the next higher region's inhibitory and excitatory neurons. The inhibitory neurons will laterally inhibit the least active excitatory neurons within the same level. Some excitatory neurons will then send connections to the next higher region (feedforward) and the same process repeats. Other excitatory neurons will send connections to the lower region that connects to it (can consider these connections feedback), but will generally connect in a way to inhibit the excitatory neurons of the lower region by activating the inhibitory neurons (which work locally, thus only within that region).

With many more details excluded, this leads to a self-stabilizing framework where very little fine-tuning of arbitrary parameters is required (which apparently is a problem that you are well aware of in trying to find that balance). However, a circular network is just one substrate that is in use by the brain to understand context in time series. Unfortunately, circular connections actually have very little to do with contextual time series. Lateral connections seem to have much more to do with that than feedforward and feedback connections. With that said, there is much to be gained in combining the HTM's contextual time series analysis approach with a valid feedforward and feedback connection scheme between levels.

Feedforward connections mainly serve to make input signals much more abstract. Feedback connections serve as a way to fine-tune the lower levels and "tell" those levels which activations it is allowed to have.

If you want to learn more about circular connections and their role, I strongly recommend you study the retina. It is simpler than understanding the cortex and subcortical components, and we know much more about its cognitive processing than we do of the brain. Furthermore, the retina is actually an extension of the brain. It even comes complete with neurons that process signals in complex ways. You find the circular connections of which you speak in there, inhibitory/excitatory neurons, distinct hierarchical levels, and the higher cells even demonstrate characteristics of context-dependent input series on very small timescales.

> **Comments:**
> **username_0**: This answer is very informative, but it would be better if it had a conclusion that summed up the answer overall at the end.  
> **username_3**: Unforunately, there is no way to summarize further without going into details. I tried, and discovered I could continue talking on this topic until there is such a detailed picture as to recreate the entire brain (and that is where it would inevitably have to go, regardless of the fact that I couldn't even do that, hah), but that's impractical for the purposes of this website. Could you repost a more specific question describing your current setup and what problems you have with it? Or better yet, send me a message and we can brainstorm.  
> **username_4**: I don't agree with your characterisation of network stability relying on feed-forward and feed-back connections. It's much more likely to rely on the balance between local recurrent excitation and inhibition within a network. The simpler structure of retinal networks probably makes them very different from cortical networks.  
> **username_4**: You also write in a very definite way about the role of local (lateral?) and inter-areal cortical connections in processing time series. I don't think what you wrote is backed up experimentally — please provide references if there's more to it than hand-waving by Jeff Hawkins.  
> **username_3**: I despise using FF/FB and higher/lower terminology unless I have to. There really aren't such things in biology, so my definitions might just differ from yours. We define those connections based on our faulty and abstract assumptions about the flow of "information". Also, as I already mentioned, the "hand-waving by Jeff Hawkins" was merely brought up anecdotally.  
> **username_3**: My point is just that network stability cannot come from local connections alone because then competition couldn't arise between higher/lower regions. Lower networks would have a strong recruitment bias for neurons and ultimately arbitrary variables (e.g. defining the "size" of each middle network) would be needed to prevent this. Otherwise, no higher regions would be able to self-organize. FF/FB communication between these regions is a way that allows the regions to "decide" the proper size of each region. You just can't get that autonomous behavior from local connections.  
> **username_4**: I disagree with everything you wrote in your comment above. Network stability absolutely can arise from local connections. I'm not sure what you're referring to regarding "competition between higher/lower regions". The size of regions of cortex is most likely defined by patterns of input coming from the thalamus, which is determined genetically (i.e. as a pre-defined value). We have no idea how the functional responses of cortical areas in the visual hierarchy are defined or if they self-organise, so I'm not sure how you can justify writing that.  
> **username_3**: Of course they self-organize. Simple watered-down example: how else can you explain the ability of a rodent's cortex to process visual input when the optical pathway is surgically rerouted to the cortical areas that originally processed other kinds of sensory information? And the size of a region is defined by patterns of input, but I disagree that the patterns are determined genetically in the thalamus. A much more plausible solution is that regions (or single cells) organize in a way to minimize information entropy. FYI, my meaning of "higher/lower" is like primary visual cortex's V2/V1.  

---

### Answer by username_1 on 2017-03-27. Score: -1

The Neural Engineering Framework uses [Principle 3](https://nbviewer.jupyter.org/github/celiasmith/syde556/blob/master/SYDE%20556%20Lecture%205%20Dynamics.ipynb) to describe these "circular" or "recurrent" connections as dynamical systems similar to those used in control theory. However, the only reason it's able to do this is it's focus on spiking neurons and modelling their dendrites as filtering, so I'm not sure how much this applies to your case, but I thought it might still be a helpful hint on a different approach if you're interested.


---

