<p>As mentioned in the definitions section on page 2 of the paper, $D(q||p) = &lt;ln(q/p)&gt;_q$ is the Kullback-Leibler divergence or cross-entropy between two densities.</p>

<p>The reason for the negative logs is that is the convention when discussing entropy in the context of information theory.  This allows information to be combined additively. (<a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Rationale" rel="nofollow">https://en.wikipedia.org/wiki/Entropy_(information_theory)#Rationale</a>)</p>

CC BY-SA 3.0
2015-11-18T14:36:51.060Z
i picked this article up as a result of this question and plan to give a more complete explanation of the equations soon
honi
CC BY-SA 3.0
2015-11-19T16:25:11.483Z
Great! I eagerly await your take on what Friston is saying here. i find his work very interesting, but due to my lack of math skills, I cannot grasp the details and do not know how to evaluate it.
johnhidley
CC BY-SA 3.0
2015-11-16T20:06:18.133Z
2015-11-16T20:06:18.133Z
honi
<ol>
<li>In physics, exponential of negative energy is often probability (e.g. in maximum entropy models).</li>
<li>|| is a notation used in divergences (or info theory) where the order matters. That is $D(A||B) \neq D(B||A)$ as in KL divergence</li>
</ol>

CC BY-SA 3.0
2015-12-17T21:05:34.800Z
2015-12-17T21:05:34.800Z
Memming
<p>I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertible and refer to both physical and Shannon entropy. They appear as equation (5) in <a href="http://www.fil.ion.ucl.ac.uk/spm/doc/papers/Action_and_behavior_A_free-energy_formulation.pdf">this article (PDF</a>; DOI 10.1007/s00422-010-0364-z). Here they are:</p>

<ul>
<li>Energy minus entropy: $F = −{\ln p(\tilde s,Ψ|m)}q + {\ln q(Ψ|μ)}q$</li>
<li>Divergence plus surprise: = $D(q(Ψ|μ)||p(Ψ|\tilde s,m)) − \ln p (\tilde s|m)$</li>
<li>Complexity minus accuracy: = $D(q(Ψ|μ)||p(Ψ|m)) − {\ln p(\tilde s|Ψ,m)}q$</li>
</ul>

<p>The things I am struggling with at this point are:</p>

<ol>
<li>the meaning of the || in the 2nd and 3rd versions of the equations</li>
<li>the negative logs</li>
</ol>

<p>Any help in understanding how these equations are actually what Fristen claims them to be would be greatly appreciated. For example, in the 1st equation, in what sense is the first term energy, etc.?</p>

CC BY-SA 3.0
2014-07-11T23:48:42.980Z
[Cross-posted on Cross Validated](http://stats.stackexchange.com/q/107653/32036).
Nick Stauner
CC BY-SA 3.0
2014-07-15T17:02:14.973Z
+1 for also being confused by much of Fristen's writing.
zergylord
CC BY-SA 3.0
2015-11-16T19:52:33.127Z
[Cross-posted on Mathematics](http://math.stackexchange.com/questions/865206/trying-to-understand-the-math-in-a-neuroscience-article-by-karl-friston)
honi
CC BY-SA 3.0
2015-11-17T18:59:44.687Z
[Cross-posted on Data Science](http://datascience.stackexchange.com/questions/726/trying-to-understand-free-energy-equations-in-a-karl-friston-neuroscience-articl)
honi
<neuroscience><perception><motor><perceptual-learning>
Trying to understand equations in Karl Friston article
johnhidley
# Trying to understand equations in Karl Friston article

(Asked by: username_0 on 2014-07-11)

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertible and refer to both physical and Shannon entropy. They appear as equation (5) in [this article (PDF](http://www.fil.ion.ucl.ac.uk/spm/doc/papers/Action_and_behavior_A_free-energy_formulation.pdf); DOI 10.1007/s00422-010-0364-z). Here they are:

* Energy minus entropy: $F = −{\ln p(\tilde s,Ψ|m)}q + {\ln q(Ψ|μ)}q$
* Divergence plus surprise: = $D(q(Ψ|μ)||p(Ψ|\tilde s,m)) − \ln p (\tilde s|m)$
* Complexity minus accuracy: = $D(q(Ψ|μ)||p(Ψ|m)) − {\ln p(\tilde s|Ψ,m)}q$

The things I am struggling with at this point are:

1. the meaning of the || in the 2nd and 3rd versions of the equations
2. the negative logs

Any help in understanding how these equations are actually what Fristen claims them to be would be greatly appreciated. For example, in the 1st equation, in what sense is the first term energy, etc.?

**Tags:** neuroscience, perception, motor, perceptual-learning

> **Comments:**
> **username_1**: [Cross-posted on Cross Validated](http://stats.stackexchange.com/q/107653/32036).  
> **username_2**: +1 for also being confused by much of Fristen's writing.  
> **username_3**: [Cross-posted on Mathematics](http://math.stackexchange.com/questions/865206/trying-to-understand-the-math-in-a-neuroscience-article-by-karl-friston)  
> **username_3**: [Cross-posted on Data Science](http://datascience.stackexchange.com/questions/726/trying-to-understand-free-energy-equations-in-a-karl-friston-neuroscience-articl)  

---

## 2 answers

### Answer by username_3 on 2015-11-16. Score: 2

As mentioned in the definitions section on page 2 of the paper, $D(q||p) = \_q$ is the Kullback-Leibler divergence or cross-entropy between two densities.

The reason for the negative logs is that is the convention when discussing entropy in the context of information theory. This allows information to be combined additively. (<https://en.wikipedia.org/wiki/Entropy_(information_theory)#Rationale>)

> **Comments:**
> **username_3**: i picked this article up as a result of this question and plan to give a more complete explanation of the equations soon  
> **username_0**: Great! I eagerly await your take on what Friston is saying here. i find his work very interesting, but due to my lack of math skills, I cannot grasp the details and do not know how to evaluate it.  

---

### Answer by username_4 on 2015-12-17. Score: 1

1. In physics, exponential of negative energy is often probability (e.g. in maximum entropy models).
2. || is a notation used in divergences (or info theory) where the order matters. That is $D(A||B) \neq D(B||A)$ as in KL divergence


---

