<p>The books below may be more to your liking, but in general, the texts/lecture notes are written for the use of (mainly) postgraduate students in engineering and cannot presume deep knowledge of convex analysis. </p>

<ol>
<li>Csizsar, I., and Korner, J., Information Theory: Coding Theorems for Discrete Memoryless Systems, 2nd Ed, Cambridge.</li>
<li>Berger, T., Rate Distortion Theory, quite old probably late 70's or early 80s, can't remember the publisher maybe Wiley.</li>
<li>Yeung, R.W., A First Course in Information Theory, Springer.</li>
</ol>

<p>The research articles on Shannon theory and related fields in, say, IEEE Transactions on Information Theory, may fit the bill better, though not always. </p>

<p>An older text which may also be of interest is </p>

<p>Wolfowitz, J., Coding Theorems of Information Theory, Springer, 1960's.</p>

CC BY-SA 3.0
2015-05-04T23:56:06.663Z
just a comment, it depends on what one would consider as "deep", for example i consider information theory as deeper than convex analysis (for example because it has more applications to diverse fields) and i prefer to reduce convex analysis to information theory instead of vice-versa. It can be done if one wants to (lets say how category-theorists would like to reduce everything to categoriers instead of algebra :)).
Nikos M.
CC BY-SA 3.0
2015-05-05T00:05:09.650Z
@ Nikos M., absolutely, and I sympathise. For example, some information theory problems give rise to non-convex capacity regions, e.g., OFDMA.
kodlu
CC BY-SA 3.0
2015-04-29T10:00:08.560Z
2015-04-29T10:00:08.560Z
kodlu
<p>I'm taking a graduate level course in information theory and I'm constantly struck by how much convex optimization there is in this subject. However, the proofs seem to shy away from using the full machinery of relaxation theory, duality, etc. This is understandable since you don't want to require a full semester of convex optimization in order to teach this stuff. But as someone fairly well-versed in optimization, I feel like I'm missing out on a lot of elegance and intuition when these links aren't explored more. I often notice proofs that would be way shorter if you had utilized convex analysis as well.</p>

<p>Are there books that cover information theory more from this perspective? We're mostly using lecture notes from Stefan Moser, Y. Polyanskiy and Y. Wu, as well as <em>Network Information Theory</em> by El Gamal.</p>

CC BY-SA 3.0
2015-04-28T21:22:58.823Z
Can you give an example of such "elegance and intuition" in another context?
kodlu
CC BY-SA 3.0
2015-04-29T05:44:40.947Z
Well, as one of many examples, we were talking about saddle point properties of the channel capacity. I believe it's called the KL divergence minmax formulation or something. It's covered fairly well in the Polyanskiy notes I mentioned above. What struck me is that it seemed to be exactly a re-statement of Lagrangian relaxation/duality in a different context.
luegofuego
CC BY-SA 3.0
2015-04-29T05:47:33.473Z
A slightly more mundane example might be when we were asked to prove the convexity of the rate distortion function, which is a one line proof if you recall some properties of infimums over convex sets etc.
luegofuego
<reference-request><it.information-theory><convex-optimization>
Information theory and convex optimization
luegofuego
# Information theory and convex optimization

(Asked by: username_0 on 2015-04-28)

I'm taking a graduate level course in information theory and I'm constantly struck by how much convex optimization there is in this subject. However, the proofs seem to shy away from using the full machinery of relaxation theory, duality, etc. This is understandable since you don't want to require a full semester of convex optimization in order to teach this stuff. But as someone fairly well-versed in optimization, I feel like I'm missing out on a lot of elegance and intuition when these links aren't explored more. I often notice proofs that would be way shorter if you had utilized convex analysis as well.

Are there books that cover information theory more from this perspective? We're mostly using lecture notes from Stefan Moser, Y. Polyanskiy and Y. Wu, as well as *Network Information Theory* by El Gamal.

**Tags:** reference-request, it.information-theory, convex-optimization

> **Comments:**
> **username_1**: Can you give an example of such "elegance and intuition" in another context?  
> **username_0**: Well, as one of many examples, we were talking about saddle point properties of the channel capacity. I believe it's called the KL divergence minmax formulation or something. It's covered fairly well in the Polyanskiy notes I mentioned above. What struck me is that it seemed to be exactly a re-statement of Lagrangian relaxation/duality in a different context.  
> **username_0**: A slightly more mundane example might be when we were asked to prove the convexity of the rate distortion function, which is a one line proof if you recall some properties of infimums over convex sets etc.  

---

## 1 answers

### Answer by username_1 on 2015-04-29. Score: 4

The books below may be more to your liking, but in general, the texts/lecture notes are written for the use of (mainly) postgraduate students in engineering and cannot presume deep knowledge of convex analysis.

1. Csizsar, I., and Korner, J., Information Theory: Coding Theorems for Discrete Memoryless Systems, 2nd Ed, Cambridge.
2. Berger, T., Rate Distortion Theory, quite old probably late 70's or early 80s, can't remember the publisher maybe Wiley.
3. Yeung, R.W., A First Course in Information Theory, Springer.

The research articles on Shannon theory and related fields in, say, IEEE Transactions on Information Theory, may fit the bill better, though not always.

An older text which may also be of interest is

Wolfowitz, J., Coding Theorems of Information Theory, Springer, 1960's.

> **Comments:**
> **username_2**: just a comment, it depends on what one would consider as "deep", for example i consider information theory as deeper than convex analysis (for example because it has more applications to diverse fields) and i prefer to reduce convex analysis to information theory instead of vice-versa. It can be done if one wants to (lets say how category-theorists would like to reduce everything to categoriers instead of algebra :)).  
> **username_1**: @ username_2, absolutely, and I sympathise. For example, some information theory problems give rise to non-convex capacity regions, e.g., OFDMA.  

---

