<p>A simplee trick (<a href="https://facwiki.cs.byu.edu/nlp/index.php/Log_Domain_Computations" rel="nofollow">explained here</a>) that let you use the $\log$ approach even when you must sum probabilities.</p>

<p>Problem: $\log(\exp(a) + \exp(b))$ can lead to an underflow, to avoid it you can use this formula:</p>

<p>$\log(x + y) = \log(x) + \log(1.0 + \exp( \log(y) - \log(x) ) )$</p>

<p>Or use <a href="http://www.cs.ubc.ca/~murphyk/Teaching/CS532c_Fall04/Lectures/lec12x4.pdf" rel="nofollow">another approach</a>:</p>

<p>$\log(\exp(a) + \exp(b)) = \log( \exp(a - C) + \exp(b - C)) + C$</p>

<p>Setting $C = \max(a,b)$</p>

<p>For example: $\log(e^{-120}+e^{-121}) = \log(e^{-120}(e^0 + e^{-1}))= \log(e^0+e^{-1})-120$</p>

CC BY-SA 2.5
2011-03-11T15:45:46.630Z
2011-03-11T15:45:46.630Z
Marzio De Biasi
<p>I need to translate a training algorithm that involves sums and multiplications of probabilities to actual code. For that I need some sort of scaling procedure that allows me to avoid underflows, that is, misleading 0 probabilities.</p>

<p>A typical method is to apply the logs of probabilities but because of the sums this is not readily possible for my case. Another approach I saw in Rabiner's tutorial on HMMs, was his scaling procedure only dependent on t (time) applied to the forward algorithm and (the other way around) the backward algorithm, that when combined cancel each other to obtain the desired trained probabilities.</p>

<p><strong>My question</strong>
I wonder if there are books or text resources explaining common approaches to tackle the underflow problem that results in working with continuous multiplications of probabilities. Do you know any? </p>

<p>I hope I can get some ideas from that.</p>

CC BY-SA 2.5
2011-03-15T20:27:19.543Z
@Suresh As a matter of fact I'm interested in the numerical analysis aspect of the problem, although I didn't know how to name it. I'd would be glad if you could provide more textual references
juanmirocks
CC BY-SA 2.5
2011-03-11T13:52:43.263Z
Perhaps you can use an arbitrary precision arithmetic library (http://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic) and "sometimes" (if your problem allows it) in your code round values to zero if they are less than a given $\epsilon$
Marzio De Biasi
CC BY-SA 2.5
2011-03-11T17:05:13.627Z
This seems out of scope for this site.
Suresh Venkat
CC BY-SA 2.5
2011-03-11T18:01:15.240Z
@Suresh: Care to explain why?
Tsuyoshi Ito
CC BY-SA 2.5
2011-03-11T18:21:33.590Z
Well, as stated it appears to be about implementation issues when dealing with discrete probabilities.
Suresh Venkat
CC BY-SA 2.5
2011-03-11T20:22:06.827Z
If you're interested in practical methods for improving implementations when dealing with probabilities, you might want to ask the question also on StackOverflow. On the other hand, Vor's answer looks pretty good.
Peter Shor 
CC BY-SA 2.5
2011-03-11T21:22:11.630Z
@Suresh: I think that numerical analysis is in the scope of theoretical computer science.  Your comment seems to be dismissing the whole field of numerical analysis as “implementation issues,” although I do not think that that is the intent of your comment.
Tsuyoshi Ito
CC BY-SA 2.5
2011-03-11T21:30:47.377Z
@Tsuyoshi I think what I'm suggesting is that there's a better question lurking within, that *would* be about numerical analysis in general. But the question *as stated* is much more limited in scope.
Suresh Venkat
<pr.probability><na.numerical-analysis>
Scaling procedures to address false 0's after multiplying probabilities
juanmirocks
# Scaling procedures to address false 0's after multiplying probabilities

(Asked by: username_0 on 2011-03-11)

I need to translate a training algorithm that involves sums and multiplications of probabilities to actual code. For that I need some sort of scaling procedure that allows me to avoid underflows, that is, misleading 0 probabilities.

A typical method is to apply the logs of probabilities but because of the sums this is not readily possible for my case. Another approach I saw in Rabiner's tutorial on HMMs, was his scaling procedure only dependent on t (time) applied to the forward algorithm and (the other way around) the backward algorithm, that when combined cancel each other to obtain the desired trained probabilities.

**My question**
I wonder if there are books or text resources explaining common approaches to tackle the underflow problem that results in working with continuous multiplications of probabilities. Do you know any?

I hope I can get some ideas from that.

**Tags:** pr.probability, na.numerical-analysis

> **Comments:**
> **username_1**: Perhaps you can use an arbitrary precision arithmetic library (http://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic) and "sometimes" (if your problem allows it) in your code round values to zero if they are less than a given $\epsilon$  
> **username_2**: This seems out of scope for this site.  
> **username_3**: @Suresh: Care to explain why?  
> **username_2**: Well, as stated it appears to be about implementation issues when dealing with discrete probabilities.  
> **username_4**: If you're interested in practical methods for improving implementations when dealing with probabilities, you might want to ask the question also on StackOverflow. On the other hand, Vor's answer looks pretty good.  
> **username_3**: @Suresh: I think that numerical analysis is in the scope of theoretical computer science.  Your comment seems to be dismissing the whole field of numerical analysis as “implementation issues,” although I do not think that that is the intent of your comment.  
> **username_2**: @Tsuyoshi I think what I'm suggesting is that there's a better question lurking within, that *would* be about numerical analysis in general. But the question *as stated* is much more limited in scope.  
> **username_0**: @Suresh As a matter of fact I'm interested in the numerical analysis aspect of the problem, although I didn't know how to name it. I'd would be glad if you could provide more textual references  

---

## 1 answers

### Answer by username_1 on 2011-03-11. Score: 3 | Accepted answer

A simplee trick ([explained here](https://facwiki.cs.byu.edu/nlp/index.php/Log_Domain_Computations)) that let you use the $\log$ approach even when you must sum probabilities.

Problem: $\log(\exp(a) + \exp(b))$ can lead to an underflow, to avoid it you can use this formula:

$\log(x + y) = \log(x) + \log(1.0 + \exp( \log(y) - \log(x) ) )$

Or use [another approach](http://www.cs.ubc.ca/~murphyk/Teaching/CS532c_Fall04/Lectures/lec12x4.pdf):

$\log(\exp(a) + \exp(b)) = \log( \exp(a - C) + \exp(b - C)) + C$

Setting $C = \max(a,b)$

For example: $\log(e^{-120}+e^{-121}) = \log(e^{-120}(e^0 + e^{-1}))= \log(e^0+e^{-1})-120$


---

