Lets say I have three independent sources and each of them make predictions for the weather tomorrow. The first one says that the probability of rain tomorrow is 0, then the second one says that the probability is 1, and finally the last one says that the probability is 50%. I would like to know the total probability given that information.

If apply the multiplication theorem for independent events I get 0, which doesn't seem correct. Why is not possible to multiply all three if all sources are independent? Is there some Bayesian way to update the prior as I get new information?

Note: This is not homework, is something that I was thinking about.

You ask about three things: (a) how to combine several forecasts to get single forecast, (b) if Bayesian approach can be used in here, and (c) how to deal with zero-probabilities.

Combining forecasts, is a common practice. If you have several forecasts than if you take average of those forecasts the resulting combined forecast should be better in terms of accuracy than any of the individual forecasts. To average them you could use weighted average where weights are based on inverse errors (i.e. precision), or information content. If you had knowledge on reliability of each source you could assign weights that are proportional to reliability of each source, so more reliable sources have greater impact on the final combined forecast. In your case you do not have any knowledge about their reliability so each of the forecasts have the same weight and so you can use simple arithmetic mean of the three forecasts

$$ 0\%\times.33+50\%\times.33+100\%\times.33 = (0\%+50\%+100\%)/3=50\% $$

As was suggested in comments by @AndyW and @ArthurB., other methods besides simple weighted mean are available. Many such methods are described in literature about averaging expert forecasts, that I was not familiar with before, so thanks guys. In averaging expert forecasts sometimes we want to correct for the fact that experts tend to regress to the mean (Baron et al, 2013), or make their forecasts more extreme (Ariely et al, 2000; Erev et al, 1994). To achieve this one could use transformations of individual forecasts $p_i$, e.g. logit function

$$ \mathrm{logit}(p_i) = \log\left( \frac{p_i}{1-p_i} \right) \tag{1} $$

odds to the $a$-th power

$$ g(p_i) = \left( \frac{p_i}{1-p_i} \right)^a \tag{2} $$

where $0 < a < 1$, or more general transformation of form

$$ t(p_i) = \frac{p_i^a}{p_i^a + (1-p_i)^a} \tag{3} $$

where if $a=1$ no transformation is applied, if $a>1$ individual forecasts are made more extreme, if $0 < a<1$ forecasts are made less extreme, what is shown on picture below (see Karmarkar, 1978; Baron et al, 2013).



After such transformation forecasts are averaged (using arithmetic mean, median, weighted mean, or other method). If equations (1) or (2) were used results need to be back-transformed using inverse logit for (1) and inverse odds for (2). Alternatively, geometric mean can be used (see Genest and Zidek, 1986; cf. Dietrich and List, 2014)

$$ \hat p = \frac{ \prod_{i=1}^N p_i^{w_i} }{ \prod_{i=1}^N p_i^{w_i} + \prod_{i=1}^N (1 - p_i)^{w_i} } \tag{4}$$

or approach proposed by Satopää et al (2014)

$$ \hat p = \frac{ \left[ \prod_{i=1}^N \left(\frac{p_i}{1-p_i} \right)^{w_i} \right]^a }{  1 + \left[ \prod_{i=1}^N \left(\frac{p_i}{1-p_i} \right)^{w_i} \right]^a } \tag{5}$$

where $w_i$ are weights. In most cases equal weights $w_i = 1/N$ are used unless a priori information that suggests other choice exists. Such methods are used in averaging expert forecasts so to correct for under- or overconfidence. In other cases you should consider if transforming forecasts to more, or less extreme is justified since it can make resulting aggregate estimate fall out of the boundaries marked by the lowest and the greatest individual forecast.

If you have a priori knowledge about rain probability you can apply Bayes theorem to update the forecasts given the a priori probability of rain in similar fashion as described in here. There is also a simple approach that could be applied, i.e. calculate weighted average of your $p_i$ forecasts (as described above) where prior probability $\pi$ is treated as additional data point with some prespecified weight $w_{\pi}$ as in this IMDB example (see also source, or here and here for discussion; cf. Genest and Schervish, 1985), i.e.

$$ \hat p = \frac{ \left(\sum_{i=1}^N p_i w_i \right) + \pi w_{\pi} }{ \left(\sum_{i=1}^N w_i \right) + w_{\pi} } \tag{6}$$

From your question however it does not follow that you have any a priori knowledge about your problem so you would probably use uniform prior, i.e. assume a priori $50\%$ chance of rain and this does not really change much in case of example that you provided.

For dealing with zeros, there are several different approaches possible. First you should notice that $0\%$ chance of rain is not really reliable value, since it says that it is impossible that it will rain. Similar problems often occur in natural language processing when in your data you do not observe some values that possibly can occur (e.g. you count frequencies of letters and in your data some uncommon letter does not occur at all). In this case the classical estimator for probability, i.e.

$$ p_i = \frac{n_i}{\sum_i n_i} $$

where $n_i$ is a number of occurrences of $i$th value (out of $d$ categories), gives you $p_i = 0$ if $n_i = 0$. This is called zero-frequency problem. For such values you know that their probability is nonzero (they exist!), so this estimate is obviously incorrect. There is also a practical concern: multiplying and dividing by zeros leads to zeros or undefined results, so zeros are problematic in dealing with.

The easy and commonly applied fix is, to add some constant $\beta$ to your counts, so that

$$ p_i = \frac{n_i + \beta}{(\sum_i n_i) + d\beta} $$

The common choice for $\beta$ is $1$, i.e. applying uniform prior based on Laplace's rule of succession, $1/2$ for Krichevsky-Trofimov estimate, or $1/d$ for Schurmann-Grassberger (1996) estimator. Notice however that what you do here is you apply out-of-data (prior) information in your model, so it gets subjective, Bayesian flavor. With using this approach you have to remember of assumptions you made and take them into consideration. The fact that we have strong a priori knowledge that there should not be any zero probabilities in our data directly justifies the Bayesian approach in here. In your case you do not have frequencies but probabilities, so you would be adding some very small value so to correct for zeros. Notice however that in some cases this approach may have bad consequences (e.g. when dealing with logs) so it should be used with caution.



Schurmann, T., and P. Grassberger. (1996). Entropy estimation of symbol sequences. Chaos, 6, 41-427. 

Ariely, D., Tung Au, W., Bender, R.H., Budescu, D.V., Dietz, C.B., Gu, H., Wallsten, T.S. and  Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6(2), 130.

Baron, J., Mellers, B.A., Tetlock, P.E., Stone, E. and Ungar, L.H. (2014). Two reasons to make aggregated probability forecasts more extreme. Decision Analysis, 11(2), 133-145.

Erev, I., Wallsten, T.S., and Budescu, D.V. (1994). Simultaneous over-and underconfidence: The role of error in judgment processes. Psychological review, 101(3), 519.

Karmarkar, U.S. (1978). Subjectively weighted utility: A descriptive extension of the expected utility model. Organizational behavior and human performance, 21(1), 61-72.

Turner, B.M., Steyvers, M., Merkle, E.C., Budescu, D.V., and Wallsten, T.S. (2014). Forecast aggregation via recalibration. Machine learning, 95(3), 261-289.

Genest, C., and Zidek, J. V. (1986). Combining probability distributions: a
critique and an annotated bibliography. Statistical Science, 1, 114–135.

Satopää, V.A., Baron, J., Foster, D.P., Mellers, B.A., Tetlock, P.E., and Ungar, L.H. (2014). Combining multiple probability predictions using a simple logit model. International Journal of Forecasting, 30(2), 344-356.

Genest, C., and Schervish, M. J. (1985). Modeling expert judgments for Bayesian updating. The Annals of Statistics, 1198-1212.

Dietrich, F., and List, C. (2014). Probabilistic Opinion Pooling. (Unpublished)
