<p>I agree with D.W. that this should just be a dynamic programming question.
Assume that $P_B$ is known and that we have a prior on $P_A$ and that $N$ is known. (Without a prior on $P_A$ or a known $N$, I do not see how your objective or "optimal" are well-defined.)
Let optimal mean "maximizes expected number of heads" with the expectation over both the coin flips and the draw of $P_A$ from the prior. (This seems to me the only natural definition of (edit: <em>achievable</em>) "optimal" here.)</p>

<p>Let $Val(t,prior)$ be the expected number of heads of the optimal strategy starting at time $t$ with prior $prior$ on $P_A$. Then immediately
 $$ Val(N,prior) = \max\{ P_B, ~ \mathbb{E}P_A \} $$
since at the last step we should just choose the higher chance of flipping heads, and $\mathbb{E}P_A$ with respect to $prior$ is the probability of getting heads from flipping coin $A$. So we know how to choose the coin at the last step for any distribution on $P_A$.</p>

<p>At step $t &lt; N$,
  $$ Val(t,prior) = \max\{ P_B + Val(t+1,prior), ~~ \text{val if flip $A$}\} $$
where
  $$ \text{val if flip $A$} ~~ = \mathbb{E}P_A ~~ + (\mathbb{E}P_A) Val(t+1,posterior(prior,heads)) ~~ + (1-\mathbb{E}P_A) Val(t+1,posterior(prior,tails)) . $$</p>

<p>Explanation: The value for flipping $B$ is $P_B$ for this coin flip, plus the value of the optimal strategy at time $t+1$ with the same prior on $P_A$ as we currently have. The value for flipping $A$ is the expected number of heads for this round, $\mathbb{E}P_A$, plus our expected value over the two possible cases. First, if the coin comes up heads (which happens with probability $\mathbb{E}P_A$), then we will update our prior to a posterior on $P_A$ conditioned on that event, and we will play the optimal strategy at time $t+1$ with that new distribution. Similarly if it comes up tails.</p>

<p>There are at the very most $O(n^3)$ cases to calculate because there are $N$ time steps, and at each time step $t$, we have a distribution on $P_A$ that consists of the original prior updated after observing $j=0,\dots,t$ heads and $k=0,\dots,t-j$ tails. But this should be reduced with your observation that we need only consider strategies that try $A$ for some amount of time, then (optionally) flip $B$ for the rest of the time.</p>

<p>Why this is optimal: Assume that the strategy is optimal at time $t+1$ for the specified prior distribution. But the "correct" prior distribution is always exactly the one we plug in to the formula: After observing a heads, the correct distribution over $P_A$ is precisely $posterior(prior,heads)$, and so on. When I say correct, what I mean is that the posterior gives the exact distribution over $P_A$ given that $P_A$ is drawn from the prior and these outcomes are observed. So we really are maximizing over the "average case" in each given scenario.</p>

CC BY-SA 3.0
2014-06-24T22:19:09.387Z
I disagree. The optimal solution value is $N\max \{P_A,P_B\}$, where $P_A$ was drawn from the prior distribution.
In fact, you could occasionally beat $N\max \{\mathcal{E} P_A,P_B\}$, where $\mathcal{E} P_A$ is the expectation of the prior of $A$, as you can learn the actual value of $A$ through sampling.
R B
CC BY-SA 3.0
2014-06-25T00:39:31.110Z
@RB, I don't understand your comment.  The optimal solution value is the expected value of the best strategy, where "expected value" is as defined in the first paragraph of this answer.  You seem to be assuming there exists a strategy whose expected value is $N \max(P_A,P_B)$.  I don't see how to construct a strategy with that expected value.  (Remember, the expectation is over not just the coin flips but also the random choice of $P_A$.)
D.W.
CC BY-SA 3.0
2014-06-25T01:57:02.613Z
@RB, to expand on D.W.'s point, I think the way to look at it is: sure, we could compare to some "omniscient" optimal and try to do our best relative to that, but either way "do our best" means "maximize our expected number of heads", so I'm not sure it helps to think about this omniscient optimal solution. ... In a case where the optimal achievable strategy is not well-defined or is unknown, it makes sense to define some upper-bound and talk about approximation ratio or regret relative to that. But if we can define a single best strategy, then I'm not sure regret is helpful.
usul
CC BY-SA 3.0
2014-06-25T06:05:02.953Z
@D.W - There doesn't has to be a way to construct a $N\max(P_A,P_B)$ strategy to compare ours to it.. In fact, this is done frequently in regret minimization, you compare yourself to the best offline strategy. Beating $N\max\{\mathbb{E}P_A,P_B\}$ is sometimes trivial (consider $P_B=0.5$ and $P_A\in\{0,1\}$ uniformly - you can easily beat it w.p. 1/2).
R B
CC BY-SA 3.0
2014-06-24T16:18:56.480Z
2014-06-25T15:50:24.503Z
2014-06-25T15:50:24.503Z
usul
<p>This question can be reduced to that of an online auction of a digital good, the optimization of which is structured in a similar way.</p>

<p>For example, imagine there are $N$ bidders in an auction, and you -- the auctioneer -- would like to maximize your revenue. Each bidder has some valuation for the object that we can assume is drawn from a reasonable prior distribution. The auctioneer posts a price, and he has the option of taking it or leaving it.</p>

<p>The auctioneer can post any price in the set $X = {x_1, ..., x_n}$ –- and without loss of generality we can assume $X \in [0, 1]$. There is some optimal, fixed price, that maximizes your revenue so your regret is $P_{fixed} - Empirical$. </p>

<p>And the question of the algorithm used to generate $Empirical$ is very well-studied in economics and auction theory. </p>

<p>You're asking the same question, for 100 items and 2 bidders. A heads is analogous to a bidder accepting a posted price and the coins are like the prices themselves.</p>

<p>You could start with <a href="http://www.cs.cmu.edu/~avrim/Papers/onlineauction.pdf" rel="nofollow">Online Learning in Online Auctions</a> for a host of optimal strategies for your play. One option is a variant of the fairly well-known weighted majority voting algorithm, which guarantees a regret that grows as a square root of the number of bidders.</p>

CC BY-SA 3.0
2014-06-24T14:15:12.757Z
2014-06-24T14:15:12.757Z
Ashok M Rao
<p>Assume you have two coins $A,B$ with biases $P_A,P_B$ respectively.
We would like to make $N$ coin tosses and get the maximal number of heads possible.
Unfortunately, we know $P_B$, but $P_A$ is unknown at the beginning of the process.</p>

<p><strong>What is the best possible strategy for selecting which coin to toss at each step?</strong></p>

<p>Note: I'm looking for actual optimization of the selection, not heuristic methods. You may assume a reasonable prior distribution on $P_A$ if needed.</p>

<hr>

<p>EDIT: </p>

<p>Here is a formal definition of the objective.
Let $P_A,P_B\in (0,1)$ and $P=max\{P_A,P_B\}$. The optimal strategy in hindsight will achieve on average $P\cdot N$ head tosses.
Let $a(P_A,P_B)$ be the expected number of times our strategy picked coin $A$ given that the probabilities were $P_A,P_B$.</p>

<p><strong>The Optimization Objective:</strong> minimize the expected regret, i.e. $$Regret=P\cdot N - P_A\cdot a(P_A,P_B) - P_B\cdot (N-a(P_A,P_B))$$</p>

<p><strong>The Goal:</strong> Devise a selection strategy that guarantees (expected) $Regret\leq r$ for any values of $P_A,P_B$ (for the smallest value of $r$ possible).</p>

<p>Note that if $P_A\approx P_B$ this is not a problem in this definition as it means our regret will be small regardless of which coin we flip. </p>

<hr>

<p>If $N$ is unknown (i.e. the process may end arbitrarily after an adversarialy chosen number of tosses), the best strategy would be to occasionally explore coin $A$ to get a better $\widehat {P_A}$ estimation, and in the rest of the time toss coin $A$ iff $\widehat {P_A} &gt; P_B$.</p>

<p>In the scenario $N$ is known, it's not hard to argue that the optimal strategy makes a series of tosses of coin $A$ and then either keeps on tossing $A$ for the remaining tosses or starts tossing only coin $B$, but how can we optimize the cutoff (i.e. the switch to tossing coin $B$) to maximize the expectancy of head results?</p>

CC BY-SA 3.0
2014-06-23T16:47:38.607Z
I disagree, if I understood your last statement correctly. While I agree that first we should toss A and then B, there need not be a cut-off. Image that first it seems that $P_A>P_B$, so we start tossing A and then it turns out that most probably $P_A<P_B$. In this case we would like to switch to B.
domotorp
CC BY-SA 3.0
2014-06-23T16:48:15.197Z
I also think this question is more suitable for mathoverflow.
domotorp
CC BY-SA 3.0
2014-06-23T22:10:12.497Z
@domotorp - What I meant by cutoff is that once (and if) we switch to tossing coin $B$, we will not toss $A$ ever again (in the optimal strategy), so the question becomes when to make the switch given the current sequence of $A$ tosses.
R B
CC BY-SA 3.0
2014-06-23T23:04:35.033Z
What measure are you using, to define "best" possible strategy?  Expected performance?  Worst-case performance, over the worst possible value of $P_A$ (and $N$, if $N$ is not known)?  If we have a Bayesian prior on $P_A$, then it sounds like you might be talking about expected performance (expectation taken with respect to $P_A$).
D.W.
CC BY-SA 3.0
2014-06-23T23:14:14.133Z
For your second case, a strategy is basically a function $f:\{1,2,\dots,N\}\to \{0,1,2,\dots,N\}$ that says: if you've tossed $A$ for $k$ times and if the number of heads is $< f(k)$, then switch to tossing $B$.  For any fixed $N$, you should be able to find the optimal strategy using dynamic programming.  Each subproblem is: suppose that after $k$ flips, we have seen $h$ heads and have/haven't switched to tossing coin $B$; then what's the optimum strategy from here on?  There are only $O(N^2)$ subproblems.
D.W.
CC BY-SA 3.0
2014-06-24T10:07:34.813Z
@D.W., I guess the most natural objective would be minimizing the expected regret, which is equivalent to minimizing the expected number of times you picked the inferior coin. Since the separability depends on $||P_A-P_B||$, it would be nice to have a regret which is at most $\epsilon N$ or something similar, so if the bias different is small you can do whatever you like.
R B
CC BY-SA 3.0
2014-06-24T11:38:38.693Z
If the bias approaches 0, the expected regret approaches $N/2$, since you can't determine which coin is better. But the actual strategy won't make very much difference to how many heads you get. So by minimizing expected regret, you may be putting too much weight on a case (bias near 0) you don't really care about. Maybe you should minimize expected regret times $||P_A-P_B||$.
Peter Shor 
CC BY-SA 3.0
2014-06-24T12:41:27.820Z
@PeterShor - please see my edit to the question. I define the regret as the expected loss of "head" flip, so if the the bias difference is very small, it doesn't matter which coin you choose.
R B
<ds.algorithms><pr.probability><st.statistics><exploration-exploitation>
Maximizing the number of heads in $N$ tosses by choosing which coin to toss
R B
# Maximizing the number of heads in $N$ tosses by choosing which coin to toss

(Asked by: username_0 on 2014-06-23)

Assume you have two coins $A,B$ with biases $P\_A,P\_B$ respectively.
We would like to make $N$ coin tosses and get the maximal number of heads possible.
Unfortunately, we know $P\_B$, but $P\_A$ is unknown at the beginning of the process.

**What is the best possible strategy for selecting which coin to toss at each step?**

Note: I'm looking for actual optimization of the selection, not heuristic methods. You may assume a reasonable prior distribution on $P\_A$ if needed.

---

EDIT:

Here is a formal definition of the objective.
Let $P\_A,P\_B\in (0,1)$ and $P=max\{P\_A,P\_B\}$. The optimal strategy in hindsight will achieve on average $P\cdot N$ head tosses.
Let $a(P\_A,P\_B)$ be the expected number of times our strategy picked coin $A$ given that the probabilities were $P\_A,P\_B$.

**The Optimization Objective:** minimize the expected regret, i.e. $$Regret=P\cdot N - P\_A\cdot a(P\_A,P\_B) - P\_B\cdot (N-a(P\_A,P\_B))$$

**The Goal:** Devise a selection strategy that guarantees (expected) $Regret\leq r$ for any values of $P\_A,P\_B$ (for the smallest value of $r$ possible).

Note that if $P\_A\approx P\_B$ this is not a problem in this definition as it means our regret will be small regardless of which coin we flip.

---

If $N$ is unknown (i.e. the process may end arbitrarily after an adversarialy chosen number of tosses), the best strategy would be to occasionally explore coin $A$ to get a better $\widehat {P\_A}$ estimation, and in the rest of the time toss coin $A$ iff $\widehat {P\_A} > P\_B$.

In the scenario $N$ is known, it's not hard to argue that the optimal strategy makes a series of tosses of coin $A$ and then either keeps on tossing $A$ for the remaining tosses or starts tossing only coin $B$, but how can we optimize the cutoff (i.e. the switch to tossing coin $B$) to maximize the expectancy of head results?

**Tags:** ds.algorithms, pr.probability, st.statistics, exploration-exploitation
> **Comments:**
> **username_1**: I disagree, if I understood your last statement correctly. While I agree that first we should toss A and then B, there need not be a cut-off. Image that first it seems that $P_A>P_B$, so we start tossing A and then it turns out that most probably $P_A<P_B$. In this case we would like to switch to B.  
> **username_1**: I also think this question is more suitable for mathoverflow.  
> **username_0**: @domotorp - What I meant by cutoff is that once (and if) we switch to tossing coin $B$, we will not toss $A$ ever again (in the optimal strategy), so the question becomes when to make the switch given the current sequence of $A$ tosses.  
> **username_2**: What measure are you using, to define "best" possible strategy?  Expected performance?  Worst-case performance, over the worst possible value of $P_A$ (and $N$, if $N$ is not known)?  If we have a Bayesian prior on $P_A$, then it sounds like you might be talking about expected performance (expectation taken with respect to $P_A$).  
> **username_2**: For your second case, a strategy is basically a function $f:\{1,2,\dots,N\}\to \{0,1,2,\dots,N\}$ that says: if you've tossed $A$ for $k$ times and if the number of heads is $< f(k)$, then switch to tossing $B$.  For any fixed $N$, you should be able to find the optimal strategy using dynamic programming.  Each subproblem is: suppose that after $k$ flips, we have seen $h$ heads and have/haven't switched to tossing coin $B$; then what's the optimum strategy from here on?  There are only $O(N^2)$ subproblems.  
> **username_0**: @D.W., I guess the most natural objective would be minimizing the expected regret, which is equivalent to minimizing the expected number of times you picked the inferior coin. Since the separability depends on $||P_A-P_B||$, it would be nice to have a regret which is at most $\epsilon N$ or something similar, so if the bias different is small you can do whatever you like.  
> **username_3**: If the bias approaches 0, the expected regret approaches $N/2$, since you can't determine which coin is better. But the actual strategy won't make very much difference to how many heads you get. So by minimizing expected regret, you may be putting too much weight on a case (bias near 0) you don't really care about. Maybe you should minimize expected regret times $||P_A-P_B||$.  
> **username_0**: @PeterShor - please see my edit to the question. I define the regret as the expected loss of "head" flip, so if the the bias difference is very small, it doesn't matter which coin you choose.  

---

## 2 answers

### Answer by username_4 on 2014-06-24. Score: 2

I agree with D.W. that this should just be a dynamic programming question.
Assume that $P\_B$ is known and that we have a prior on $P\_A$ and that $N$ is known. (Without a prior on $P\_A$ or a known $N$, I do not see how your objective or "optimal" are well-defined.)
Let optimal mean "maximizes expected number of heads" with the expectation over both the coin flips and the draw of $P\_A$ from the prior. (This seems to me the only natural definition of (edit: *achievable*) "optimal" here.)

Let $Val(t,prior)$ be the expected number of heads of the optimal strategy starting at time $t$ with prior $prior$ on $P\_A$. Then immediately
$$ Val(N,prior) = \max\{ P\_B, ~ \mathbb{E}P\_A \} $$
since at the last step we should just choose the higher chance of flipping heads, and $\mathbb{E}P\_A$ with respect to $prior$ is the probability of getting heads from flipping coin $A$. So we know how to choose the coin at the last step for any distribution on $P\_A$.

At step $t < N$,
$$ Val(t,prior) = \max\{ P\_B + Val(t+1,prior), ~~ \text{val if flip $A$}\} $$
where
$$ \text{val if flip $A$} ~~ = \mathbb{E}P\_A ~~ + (\mathbb{E}P\_A) Val(t+1,posterior(prior,heads)) ~~ + (1-\mathbb{E}P\_A) Val(t+1,posterior(prior,tails)) . $$

Explanation: The value for flipping $B$ is $P\_B$ for this coin flip, plus the value of the optimal strategy at time $t+1$ with the same prior on $P\_A$ as we currently have. The value for flipping $A$ is the expected number of heads for this round, $\mathbb{E}P\_A$, plus our expected value over the two possible cases. First, if the coin comes up heads (which happens with probability $\mathbb{E}P\_A$), then we will update our prior to a posterior on $P\_A$ conditioned on that event, and we will play the optimal strategy at time $t+1$ with that new distribution. Similarly if it comes up tails.

There are at the very most $O(n^3)$ cases to calculate because there are $N$ time steps, and at each time step $t$, we have a distribution on $P\_A$ that consists of the original prior updated after observing $j=0,\dots,t$ heads and $k=0,\dots,t-j$ tails. But this should be reduced with your observation that we need only consider strategies that try $A$ for some amount of time, then (optionally) flip $B$ for the rest of the time.

Why this is optimal: Assume that the strategy is optimal at time $t+1$ for the specified prior distribution. But the "correct" prior distribution is always exactly the one we plug in to the formula: After observing a heads, the correct distribution over $P\_A$ is precisely $posterior(prior,heads)$, and so on. When I say correct, what I mean is that the posterior gives the exact distribution over $P\_A$ given that $P\_A$ is drawn from the prior and these outcomes are observed. So we really are maximizing over the "average case" in each given scenario.

> **Comments:**
> **username_0**: I disagree. The optimal solution value is $N\max \{P_A,P_B\}$, where $P_A$ was drawn from the prior distribution. In fact, you could occasionally beat $N\max \{\mathcal{E} P_A,P_B\}$, where $\mathcal{E} P_A$ is the expectation of the prior of $A$, as you can learn the actual value of $A$ through sampling.  
> **username_2**: @RB, I don't understand your comment.  The optimal solution value is the expected value of the best strategy, where "expected value" is as defined in the first paragraph of this answer.  You seem to be assuming there exists a strategy whose expected value is $N \max(P_A,P_B)$.  I don't see how to construct a strategy with that expected value.  (Remember, the expectation is over not just the coin flips but also the random choice of $P_A$.)  
> **username_4**: @RB, to expand on D.W.'s point, I think the way to look at it is: sure, we could compare to some "omniscient" optimal and try to do our best relative to that, but either way "do our best" means "maximize our expected number of heads", so I'm not sure it helps to think about this omniscient optimal solution. ... In a case where the optimal achievable strategy is not well-defined or is unknown, it makes sense to define some upper-bound and talk about approximation ratio or regret relative to that. But if we can define a single best strategy, then I'm not sure regret is helpful.  
> **username_0**: @D.W - There doesn't has to be a way to construct a $N\max(P_A,P_B)$ strategy to compare ours to it.. In fact, this is done frequently in regret minimization, you compare yourself to the best offline strategy. Beating $N\max\{\mathbb{E}P_A,P_B\}$ is sometimes trivial (consider $P_B=0.5$ and $P_A\in\{0,1\}$ uniformly - you can easily beat it w.p. 1/2).  

---

### Answer by username_5 on 2014-06-24. Score: 1

This question can be reduced to that of an online auction of a digital good, the optimization of which is structured in a similar way.

For example, imagine there are $N$ bidders in an auction, and you -- the auctioneer -- would like to maximize your revenue. Each bidder has some valuation for the object that we can assume is drawn from a reasonable prior distribution. The auctioneer posts a price, and he has the option of taking it or leaving it.

The auctioneer can post any price in the set $X = {x\_1, ..., x\_n}$ –- and without loss of generality we can assume $X \in [0, 1]$. There is some optimal, fixed price, that maximizes your revenue so your regret is $P\_{fixed} - Empirical$.

And the question of the algorithm used to generate $Empirical$ is very well-studied in economics and auction theory.

You're asking the same question, for 100 items and 2 bidders. A heads is analogous to a bidder accepting a posted price and the coins are like the prices themselves.

You could start with [Online Learning in Online Auctions](http://www.cs.cmu.edu/~avrim/Papers/onlineauction.pdf) for a host of optimal strategies for your play. One option is a variant of the fairly well-known weighted majority voting algorithm, which guarantees a regret that grows as a square root of the number of bidders.


---

