<p>The main idea of following upper confidence bounds is minimizing regret, that is, the difference between the received reward and the maximum. The paper to read on this is: "Finite-time analysis of the multiarmed bandit problem" by Peter Auer, Nicoló Cesa-Bianchi, and Peter Fischer.</p>

<p>If we interpret life as a Markov Decision Process, then most reinforcement learning theories apply quite well. I.e. we want to keep an open mind, but we want to do what we think is best exponentially more often than exploratory actions.</p>

<p>Of course, interpreting life as an MDP is a bit of a stretch — foresight, for example, has to be interpreted as simulated experience, but I believe one can find quite a lot of wisdom in reinforcement learning and artificial intelligence in general.</p>

CC BY-SA 3.0
2011-08-15T15:57:30.163Z
Yes, following the UCB policy gives $\tilde{O}(\sqrt{T})$ regret, which is asymptotically optimal for the MAB problem.  The MAB problem is a special case of reinforcement learning, with a time horizon of $1$.
Lev Reyzin
CC BY-SA 3.0
2011-08-13T21:09:12.250Z
2011-08-13T21:09:12.250Z
Don Reba
<p>Sutton recommended using an optimistic heuristic on decision making by considering the upper bound of the confidence range of an action-value.  From my testing, it seems to work.  I find this fascinating and wonder if it could be proven that optimistic thinking (within these limits) is actually a sound policy for life in general.  This may sound a bit off-topic, but I don't think so.  I universal truth that can actually be codified would be a real discovery, imho.</p>

<p>Question: is it a universally sound policy to use the "high" confidence probability in the formula: </p>

<p>$ActionValue = P(X) \cdot Value(X)$ &lt;- utility theory, reinforcement learning</p>

<p>So, $P(X)$ will have a confidence interval based on the number of samples.  You can optimistically just use $P(X) + confInterval / 2$ to get $Popt(X)$.  This seems to work well in practice.  Is this a universal truth?</p>

CC BY-SA 3.0
2011-08-13T03:07:54.013Z
what is the question?
Artem Kaznatcheev
<machine-learning>
Reinforcement Learning and Optimistic Decisions
Tom Jones
# Reinforcement Learning and Optimistic Decisions

(Asked by: username_0 on 2011-08-13)

Sutton recommended using an optimistic heuristic on decision making by considering the upper bound of the confidence range of an action-value. From my testing, it seems to work. I find this fascinating and wonder if it could be proven that optimistic thinking (within these limits) is actually a sound policy for life in general. This may sound a bit off-topic, but I don't think so. I universal truth that can actually be codified would be a real discovery, imho.

Question: is it a universally sound policy to use the "high" confidence probability in the formula:

$ActionValue = P(X) \cdot Value(X)$ <- utility theory, reinforcement learning

So, $P(X)$ will have a confidence interval based on the number of samples. You can optimistically just use $P(X) + confInterval / 2$ to get $Popt(X)$. This seems to work well in practice. Is this a universal truth?

**Tags:** machine-learning

> **Comments:**
> **username_1**: what is the question?  

---

## 1 answers

### Answer by username_2 on 2011-08-13. Score: 2

The main idea of following upper confidence bounds is minimizing regret, that is, the difference between the received reward and the maximum. The paper to read on this is: "Finite-time analysis of the multiarmed bandit problem" by Peter Auer, Nicoló Cesa-Bianchi, and Peter Fischer.

If we interpret life as a Markov Decision Process, then most reinforcement learning theories apply quite well. I.e. we want to keep an open mind, but we want to do what we think is best exponentially more often than exploratory actions.

Of course, interpreting life as an MDP is a bit of a stretch — foresight, for example, has to be interpreted as simulated experience, but I believe one can find quite a lot of wisdom in reinforcement learning and artificial intelligence in general.

> **Comments:**
> **username_3**: Yes, following the UCB policy gives $\tilde{O}(\sqrt{T})$ regret, which is asymptotically optimal for the MAB problem.  The MAB problem is a special case of reinforcement learning, with a time horizon of $1$.  

---

