  We introduce a class of learning problems where the agent is presented with a
series of tasks. Intuitively, if there is relation among those tasks, then the
information gained during execution of one task has value for the execution of
another task. Consequently, the agent is intrinsically motivated to explore its
environment beyond the degree necessary to solve the current task it has at
hand. We develop a decision theoretic setting that generalises standard
reinforcement learning tasks and captures this intuition. More precisely, we
consider a multi-stage stochastic game between a learning agent and an
opponent. We posit that the setting is a good model for the problem of
life-long learning in uncertain environments, where while resources must be
spent learning about currently important tasks, there is also the need to
allocate effort towards learning about aspects of the world which are not
relevant at the moment. This is due to the fact that unpredictable future
events may lead to a change of priorities for the decision maker. Thus, in some
sense, the model "explains" the necessity of curiosity. Apart from introducing
the general formalism, the paper provides algorithms. These are evaluated
experimentally in some exemplary domains. In addition, performance bounds are
proven for some cases of this problem.
