  In this paper, we consider the problem of adjusting the exploration rate when
using value-of-information-based exploration. We do this by converting the
value-of-information optimization into a problem of finding equilibria of a
flow for a changing exploration rate. We then develop an efficient
path-following scheme for converging to these equilibria and hence uncovering
optimal action-selection policies. Under this scheme, the exploration rate is
automatically adapted according to the agent's experiences. Global convergence
is theoretically assured.
  We first evaluate our exploration-rate adaptation on the Nintendo GameBoy
games Centipede and Millipede. We demonstrate aspects of the search process,
like that it yields a hierarchy of state abstractions. We also show that our
approach returns better policies in fewer episodes than conventional search
strategies relying on heuristic, annealing-based exploration-rate adjustments.
We then illustrate that these trends hold for deep, value-of-information-based
agents that learn to play ten simple games and over forty more complicated
games for the Nintendo GameBoy system. Performance either near or well above
the level of human play is observed.
