  We propose HyperAgent, a reinforcement learning (RL) algorithm based on the
hypermodel framework for exploration in RL. HyperAgent allows for the efficient
incremental approximation of posteriors associated with an optimal action-value
function ($Q^\star$) without the need for conjugacy and follows the greedy
policies w.r.t. these approximate posterior samples. We demonstrate that
HyperAgent offers robust performance in large-scale deep RL benchmarks. It can
solve Deep Sea hard exploration problems with episodes that optimally scale
with problem size and exhibits significant efficiency gains in the Atari suite.
Implementing HyperAgent requires minimal code addition to well-established deep
RL frameworks like DQN. We theoretically prove that, under tabular assumptions,
HyperAgent achieves logarithmic per-step computational complexity while
attaining sublinear regret, matching the best known randomized tabular RL
algorithm.
