  The vast majority of Reinforcement Learning methods is largely impacted by
the computation effort and data requirements needed to obtain effective
estimates of action-value functions, which in turn determine the quality of the
overall performance and the sample-efficiency of the learning procedure.
Typically, action-value functions are estimated through an iterative scheme
that alternates the application of an empirical approximation of the Bellman
operator and a subsequent projection step onto a considered function space. It
has been observed that this scheme can be potentially generalized to carry out
multiple iterations of the Bellman operator at once, benefiting the underlying
learning algorithm. However, till now, it has been challenging to effectively
implement this idea, especially in high-dimensional problems. In this paper, we
introduce iterated $Q$-Network (iQN), a novel principled approach that enables
multiple consecutive Bellman updates by learning a tailored sequence of
action-value functions where each serves as the target for the next. We show
that iQN is theoretically grounded and that it can be seamlessly used in
value-based and actor-critic methods. We empirically demonstrate the advantages
of iQN in Atari $2600$ games and MuJoCo continuous control problems.
