  Value-function approximation methods that operate in batch mode have
foundational importance to reinforcement learning (RL). Finite sample
guarantees for these methods often crucially rely on two types of assumptions:
(1) mild distribution shift, and (2) representation conditions that are
stronger than realizability. However, the necessity ("why do we need them?")
and the naturalness ("when do they hold?") of such assumptions have largely
eluded the literature. In this paper, we revisit these assumptions and provide
theoretical results towards answering the above questions, and make steps
towards a deeper understanding of value-function approximation.
