  We study the interplay between the data distribution and Q-learning-based
algorithms with function approximation. We provide a unified theoretical and
empirical analysis as to how different properties of the data distribution
influence the performance of Q-learning-based algorithms. We connect different
lines of research, as well as validate and extend previous results. We start by
reviewing theoretical bounds on the performance of approximate dynamic
programming algorithms. We then introduce a novel four-state MDP specifically
tailored to highlight the impact of the data distribution in the performance of
Q-learning-based algorithms with function approximation, both online and
offline. Finally, we experimentally assess the impact of the data distribution
properties on the performance of two offline Q-learning-based algorithms under
different environments. According to our results: (i) high entropy data
distributions are well-suited for learning in an offline manner; and (ii) a
certain degree of data diversity (data coverage) and data quality (closeness to
optimal policy) are jointly desirable for offline learning.
