  Value function based reinforcement learning (RL) algorithms, for example,
$Q$-learning, learn optimal policies from datasets of actions, rewards, and
state transitions. However, when the underlying state transition dynamics are
stochastic and evolve on a high-dimensional space, generating independent and
identically distributed (IID) data samples for creating these datasets poses a
significant challenge due to the intractability of the associated normalizing
integral. In these scenarios, Hamiltonian Monte Carlo (HMC) sampling offers a
computationally tractable way to generate data for training RL algorithms. In
this paper, we introduce a framework, called \textit{Hamiltonian $Q$-Learning},
that demonstrates, both theoretically and empirically, that $Q$ values can be
learned from a dataset generated by HMC samples of actions, rewards, and state
transitions. Furthermore, to exploit the underlying low-rank structure of the
$Q$ function, Hamiltonian $Q$-Learning uses a matrix completion algorithm for
reconstructing the updated $Q$ function from $Q$ value updates over a much
smaller subset of state-action pairs. Thus, by providing an efficient way to
apply $Q$-learning in stochastic, high-dimensional settings, the proposed
approach broadens the scope of RL algorithms for real-world applications.
