  Low-Rank Markov Decision Processes (MDPs) have recently emerged as a
promising framework within the domain of reinforcement learning (RL), as they
allow for provably approximately correct (PAC) learning guarantees while also
incorporating ML algorithms for representation learning. However, current
methods for low-rank MDPs are limited in that they only consider finite action
spaces, and give vacuous bounds as $|\mathcal{A}| \to \infty$, which greatly
limits their applicability. In this work, we study the problem of extending
such methods to settings with continuous actions, and explore multiple concrete
approaches for performing this extension. As a case study, we consider the
seminal FLAMBE algorithm (Agarwal et al., 2020), which is a reward-agnostic
method for PAC RL with low-rank MDPs. We show that, without any modifications
to the algorithm, we obtain a similar PAC bound when actions are allowed to be
continuous. Specifically, when the model for transition functions satisfies a
H\"older smoothness condition w.r.t. actions, and either the policy class has a
uniformly bounded minimum density or the reward function is also H\"older
smooth, we obtain a polynomial PAC bound that depends on the order of
smoothness.
