  To achieve sample efficiency in reinforcement learning (RL), it necessitates
efficiently exploring the underlying environment. Under the offline setting,
addressing the exploration challenge lies in collecting an offline dataset with
sufficient coverage. Motivated by such a challenge, we study the reward-free RL
problem, where an agent aims to thoroughly explore the environment without any
pre-specified reward function. Then, given any extrinsic reward, the agent
computes the policy via a planning algorithm with offline data collected in the
exploration phase. Moreover, we tackle this problem under the context of
function approximation, leveraging powerful function approximators.
  Specifically, we propose to explore via an optimistic variant of the
value-iteration algorithm incorporating kernel and neural function
approximations, where we adopt the associated exploration bonus as the
exploration reward. Moreover, we design exploration and planning algorithms for
both single-agent MDPs and zero-sum Markov games and prove that our methods can
achieve $\widetilde{\mathcal{O}}(1 /\varepsilon^2)$ sample complexity for
generating a $\varepsilon$-suboptimal policy or $\varepsilon$-approximate Nash
equilibrium when given an arbitrary extrinsic reward. To the best of our
knowledge, we establish the first provably efficient reward-free RL algorithm
with kernel and neural function approximators.
