  Hybrid Reinforcement Learning (RL), where an agent learns from both an
offline dataset and online explorations in an unknown environment, has garnered
significant recent interest. A crucial question posed by Xie et al. (2022) is
whether hybrid RL can improve upon the existing lower bounds established in
purely offline and purely online RL without relying on the single-policy
concentrability assumption. While Li et al. (2023) provided an affirmative
answer to this question in the tabular PAC RL case, the question remains
unsettled for both the regret-minimizing RL case and the non-tabular case.
  In this work, building upon recent advancements in offline RL and
reward-agnostic exploration, we develop computationally efficient algorithms
for both PAC and regret-minimizing RL with linear function approximation,
without single-policy concentrability. We demonstrate that these algorithms
achieve sharper error or regret bounds that are no worse than, and can improve
on, the optimal sample complexity in offline RL (the first algorithm, for PAC
RL) and online RL (the second algorithm, for regret-minimizing RL) in linear
Markov decision processes (MDPs), regardless of the quality of the behavior
policy. To our knowledge, this work establishes the tightest theoretical
guarantees currently available for hybrid RL in linear MDPs.
