  Recent studies in reinforcement learning (RL) have made significant progress
by leveraging function approximation to alleviate the sample complexity hurdle
for better performance. Despite the success, existing provably efficient
algorithms typically rely on the accessibility of immediate feedback upon
taking actions. The failure to account for the impact of delay in observations
can significantly degrade the performance of real-world systems due to the
regret blow-up. In this work, we tackle the challenge of delayed feedback in RL
with linear function approximation by employing posterior sampling, which has
been shown to empirically outperform the popular UCB algorithms in a wide range
of regimes. We first introduce Delayed-PSVI, an optimistic value-based
algorithm that effectively explores the value function space via noise
perturbation with posterior sampling. We provide the first analysis for
posterior sampling algorithms with delayed feedback in RL and show our
algorithm achieves $\widetilde{O}(\sqrt{d^3H^3 T} + d^2H^2 E[\tau])$ worst-case
regret in the presence of unknown stochastic delays. Here $E[\tau]$ is the
expected delay. To further improve its computational efficiency and to expand
its applicability in high-dimensional RL problems, we incorporate a
gradient-based approximate sampling scheme via Langevin dynamics for
Delayed-LPSVI, which maintains the same order-optimal regret guarantee with
$\widetilde{O}(dHK)$ computational cost. Empirical evaluations are performed to
demonstrate the statistical and computational efficacy of our algorithms.
