  In our study, we delve into average-reward reinforcement learning with
general policy parametrization. Within this domain, current guarantees either
fall short with suboptimal guarantees or demand prior knowledge of mixing time.
To address these issues, we introduce Randomized Accelerated Natural Actor
Critic, a method that integrates Multi-level Monte-Carlo and Natural Actor
Critic. Our approach is the first to achieve global convergence rate of
$\tilde{\mathcal{O}}(1/\sqrt{T})$ without requiring knowledge of mixing time,
significantly surpassing the state-of-the-art bound of
$\tilde{\mathcal{O}}(1/T^{1/4})$.
