  Experience replay serves as a key component in the success of online
reinforcement learning (RL). Prioritized experience replay (PER) reweights
experiences by the temporal difference (TD) error empirically enhancing the
performance. However, few works have explored the motivation of using TD error.
In this work, we provide an alternative perspective on TD-error-based
reweighting. We show the connections between the experience prioritization and
occupancy optimization. By using a regularized RL objective with $f-$divergence
regularizer and employing its dual form, we show that an optimal solution to
the objective is obtained by shifting the distribution of off-policy data in
the replay buffer towards the on-policy optimal distribution using
TD-error-based occupancy ratios. Our derivation results in a new pipeline of TD
error prioritization. We specifically explore the KL divergence as the
regularizer and obtain a new form of prioritization scheme, the regularized
optimal experience replay (ROER). We evaluate the proposed prioritization
scheme with the Soft Actor-Critic (SAC) algorithm in continuous control MuJoCo
and DM Control benchmark tasks where our proposed scheme outperforms baselines
in 6 out of 11 tasks while the results of the rest match with or do not deviate
far from the baselines. Further, using pretraining, ROER achieves noticeable
improvement on difficult Antmaze environment where baselines fail, showing
applicability to offline-to-online fine-tuning. Code is available at
\url{https://github.com/XavierChanglingLi/Regularized-Optimal-Experience-Replay}.
