  Reinforcement Learning (RL) is a widely employed technique in decision-making
problems, encompassing two fundamental operations -- policy evaluation and
policy improvement. Enhancing learning efficiency remains a key challenge in
RL, with many efforts focused on using ensemble critics to boost policy
evaluation efficiency. However, when using multiple critics, the actor in the
policy improvement process can obtain different gradients. Previous studies
have combined these gradients without considering their disagreements.
Therefore, optimizing the policy improvement process is crucial to enhance
learning efficiency. This study focuses on investigating the impact of gradient
disagreements caused by ensemble critics on policy improvement. We introduce
the concept of uncertainty of gradient directions as a means to measure the
disagreement among gradients utilized in the policy improvement process.
Through measuring the disagreement among gradients, we find that transitions
with lower uncertainty of gradient directions are more reliable in the policy
improvement process. Building on this analysis, we propose a method called von
Mises-Fisher Experience Resampling (vMFER), which optimizes the policy
improvement process by resampling transitions and assigning higher confidence
to transitions with lower uncertainty of gradient directions. Our experiments
demonstrate that vMFER significantly outperforms the benchmark and is
particularly well-suited for ensemble structures in RL.
