  In reinforcement learning (RL) with experience replay, experiences stored in
a replay buffer influence the RL agent's performance. Information about the
influence is valuable for various purposes, including experience cleansing and
analysis. One method for estimating the influence of individual experiences is
agent comparison, but it is prohibitively expensive when there is a large
number of experiences. In this paper, we present PI+ToD as a method for
efficiently estimating the influence of experiences. PI+ToD is a policy
iteration that efficiently estimates the influence of experiences by utilizing
turn-over dropout. We demonstrate the efficiency of PI+ToD with experiments in
MuJoCo environments.
