  Policy-based reinforcement learning algorithms are widely used in various
fields. Among them, mainstream policy optimization algorithms such as TRPO and
PPO introduce importance sampling into policy iteration, which allows the reuse
of historical data. However, this can also lead to a high variance of the
surrogate objective and indirectly affects the stability and convergence of the
algorithm. In this paper, we first derived an upper bound of the surrogate
objective variance, which can grow quadratically with the increase of the
surrogate objective. Next, we proposed the dropout technique to avoid the
excessive increase of the surrogate objective variance caused by importance
sampling. Then, we introduced a general reinforcement learning framework
applicable to mainstream policy optimization methods, and applied the dropout
technique to the PPO algorithm to obtain the D-PPO variant. Finally, we conduct
comparative experiments between D-PPO and PPO algorithms in the Atari 2600
environment, and the results show that D-PPO achieved significant performance
improvements compared to PPO, and effectively limited the excessive increase of
the surrogate objective variance during training.
