  Policy-based reinforcement learning methods suffer from the policy collapse
problem. We find valued-based reinforcement learning methods with
{\epsilon}-greedy mechanism are capable of enjoying three characteristics,
Closed-form Diversity, Objective-invariant Exploration and Adaptive Trade-off,
which help value-based methods avoid the policy collapse problem. However,
there does not exist a parallel mechanism for policy-based methods that
achieves all three characteristics. In this paper, we propose an entropy
regularization free mechanism that is designed for policy-based methods, which
achieves Closed-form Diversity, Objective-invariant Exploration and Adaptive
Trade-off. Our experiments show that our mechanism is super sample-efficient
for policy-based methods and boosts a policy-based baseline to a new
State-Of-The-Art on Arcade Learning Environment.
