  In recent years, deep reinforcement learning has been shown to be adept at
solving sequential decision processes with high-dimensional state spaces such
as in the Atari games. Many reinforcement learning problems, however, involve
high-dimensional discrete action spaces as well as high-dimensional state
spaces. This paper considers entropy bonus, which is used to encourage
exploration in policy gradient. In the case of high-dimensional action spaces,
calculating the entropy and its gradient requires enumerating all the actions
in the action space and running forward and backpropagation for each action,
which may be computationally infeasible. We develop several novel unbiased
estimators for the entropy bonus and its gradient. We apply these estimators to
several models for the parameterized policies, including Independent Sampling,
CommNet, Autoregressive with Modified MDP, and Autoregressive with LSTM.
Finally, we test our algorithms on two environments: a multi-hunter
multi-rabbit grid game and a multi-agent multi-arm bandit problem. The results
show that our entropy estimators substantially improve performance with
marginal additional computational cost.
