  Reinforcement learning (RL) in discrete action space is ubiquitous in
real-world applications, but its complexity grows exponentially with the
action-space dimension, making it challenging to apply existing on-policy
gradient based deep RL algorithms efficiently. To effectively operate in
multidimensional discrete action spaces, we construct a critic to estimate
action-value functions, apply it on correlated actions, and combine these
critic estimated action values to control the variance of gradient estimation.
We follow rigorous statistical analysis to design how to generate and combine
these correlated actions, and how to sparsify the gradients by shutting down
the contributions from certain dimensions. These efforts result in a new
discrete action on-policy RL algorithm that empirically outperforms related
on-policy algorithms relying on variance control techniques. We demonstrate
these properties on OpenAI Gym benchmark tasks, and illustrate how discretizing
the action space could benefit the exploration phase and hence facilitate
convergence to a better local optimal solution thanks to the flexibility of
discrete policy.
