  As the third-generation neural networks, Spiking Neural Networks (SNNs) have
great potential on neuromorphic hardware because of their high
energy-efficiency. However, Deep Spiking Reinforcement Learning (DSRL), i.e.,
the Reinforcement Learning (RL) based on SNNs, is still in its preliminary
stage due to the binary output and the non-differentiable property of the
spiking function. To address these issues, we propose a Deep Spiking Q-Network
(DSQN) in this paper. Specifically, we propose a directly-trained deep spiking
reinforcement learning architecture based on the Leaky Integrate-and-Fire (LIF)
neurons and Deep Q-Network (DQN). Then, we adapt a direct spiking learning
algorithm for the Deep Spiking Q-Network. We further demonstrate the advantages
of using LIF neurons in DSQN theoretically. Comprehensive experiments have been
conducted on 17 top-performing Atari games to compare our method with the
state-of-the-art conversion method. The experimental results demonstrate the
superiority of our method in terms of performance, stability, robustness and
energy-efficiency. To the best of our knowledge, our work is the first one to
achieve state-of-the-art performance on multiple Atari games with the
directly-trained SNN.
