  Deep reinforcement learning is successful in decision making for
sophisticated games, such as Atari, Go, etc. However, real-world decision
making often requires reasoning with partial information extracted from complex
visual observations. This paper presents Discriminative Particle Filter
Reinforcement Learning (DPFRL), a new reinforcement learning framework for
complex partial observations. DPFRL encodes a differentiable particle filter in
the neural network policy for explicit reasoning with partial observations over
time. The particle filter maintains a belief using learned discriminative
update, which is trained end-to-end for decision making. We show that using the
discriminative update instead of standard generative models results in
significantly improved performance, especially for tasks with complex visual
observations, because they circumvent the difficulty of modeling complex
observations that are irrelevant to decision making. In addition, to extract
features from the particle belief, we propose a new type of belief feature
based on the moment generating function. DPFRL outperforms state-of-the-art
POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and
in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark
introduced in this paper. Further, DPFRL performs well for visual navigation
with real-world data in the Habitat environment.
