  In an effort to better understand the different ways in which the discount
factor affects the optimization process in reinforcement learning, we designed
a set of experiments to study each effect in isolation. Our analysis reveals
that the common perception that poor performance of low discount factors is
caused by (too) small action-gaps requires revision. We propose an alternative
hypothesis that identifies the size-difference of the action-gap across the
state-space as the primary cause. We then introduce a new method that enables
more homogeneous action-gaps by mapping value estimates to a logarithmic space.
We prove convergence for this method under standard assumptions and demonstrate
empirically that it indeed enables lower discount factors for approximate
reinforcement-learning methods. This in turn allows tackling a class of
reinforcement-learning problems that are challenging to solve with traditional
methods.
