  In practical reinforcement learning (RL), the discount factor used for
estimating value functions often differs from that used for defining the
evaluation objective. In this work, we study the effect that this discrepancy
of discount factors has during learning, and discover a family of objectives
that interpolate value functions of two distinct discount factors. Our analysis
suggests new ways for estimating value functions and performing policy
optimization updates, which demonstrate empirical performance gains. This
framework also leads to new insights on commonly-used deep RL heuristic
modifications to policy optimization algorithms.
