  In deep reinforcement learning applications, maximizing discounted reward is
often employed instead of maximizing total reward to ensure the convergence and
stability of algorithms, even though the performance metric for evaluating the
policy remains the total reward. However, the optimal policies corresponding to
these two objectives may not always be consistent. To address this issue, we
analyzed the suboptimality of the policy obtained through maximizing discounted
reward in relation to the policy that maximizes total reward and identified the
influence of hyperparameters. Additionally, we proposed sufficient conditions
for aligning the optimal policies of these two objectives under various
settings. The primary contributions are as follows: We theoretically analyzed
the factors influencing performance when using discounted reward as a proxy for
total reward, thereby enhancing the theoretical understanding of this scenario.
Furthermore, we developed methods to align the optimal policies of the two
objectives in certain situations, which can improve the performance of
reinforcement learning algorithms.
