  The difficulty of manually specifying reward functions has led to an interest
in using linear temporal logic (LTL) to express objectives for reinforcement
learning (RL). However, LTL has the downside that it is sensitive to small
perturbations in the transition probabilities, which prevents probably
approximately correct (PAC) learning without additional assumptions. Time
discounting provides a way of removing this sensitivity, while retaining the
high expressivity of the logic. We study the use of discounted LTL for policy
synthesis in Markov decision processes with unknown transition probabilities,
and show how to reduce discounted LTL to discounted-sum reward via a reward
machine when all discount factors are identical.
