  In many real-world applications of control system and robotics, linear
temporal logic (LTL) is a widely-used task specification language which has a
compositional grammar that naturally induces temporally extended behaviours
across tasks, including conditionals and alternative realizations. An important
problem in RL with LTL tasks is to learn task-conditioned policies which can
zero-shot generalize to new LTL instructions not observed in the training.
However, because symbolic observation is often lossy and LTL tasks can have
long time horizon, previous works can suffer from issues such as training
sampling inefficiency and infeasibility or sub-optimality of the found
solutions. In order to tackle these issues, this paper proposes a novel
multi-task RL algorithm with improved learning efficiency and optimality. To
achieve the global optimality of task completion, we propose to learn options
dependent on the future subgoals via a novel off-policy approach. In order to
propagate the rewards of satisfying future subgoals back more efficiently, we
propose to train a multi-step value function conditioned on the subgoal
sequence which is updated with Monte Carlo estimates of multi-step discounted
returns. In experiments on three different domains, we evaluate the LTL
generalization capability of the agent trained by the proposed method, showing
its advantage over previous representative methods.
