  We study a class of reinforcement learning (RL) tasks where the objective of
the agent is to accomplish temporally extended goals. In this setting, a common
approach is to represent the tasks as deterministic finite automata (DFA) and
integrate them into the state-space for RL algorithms. However, while these
machines model the reward function, they often overlook the causal knowledge
about the environment. To address this limitation, we propose the
Temporal-Logic-based Causal Diagram (TL-CD) in RL, which captures the temporal
causal relationships between different properties of the environment. We
exploit the TL-CD to devise an RL algorithm in which an agent requires
significantly less exploration of the environment. To this end, based on a
TL-CD and a task DFA, we identify configurations where the agent can determine
the expected rewards early during an exploration. Through a series of case
studies, we demonstrate the benefits of using TL-CDs, particularly the faster
convergence of the algorithm to an optimal policy due to reduced exploration of
the environment.
