  This paper presents a model-free reinforcement learning (RL) algorithm to
synthesize a control policy that maximizes the satisfaction probability of
linear temporal logic (LTL) specifications. Due to the consideration of
environment and motion uncertainties, we model the robot motion as a
probabilistic labeled Markov decision process with unknown transition
probabilities and unknown probabilistic label functions. The LTL task
specification is converted to a limit deterministic generalized B\"uchi
automaton (LDGBA) with several accepting sets to maintain dense rewards during
learning. The novelty of applying LDGBA is to construct an embedded LDGBA
(E-LDGBA) by designing a synchronous tracking-frontier function, which enables
the record of non-visited accepting sets without increasing dimensional and
computational complexity. With appropriate dependent reward and discount
functions, rigorous analysis shows that any method that optimizes the expected
discount return of the RL-based approach is guaranteed to find the optimal
policy that maximizes the satisfaction probability of the LTL specifications. A
model-free RL-based motion planning strategy is developed to generate the
optimal policy in this paper. The effectiveness of the RL-based control
synthesis is demonstrated via simulation and experimental results.
