  In this paper, we propose a model-free reinforcement learning method to
synthesize control policies for motion planning problems with continuous states
and actions. The robot is modelled as a labeled discrete-time Markov decision
process (MDP) with continuous state and action spaces. Linear temporal logics
(LTL) are used to specify high-level tasks. We then train deep neural networks
to approximate the value function and policy using an actor-critic
reinforcement learning method. The LTL specification is converted into an
annotated limit-deterministic B\"uchi automaton (LDBA) for continuously shaping
the reward so that dense rewards are available during training. A na\"ive way
of solving a motion planning problem with LTL specifications using
reinforcement learning is to sample a trajectory and then assign a high reward
for training if the trajectory satisfies the entire LTL formula. However, the
sampling complexity needed to find such a trajectory is too high when we have a
complex LTL formula for continuous state and action spaces. As a result, it is
very unlikely that we get enough reward for training if all sample trajectories
start from the initial state in the automata. In this paper, we propose a
method that samples not only an initial state from the state space, but also an
arbitrary state in the automata at the beginning of each training episode. We
test our algorithm in simulation using a car-like robot and find out that our
method can learn policies for different working configurations and LTL
specifications successfully.
