  In this paper, we use concepts from supervisory control theory of discrete
event systems to propose a method to learn optimal control policies for a
finite-state Markov Decision Process (MDP) in which (only) certain sequences of
actions are deemed unsafe (respectively safe). We assume that the set of action
sequences that are deemed unsafe and/or safe are given in terms of a
finite-state automaton; and propose a supervisor that disables a subset of
actions at every state of the MDP so that the constraints on action sequence
are satisfied. Then we present a version of the Q-learning algorithm for
learning optimal policies in the presence of non-Markovian action-sequence and
state constraints, where we use the development of reward machines to handle
the state constraints. We illustrate the method using an example that captures
the utility of automata-based methods for non-Markovian state and action
specifications for reinforcement learning and show the results of simulations
in this setting.
