  This paper proposes a comprehensive hierarchical control framework for
autonomous decision-making arising in robotics and autonomous systems. In a
typical hierarchical control architecture, high-level decision making is often
characterised by discrete state and decision/control sets. However, a rational
decision is usually affected by not only the discrete states of the autonomous
system, but also the underlying continuous dynamics even the evolution of its
operational environment. This paper proposes a holistic and comprehensive
design process and framework for this type of challenging problems, from new
modelling and design problem formulation to control design and stability
analysis. It addresses the intricate interplay between traditional continuous
systems dynamics utilized at the low levels for control design and discrete
Markov decision processes (MDP) for facilitating high-level decision making. We
model the decision making system in complex environments as a hybrid system
consisting of a controlled MDP and autonomous (i.e. uncontrolled) continuous
dynamics. Consequently, the new formulation is called as hybrid Markov decision
process (HMDP). The design problem is formulated with a focus on ensuring both
safety and optimality while taking into account the influence of both the
discrete and continuous state variables of different levels. With the help of
the model predictive control (MPC) concept, a decision maker design scheme is
proposed for the proposed hybrid decision making model. By carefully designing
key ingredients involved in this scheme, it is shown that the recursive
feasibility and stability of the proposed autonomous decision making scheme are
guaranteed. The proposed framework is applied to develop an autonomous lane
changing system for intelligent vehicles.
