  This paper presents a convex optimization approach to control the density
distribution of autonomous mobile agents with two control modes: ON and OFF.
The main new characteristic distinguishing this model from standard Markov
decision models is the existence of the ON control mode and its observed
actions. When an agent is in the ON mode, it can measure the instantaneous
outcome of one of the actions corresponding to the ON mode and decides whether
it should take this action or not based on this new observation. If it does not
take this action, it switches to the OFF mode where it transitions to the next
state based on a predetermined set of transitional probabilities, without
making any additional observations. In this decision-making model, each agent
acts autonomously according to an ON/OFF decision policy, and the discrete
probability distribution for the agent's state evolves according to a
discrete-time Markov chain that is a linear function of the stochastic
environment (i.e., transition probabilities) and the ON/OFF decision policy.
The relevant policy synthesis is formulated as a convex optimization problem
where hard safety and convergence constraints are imposed on the resulting
Markov matrix. We first consider the case where the ON mode has a single action
and the OFF mode has deterministic transitions (rather than stochastic) to
demonstrate the model and the ideas behind our approach, which is then
generalized to the case where ON mode has multiple actions and OFF mode has
stochastic transitions.
