  Motivated from Bertsekas' recent study on policy iteration (PI) for solving
the problems of infinite-horizon discounted Markov decision processes (MDPs) in
an on-line setting, we develop an off-line PI integrated with a multi-policy
improvement method of policy switching and then adapt its asynchronous variant
into on-line PI algorithm that generates a sequence of policies over time. The
current policy is updated into the next policy by switching the action only at
the current state while ensuring the monotonicity of the value functions of the
policies in the sequence. Depending on MDP's state-transition structure, the
sequence converges in a finite time to an optimal policy for an associated
local MDP. When MDP is communicating, the sequence converges to an optimal
policy for the original MDP.
