  In this paper we study the asymptotic behavior of stochastic approximation
schemes with set-valued drift function and non-additive iterate-dependent
Markov noise. We show that a linearly interpolated trajectory of such a
recursion is an asymptotic pseudotrajectory for the flow of a limiting
differential inclusion obtained by averaging the set-valued drift function of
the recursion w.r.t. the stationary distributions of the Markov noise. The
limit set theorem in [1] is then used to characterize the limit sets of the
recursion in terms of the dynamics of the limiting differential inclusion. We
then state two variants of the Markov noise assumption under which the analysis
of the recursion is similar to the one presented in this paper. Scenarios where
our recursion naturally appears are presented as applications. These include
controlled stochastic approximation, subgradient descent, approximate drift
problem and analysis of discontinuous dynamics all in the presence of
non-additive iterate-dependent Markov noise.
