  In this paper we study the asymptotic behavior of a stochastic approximation
scheme on two timescales with set-valued drift functions and in the presence of
non-additive iterate-dependent Markov noise. It is shown that the recursion on
each timescale tracks the flow of a differential inclusion obtained by
averaging the set-valued drift function in the recursion with respect to a set
of measures which take into account both the averaging with respect to the
stationary distributions of the Markov noise terms and the interdependence
between the two recursions on different timescales. The framework studied in
this paper builds on the works of \it{A. Ramaswamy et al. }\rm by allowing for
the presence of non-additive iterate-dependent Markov noise. As an application,
we consider the problem of computing the optimum in a constrained convex
optimization problem where the objective function and the constraints are
averaged with respect to the stationary distribution of an underlying Markov
chain. Further the proposed scheme neither requires the differentiability of
the objective function nor the knowledge of the averaging measure.
