  We consider a Markov chain $(x_n)$ whose kernel is indexed by a scaling
parameter $\gamma>0$, refered to as the step size. The aim is to analyze the
behavior of the Markov chain in the doubly asymptotic regime where $n\to\infty$
then $\gamma\to 0$. First, under mild assumptions on the so-called drift of the
Markov chain, we show that the interpolated process converges narrowly to the
solutions of a Differential Inclusion (DI) involving an upper semicontinuous
set-valued map with closed and convex values. Second, we provide verifiable
conditions which ensure the stability of the iterates. Third, by putting the
above results together, we establish the long run convergence of the iterates
as $\gamma\to 0$, to the Birkhoff center of the DI. The ergodic behavior of the
iterates is also provided. Application examples are investigated. We apply our
findings to 1) the problem of nonconvex proximal stochastic optimization and 2)
a fluid model of parallel queues.
