  We study a decentralized variant of stochastic approximation, a data-driven
approach for finding the root of an operator under noisy measurements. A
network of agents, each with its own operator and data observations,
cooperatively find the fixed point of the aggregate operator over a
decentralized communication graph. Our main contribution is to provide a
finite-time analysis of this decentralized stochastic approximation method when
the data observed at each agent are sampled from a Markov process; this lack of
independence makes the iterates biased and (potentially) unbounded. Under
fairly standard assumptions, we show that the convergence rate of the proposed
method is essentially the same as if the samples were independent, differing
only by a log factor that accounts for the mixing time of the Markov processes.
The key idea in our analysis is to introduce a novel Razumikhin-Lyapunov
function, motivated by the one used in analyzing the stability of delayed
ordinary differential equations. We also discuss applications of the proposed
method on a number of interesting learning problems in multi-agent systems.
