  We consider the scheduling problem in downlink wireless networks with
heterogeneous, Markov-modulated, ON/OFF channels. It is well-known that the
performance of scheduling over fading channels relies heavily on the accuracy
of the available Channel State Information (CSI), which is costly to acquire.
Thus, we consider the CSI acquisition via a practical ARQ-based feedback
mechanism whereby channel states are revealed at the end of only scheduled
users' transmissions. In the assumed presence of temporally-correlated channel
evolutions, the desired scheduler must optimally balance the
exploitation-exploration trade-off, whereby it schedules transmissions both to
exploit those channels with up-to-date CSI and to explore the current state of
those with outdated CSI.
  In earlier works, Whittle's Index Policy had been suggested as a
low-complexity and high-performance solution to this problem. However,
analyzing its performance in the typical scenario of statistically
heterogeneous channel state processes has remained elusive and challenging,
mainly because of the highly-coupled and complex dynamics it possesses. In this
work, we overcome these difficulties to rigorously establish the asymptotic
optimality properties of Whittle's Index Policy in the limiting regime of many
users. More specifically: (1) we prove the local optimality of Whittle's Index
Policy, provided that the initial state of the system is within a certain
neighborhood of a carefully selected state; (2) we then establish the global
optimality of Whittle's Index Policy under a recurrence assumption that is
verified numerically for our problem. These results establish that Whittle's
Index Policy possesses analytically provable optimality characteristics for
scheduling over heterogeneous and temporally-correlated channels.
