  This paper studies restless multi-armed bandit (RMAB) problems with unknown
arm transition dynamics but with known correlated arm features. The goal is to
learn a model to predict transition dynamics given features, where the Whittle
index policy solves the RMAB problems using predicted transitions. However,
prior works often learn the model by maximizing the predictive accuracy instead
of final RMAB solution quality, causing a mismatch between training and
evaluation objectives. To address this shortcoming, we propose a novel approach
for decision-focused learning in RMAB that directly trains the predictive model
to maximize the Whittle index solution quality. We present three key
contributions: (i) we establish differentiability of the Whittle index policy
to support decision-focused learning; (ii) we significantly improve the
scalability of decision-focused learning approaches in sequential problems,
specifically RMAB problems; (iii) we apply our algorithm to a previously
collected dataset of maternal and child health to demonstrate its performance.
Indeed, our algorithm is the first for decision-focused learning in RMAB that
scales to real-world problem sizes.
