  Mobile health leverages personalized, contextually-tailored interventions
optimized through bandit and reinforcement learning algorithms. Despite its
promise, challenges like participant heterogeneity, nonstationarity, and
nonlinearity in rewards hinder algorithm performance. We propose a robust
contextual bandit algorithm, termed "DML-TS-NNR", that simultaneously addresses
these challenges via (1) modeling the differential reward with user- and
time-specific incidental parameters, (2) network cohesion penalties, and (3)
debiased machine learning for flexible estimation of baseline rewards. We
establish a high-probability regret bound that depends solely on the dimension
of the differential reward model. This feature enables us to achieve robust
regret bounds even when the baseline reward is highly complex. We demonstrate
the superior performance of the DML-TS-NNR algorithm in a simulation and two
off-policy evaluation studies.
