  Efficient exploration is a key challenge in contextual bandits due to the
large size of their action space, where uninformed exploration can result in
computational and statistical inefficiencies. Fortunately, the rewards of
actions are often correlated and this can be leveraged to explore them
efficiently. In this work, we capture such correlations using pre-trained
diffusion models; upon which we design diffusion Thompson sampling (dTS). Both
theoretical and algorithmic foundations are developed for dTS, and empirical
evaluation also shows its favorable performance.
