  Exploration policies in Bayesian bandits maximize the average reward over
problem instances drawn from some distribution $\mathcal{P}$. In this work, we
learn such policies for an unknown distribution $\mathcal{P}$ using samples
from $\mathcal{P}$. Our approach is a form of meta-learning and exploits
properties of $\mathcal{P}$ without making strong assumptions about its form.
To do this, we parameterize our policies in a differentiable way and optimize
them by policy gradients, an approach that is general and easy to implement. We
derive effective gradient estimators and introduce novel variance reduction
techniques. We also analyze and experiment with various bandit policy classes,
including neural networks and a novel softmax policy. The latter has regret
guarantees and is a natural starting point for our optimization. Our
experiments show the versatility of our approach. We also observe that neural
network policies can learn implicit biases expressed only through the sampled
instances.
