  Most bandit policies are designed to either minimize regret in any problem
instance, making very few assumptions about the underlying environment, or in a
Bayesian sense, assuming a prior distribution over environment parameters. The
former are often too conservative in practical settings, while the latter
require assumptions that are hard to verify in practice. We study bandit
problems that fall between these two extremes, where the learning agent has
access to sampled bandit instances from an unknown prior distribution
$\mathcal{P}$ and aims to achieve high reward on average over the bandit
instances drawn from $\mathcal{P}$. This setting is of a particular importance
because it lays foundations for meta-learning of bandit policies and reflects
more realistic assumptions in many practical domains. We propose the use of
parameterized bandit policies that are differentiable and can be optimized
using policy gradients. This provides a broadly applicable framework that is
easy to implement. We derive reward gradients that reflect the structure of
bandit problems and policies, for both non-contextual and contextual settings,
and propose a number of interesting policies that are both differentiable and
have low regret. Our algorithmic and theoretical contributions are supported by
extensive experiments that show the importance of baseline subtraction, learned
biases, and the practicality of our approach on a range problems.
