  Contextual multi-armed bandit problems arise frequently in important
industrial applications. Existing solutions model the context either linearly,
which enables uncertainty driven (principled) exploration, or non-linearly, by
using epsilon-greedy exploration policies. Here we present a deep learning
framework for contextual multi-armed bandits that is both non-linear and
enables principled exploration at the same time. We tackle the exploration vs.
exploitation trade-off through Thompson sampling by exploiting the connection
between inference time dropout and sampling from the posterior over the weights
of a Bayesian neural network. In order to adjust the level of exploration
automatically as more data is made available to the model, the dropout rate is
learned rather than considered a hyperparameter. We demonstrate that our
approach substantially reduces regret on two tasks (the UCI Mushroom task and
the Casino Parity task) when compared to 1) non-contextual bandits, 2)
epsilon-greedy deep contextual bandits, and 3) fixed dropout rate deep
contextual bandits. Our approach is currently being applied to marketing
optimization problems at HubSpot.
