  We extend Bayesian multi-armed bandit (MAB) algorithms beyond their original
setting by making use of sequential Monte Carlo (SMC) methods.
  A MAB is a sequential decision making problem where the goal is to learn a
policy that maximizes long term payoff, where only the reward of the executed
action is observed. In the stochastic MAB, the reward for each action is
generated from an unknown distribution, often assumed to be stationary. To
decide which action to take next, a MAB agent must learn the characteristics of
the unknown reward distribution, e.g., compute its sufficient statistics.
However, closed-form expressions for these statistics are analytically
intractable except for simple, stationary cases.
  We here utilize SMC for estimation of the statistics Bayesian MAB agents
compute, and devise flexible policies that can address a rich class of bandit
problems: i.e., MABs with nonlinear, stateless- and context-dependent reward
distributions that evolve over time. We showcase how non-stationary bandits,
where time dynamics are modeled via linear dynamical systems, can be
successfully addressed by SMC-based Bayesian bandit agents. We empirically
demonstrate good regret performance of the proposed SMC-based bandit policies
in several MAB scenarios that have remained elusive, i.e., in non-stationary
bandits with nonlinear rewards.
