  A standard assumption adopted in the multi-armed bandit (MAB) framework is
that the mean rewards are constant over time. This assumption can be
restrictive in the business world as decision-makers often face an evolving
environment where the mean rewards are time-varying. In this paper, we consider
a non-stationary MAB model with $K$ arms whose mean rewards vary over time in a
periodic manner. The unknown periods can be different across arms and scale
with the length of the horizon $T$ polynomially. We propose a two-stage policy
that combines the Fourier analysis with a confidence-bound-based learning
procedure to learn the periods and minimize the regret. In stage one, the
policy correctly estimates the periods of all arms with high probability. In
stage two, the policy explores the periodic mean rewards of arms using the
periods estimated in stage one and exploits the optimal arm in the long run. We
show that our learning policy incurs a regret upper bound
$\tilde{O}(\sqrt{T\sum_{k=1}^K T_k})$ where $T_k$ is the period of arm $k$.
Moreover, we establish a general lower bound $\Omega(\sqrt{T\max_{k}\{ T_k\}})$
for any policy. Therefore, our policy is near-optimal up to a factor of
$\sqrt{K}$.
