  Many sequential decision-making problems in communication networks can be
modeled as contextual bandit problems, which are natural extensions of the
well-known multi-armed bandit problem. In contextual bandit problems, at each
time, an agent observes some side information or context, pulls one arm and
receives the reward for that arm. We consider a stochastic formulation where
the context-reward tuples are independently drawn from an unknown distribution
in each trial. Motivated by networking applications, we analyze a setting where
the reward is a known non-linear function of the context and the chosen arm's
current state. We first consider the case of discrete and finite context-spaces
and propose DCB($\epsilon$), an algorithm that we prove, through a careful
analysis, yields regret (cumulative reward gap compared to a distribution-aware
genie) scaling logarithmically in time and linearly in the number of arms that
are not optimal for any context, improving over existing algorithms where the
regret scales linearly in the total number of arms. We then study continuous
context-spaces with Lipschitz reward functions and propose CCB($\epsilon,
\delta$), an algorithm that uses DCB($\epsilon$) as a subroutine.
CCB($\epsilon, \delta$) reveals a novel regret-storage trade-off that is
parametrized by $\delta$. Tuning $\delta$ to the time horizon allows us to
obtain sub-linear regret bounds, while requiring sub-linear storage. By
exploiting joint learning for all contexts we get regret bounds for
CCB($\epsilon, \delta$) that are unachievable by any existing contextual bandit
algorithm for continuous context-spaces. We also show similar performance
bounds for the unknown horizon case.
