  We study contextual bandits with budget and time constraints, referred to as
constrained contextual bandits.The time and budget constraints significantly
complicate the exploration and exploitation tradeoff because they introduce
complex coupling among contexts over time.Such coupling effects make it
difficult to obtain oracle solutions that assume known statistics of bandits.
To gain insight, we first study unit-cost systems with known context
distribution. When the expected rewards are known, we develop an approximation
of the oracle, referred to Adaptive-Linear-Programming (ALP), which achieves
near-optimality and only requires the ordering of expected rewards. With these
highly desirable features, we then combine ALP with the upper-confidence-bound
(UCB) method in the general case where the expected rewards are unknown {\it a
priori}. We show that the proposed UCB-ALP algorithm achieves logarithmic
regret except for certain boundary cases. Further, we design algorithms and
obtain similar regret analysis results for more general systems with unknown
context distribution and heterogeneous costs. To the best of our knowledge,
this is the first work that shows how to achieve logarithmic regret in
constrained contextual bandits. Moreover, this work also sheds light on the
study of computationally efficient algorithms for general constrained
contextual bandits.
