  We propose a model for learning with bandit feedback while accounting for
deterministically evolving and unobservable states that we call Bandits with
Deterministically Evolving States ($B$-$DES$). The workhorse applications of
our model are learning for recommendation systems and learning for online ads.
In both cases, the reward that the algorithm obtains at each round is a
function of the short-term reward of the action chosen and how "healthy" the
system is (i.e., as measured by its state). For example, in recommendation
systems, the reward that the platform obtains from a user's engagement with a
particular type of content depends not only on the inherent features of the
specific content, but also on how the user's preferences have evolved as a
result of interacting with other types of content on the platform. Our general
model accounts for the different rate $\lambda \in [0,1]$ at which the state
evolves (e.g., how fast a user's preferences shift as a result of previous
content consumption) and encompasses standard multi-armed bandits as a special
case. The goal of the algorithm is to minimize a notion of regret against the
best-fixed sequence of arms pulled, which is significantly harder to attain
compared to standard benchmark of the best-fixed action in hindsight. We
present online learning algorithms for any possible value of the evolution rate
$\lambda$ and we show the robustness of our results to various model
misspecifications.
