  We propose a new sequential decision-making setting, combining key aspects of
two established online learning problems with bandit feedback. The optimal
action to play at any given moment is contingent on an underlying changing
state which is not directly observable by the agent. Each state is associated
with a context distribution, possibly corrupted, allowing the agent to identify
the state. Furthermore, states evolve in a Markovian fashion, providing useful
information to estimate the current state via state history. In the proposed
problem setting, we tackle the challenge of deciding on which of the two
sources of information the agent should base its arm selection. We present an
algorithm that uses a referee to dynamically combine the policies of a
contextual bandit and a multi-armed bandit. We capture the time-correlation of
states through iteratively learning the action-reward transition model,
allowing for efficient exploration of actions. Our setting is motivated by
adaptive mobile health (mHealth) interventions. Users transition through
different, time-correlated, but only partially observable internal states,
determining their current needs. The side information associated with each
internal state might not always be reliable, and standard approaches solely
rely on the context risk of incurring high regret. Similarly, some users might
exhibit weaker correlations between subsequent states, leading to approaches
that solely rely on state transitions risking the same. We analyze our setting
and algorithm in terms of regret lower bound and upper bounds and evaluate our
method on simulated medication adherence intervention data and several
real-world data sets, showing improved empirical performance compared to
several popular algorithms.
