  Contextual bandit and reinforcement learning algorithms have been
successfully used in various interactive learning systems such as online
advertising, recommender systems, and dynamic pricing. However, they have yet
to be widely adopted in high-stakes application domains, such as healthcare.
One reason may be that existing approaches assume that the underlying
mechanisms are static in the sense that they do not change over different
environments. In many real-world systems, however, the mechanisms are subject
to shifts across environments which may invalidate the static environment
assumption. In this paper, we take a step toward tackling the problem of
environmental shifts considering the framework of offline contextual bandits.
We view the environmental shift problem through the lens of causality and
propose multi-environment contextual bandits that allow for changes in the
underlying mechanisms. We adopt the concept of invariance from the causality
literature and introduce the notion of policy invariance. We argue that policy
invariance is only relevant if unobserved variables are present and show that,
in that case, an optimal invariant policy is guaranteed to generalize across
environments under suitable assumptions. Our results establish concrete
connections among causality, invariance, and contextual bandits.
