  Many reinforcement learning (RL) environments consist of independent entities
that interact sparsely. In such environments, RL agents have only limited
influence over other entities in any particular situation. Our idea in this
work is that learning can be efficiently guided by knowing when and what the
agent can influence with its actions. To achieve this, we introduce a measure
of \emph{situation-dependent causal influence} based on conditional mutual
information and show that it can reliably detect states of influence. We then
propose several ways to integrate this measure into RL algorithms to improve
exploration and off-policy learning. All modified algorithms show strong
increases in data efficiency on robotic manipulation tasks.
