  Reinforcement learning is a powerful approach to learn behaviour through
interactions with an environment. However, behaviours are usually learned in a
purely reactive fashion, where an appropriate action is selected based on an
observation. In this form, it is challenging to learn when it is necessary to
execute new decisions. This makes learning inefficient, especially in
environments that need various degrees of fine and coarse control. To address
this, we propose a proactive setting in which the agent not only selects an
action in a state but also for how long to commit to that action. Our TempoRL
approach introduces skip connections between states and learns a skip-policy
for repeating the same action along these skips. We demonstrate the
effectiveness of TempoRL on a variety of traditional and deep RL environments,
showing that our approach is capable of learning successful policies up to an
order of magnitude faster than vanilla Q-learning.
