  We propose to learn to distinguish reversible from irreversible actions for
better informed decision-making in Reinforcement Learning (RL). From
theoretical considerations, we show that approximate reversibility can be
learned through a simple surrogate task: ranking randomly sampled trajectory
events in chronological order. Intuitively, pairs of events that are always
observed in the same order are likely to be separated by an irreversible
sequence of actions. Conveniently, learning the temporal order of events can be
done in a fully self-supervised way, which we use to estimate the reversibility
of actions from experience, without any priors. We propose two different
strategies that incorporate reversibility in RL agents, one strategy for
exploration (RAE) and one strategy for control (RAC). We demonstrate the
potential of reversibility-aware agents in several environments, including the
challenging Sokoban game. In synthetic tasks, we show that we can learn control
policies that never fail and reduce to zero the side-effects of interactions,
even without access to the reward function.
