  We propose the k-Shortest-Path (k-SP) constraint: a novel constraint on the
agent's trajectory that improves the sample efficiency in sparse-reward MDPs.
We show that any optimal policy necessarily satisfies the k-SP constraint.
Notably, the k-SP constraint prevents the policy from exploring state-action
pairs along the non-k-SP trajectories (e.g., going back and forth). However, in
practice, excluding state-action pairs may hinder the convergence of RL
algorithms. To overcome this, we propose a novel cost function that penalizes
the policy violating SP constraint, instead of completely excluding it. Our
numerical experiment in a tabular RL setting demonstrates that the SP
constraint can significantly reduce the trajectory space of policy. As a
result, our constraint enables more sample efficient learning by suppressing
redundant exploration and exploitation. Our experiments on MiniGrid, DeepMind
Lab, Atari, and Fetch show that the proposed method significantly improves
proximal policy optimization (PPO) and outperforms existing novelty-seeking
exploration methods including count-based exploration even in continuous
control tasks, indicating that it improves the sample efficiency by preventing
the agent from taking redundant actions.
