  In many real-world applications of reinforcement learning (RL), performing
actions requires consuming certain types of resources that are
non-replenishable in each episode. Typical applications include robotic control
with limited energy and video games with consumable items. In tasks with
non-replenishable resources, we observe that popular RL methods such as soft
actor critic suffer from poor sample efficiency. The major reason is that, they
tend to exhaust resources fast and thus the subsequent exploration is severely
restricted due to the absence of resources. To address this challenge, we first
formalize the aforementioned problem as a resource-restricted reinforcement
learning, and then propose a novel resource-aware exploration bonus (RAEB) to
make reasonable usage of resources. An appealing feature of RAEB is that, it
can significantly reduce unnecessary resource-consuming trials while
effectively encouraging the agent to explore unvisited states. Experiments
demonstrate that the proposed RAEB significantly outperforms state-of-the-art
exploration strategies in resource-restricted reinforcement learning
environments, improving the sample efficiency by up to an order of magnitude.
