  Both entropy-minimizing and entropy-maximizing (curiosity) objectives for
unsupervised reinforcement learning (RL) have been shown to be effective in
different environments, depending on the environment's level of natural
entropy. However, neither method alone results in an agent that will
consistently learn intelligent behavior across environments. In an effort to
find a single entropy-based method that will encourage emergent behaviors in
any environment, we propose an agent that can adapt its objective online,
depending on the entropy conditions by framing the choice as a multi-armed
bandit problem. We devise a novel intrinsic feedback signal for the bandit,
which captures the agent's ability to control the entropy in its environment.
We demonstrate that such agents can learn to control entropy and exhibit
emergent behaviors in both high- and low-entropy regimes and can learn skillful
behaviors in benchmark tasks. Videos of the trained agents and summarized
findings can be found on our project page
https://sites.google.com/view/surprise-adaptive-agents
