  In neuroscience, one of the key behavioral tests for determining whether a
subject of study exhibits model-based behavior is to study its adaptiveness to
local changes in the environment. In reinforcement learning, however, recent
studies have shown that modern model-based agents display poor adaptivity to
such changes. The main reason for this is that modern agents are typically
designed to improve sample efficiency in single task settings and thus do not
take into account the challenges that can arise in other settings. In local
adaptation settings, one particularly important challenge is in quickly
building and maintaining a sufficiently accurate model after a local change.
This is challenging for deep model-based agents as their models and replay
buffers are monolithic structures lacking distribution shift handling
capabilities. In this study, we show that the conceptually simple idea of
partial models can allow deep model-based agents to overcome this challenge and
thus allow for building locally adaptive model-based agents. By modeling the
different parts of the state space through different models, the agent can not
only maintain a model that is accurate across the state space, but it can also
quickly adapt it in the presence of a local change in the environment. We
demonstrate this by showing that the use of partial models in agents such as
deep Dyna-Q, PlaNet and Dreamer can allow for them to effectively adapt to the
local changes in their environments.
