  Reinforcement Learning (RL) algorithms can learn robotic control tasks from
visual observations, but they often require a large amount of data, especially
when the visual scene is complex and unstructured. In this paper, we explore
how the agent's knowledge of its shape can improve the sample efficiency of
visual RL methods. We propose a novel method, Disentangled Environment and
Agent Representations (DEAR), that uses the segmentation mask of the agent as
supervision to learn disentangled representations of the environment and the
agent through feature separation constraints. Unlike previous approaches, DEAR
does not require reconstruction of visual observations. These representations
are then used as an auxiliary loss to the RL objective, encouraging the agent
to focus on the relevant features of the environment. We evaluate DEAR on two
challenging benchmarks: Distracting DeepMind control suite and Franka Kitchen
manipulation tasks. Our findings demonstrate that DEAR surpasses
state-of-the-art methods in sample efficiency, achieving comparable or superior
performance with reduced parameters. Our results indicate that integrating
agent knowledge into visual RL methods has the potential to enhance their
learning efficiency and robustness.
