  Planning is a data efficient decision-making strategy where an agent selects
candidate actions by exploring possible future states. To simulate future
states when there is a high-dimensional action space, the knowledge of one's
decision making strategy must be used to limit the number of actions to be
explored. We refer to the model used to simulate one's decisions as the agent's
self-model. While self-models are implicitly used widely in conjunction with
world models to plan actions, it remains unclear how self-models should be
designed. Inspired by current reinforcement learning approaches and
neuroscience, we explore the benefits and limitations of using a distilled
policy network as the self-model. In such dual-policy agents, a model-free
policy and a distilled policy are used for model-free actions and planned
actions, respectively. Our results on a ecologically relevant, parametric
environment indicate that distilled policy network for self-model stabilizes
training, has faster inference than using model-free policy, promotes better
exploration, and could learn a comprehensive understanding of its own
behaviors, at the cost of distilling a new network apart from the model-free
policy.
