  To learn directed behaviors in complex environments, intelligent agents need
to optimize objective functions. Various objectives are known for designing
artificial agents, including task rewards and intrinsic motivation. However, it
is unclear how the known objectives relate to each other, which objectives
remain yet to be discovered, and which objectives better describe the behavior
of humans. We introduce the Action Perception Divergence (APD), an approach for
categorizing the space of possible objective functions for embodied agents. We
show a spectrum that reaches from narrow to general objectives. While the
narrow objectives correspond to domain-specific rewards as typical in
reinforcement learning, the general objectives maximize information with the
environment through latent variable models of input sequences. Intuitively,
these agents use perception to align their beliefs with the world and use
actions to align the world with their beliefs. They infer representations that
are informative of past inputs, explore future inputs that are informative of
their representations, and select actions or skills that maximally influence
future inputs. This explains a wide range of unsupervised objectives from a
single principle, including representation learning, information gain,
empowerment, and skill discovery. Our findings suggest leveraging powerful
world models for unsupervised exploration as a path toward highly adaptive
agents that seek out large niches in their environments, rendering task rewards
optional.
