  Natural behavior consists of dynamics that are complex and unpredictable,
especially when trying to predict many steps into the future. While some
success has been found in building representations of behavior under
constrained or simplified task-based conditions, many of these models cannot be
applied to free and naturalistic settings where behavior becomes increasingly
hard to model. In this work, we develop a multi-task representation learning
model for behavior that combines two novel components: (i) An action prediction
objective that aims to predict the distribution of actions over future
timesteps, and (ii) A multi-scale architecture that builds separate latent
spaces to accommodate short- and long-term dynamics. After demonstrating the
ability of the method to build representations of both local and global
dynamics in realistic robots in varying environments and terrains, we apply our
method to the MABe 2022 Multi-agent behavior challenge, where our model ranks
1st overall and on all global tasks, and 1st or 2nd on 7 out of 9 frame-level
tasks. In all of these cases, we show that our model can build representations
that capture the many different factors that drive behavior and solve a wide
range of downstream tasks.
