  Learning behavioral patterns from observational data has been a de-facto
approach to motion forecasting. Yet, the current paradigm suffers from two
shortcomings: brittle under distribution shifts and inefficient for knowledge
transfer. In this work, we propose to address these challenges from a causal
representation perspective. We first introduce a causal formalism of motion
forecasting, which casts the problem as a dynamic process with three groups of
latent variables, namely invariant variables, style confounders, and spurious
features. We then introduce a learning framework that treats each group
separately: (i) unlike the common practice mixing datasets collected from
different locations, we exploit their subtle distinctions by means of an
invariance loss encouraging the model to suppress spurious correlations; (ii)
we devise a modular architecture that factorizes the representations of
invariant mechanisms and style confounders to approximate a sparse causal
graph; (iii) we introduce a style contrastive loss that not only enforces the
structure of style representations but also serves as a self-supervisory signal
for test-time refinement on the fly. Experiments on synthetic and real datasets
show that our proposed method improves the robustness and reusability of
learned motion representations, significantly outperforming prior
state-of-the-art motion forecasting models for out-of-distribution
generalization and low-shot transfer.
