  Graph Neural Networks (GNNs) have become a prevailing tool for learning
physical dynamics. However, they still encounter several challenges: 1)
Physical laws abide by symmetry, which is a vital inductive bias accounting for
model generalization and should be incorporated into the model design. Existing
simulators either consider insufficient symmetry, or enforce excessive
equivariance in practice when symmetry is partially broken by gravity. 2)
Objects in the physical world possess diverse shapes, sizes, and properties,
which should be appropriately processed by the model. To tackle these
difficulties, we propose a novel backbone, Subequivariant Graph Neural Network,
which 1) relaxes equivariance to subequivariance by considering external fields
like gravity, where the universal approximation ability holds theoretically; 2)
introduces a new subequivariant object-aware message passing for learning
physical interactions between multiple objects of various shapes in the
particle-based representation; 3) operates in a hierarchical fashion, allowing
for modeling long-range and complex interactions. Our model achieves on average
over 3% enhancement in contact prediction accuracy across 8 scenarios on
Physion and 2X lower rollout MSE on RigidFall compared with state-of-the-art
GNN simulators, while exhibiting strong generalization and data efficiency.
