  Modeling object dynamics with a neural network is an important problem with
numerous applications. Most recent work has been based on graph neural
networks. However, physics happens in 3D space, where geometric information
potentially plays an important role in modeling physical phenomena. In this
work, we propose a novel U-net architecture based on continuous point
convolution which naturally embeds information from 3D coordinates and allows
for multi-scale feature representations with established downsampling and
upsampling procedures. Bottleneck layers in the downsampled point clouds lead
to better long-range interaction modeling. Besides, the flexibility of point
convolutions allows our approach to generalize to sparsely sampled points from
mesh vertices and dynamically generate features on important interaction points
on mesh faces. Experimental results demonstrate that our approach significantly
improves the state-of-the-art, especially in scenarios that require accurate
gravity or collision reasoning.
