  Objects rarely sit in isolation in everyday human environments. If we want
robots to operate and perform tasks in our human environments, they must
understand how the objects they manipulate will interact with structural
elements of the environment for all but the simplest of tasks. As such, we'd
like our robots to reason about how multiple objects and environmental elements
relate to one another and how those relations may change as the robot interacts
with the world. We examine the problem of predicting inter-object and
object-environment relations between previously unseen objects and novel
environments purely from partial-view point clouds. Our approach enables robots
to plan and execute sequences to complete multi-object manipulation tasks
defined from logical relations. This removes the burden of providing explicit,
continuous object states as goals to the robot. We explore several different
neural network architectures for this task. We find the best performing model
to be a novel transformer-based neural network that both predicts
object-environment relations and learns a latent-space dynamics function. We
achieve reliable sim-to-real transfer without any fine-tuning. Our experiments
show that our model understands how changes in observed environmental geometry
relate to semantic relations between objects. We show more videos on our
website: https://sites.google.com/view/erelationaldynamics.
