  We introduce ThreeDWorld (TDW), a platform for interactive multi-modal
physical simulation. TDW enables simulation of high-fidelity sensory data and
physical interactions between mobile agents and objects in rich 3D
environments. Unique properties include: real-time near-photo-realistic image
rendering; a library of objects and environments, and routines for their
customization; generative procedures for efficiently building classes of new
environments; high-fidelity audio rendering; realistic physical interactions
for a variety of material types, including cloths, liquid, and deformable
objects; customizable agents that embody AI agents; and support for human
interactions with VR devices. TDW's API enables multiple agents to interact
within a simulation and returns a range of sensor and physics data representing
the state of the world. We present initial experiments enabled by TDW in
emerging research directions in computer vision, machine learning, and
cognitive science, including multi-modal physical scene understanding, physical
dynamics predictions, multi-agent interactions, models that learn like a child,
and attention studies in humans and neural networks.
