  The ability to forecast human-environment collisions from egocentric
observations is vital to enable collision avoidance in applications such as VR,
AR, and wearable assistive robotics. In this work, we introduce the challenging
problem of predicting collisions in diverse environments from multi-view
egocentric videos captured from body-mounted cameras. Solving this problem
requires a generalizable perception system that can classify which human body
joints will collide and estimate a collision region heatmap to localize
collisions in the environment. To achieve this, we propose a transformer-based
model called COPILOT to perform collision prediction and localization
simultaneously, which accumulates information across multi-view inputs through
a novel 4D space-time-viewpoint attention mechanism. To train our model and
enable future research on this task, we develop a synthetic data generation
framework that produces egocentric videos of virtual humans moving and
colliding within diverse 3D environments. This framework is then used to
establish a large-scale dataset consisting of 8.6M egocentric RGBD frames.
Extensive experiments show that COPILOT generalizes to unseen synthetic as well
as real-world scenes. We further demonstrate COPILOT outputs are useful for
downstream collision avoidance through simple closed-loop control. Please visit
our project webpage at https://sites.google.com/stanford.edu/copilot.
