  Human perception of the world is shaped by a multitude of viewpoints and
modalities. While many existing datasets focus on scene understanding from a
certain perspective (e.g. egocentric or third-person views), our dataset offers
a panoptic perspective (i.e. multiple viewpoints with multiple data
modalities). Specifically, we encapsulate third-person panoramic and front
views, as well as egocentric monocular/binocular views with rich modalities
including video, multi-channel audio, directional binaural delay, location data
and textual scene descriptions within each scene captured, presenting
comprehensive observation of the world. Figure 1 offers a glimpse of all 28
scene categories of our 360+x dataset. To the best of our knowledge, this is
the first database that covers multiple viewpoints with multiple data
modalities to mimic how daily information is accessed in the real world.
Through our benchmark analysis, we presented 5 different scene understanding
tasks on the proposed 360+x dataset to evaluate the impact and benefit of each
data modality and perspective in panoptic scene understanding. We hope this
unique dataset could broaden the scope of comprehensive scene understanding and
encourage the community to approach these problems from more diverse
perspectives.
