  Metaverse technologies demand accurate, real-time, and immersive modeling on
consumer-grade hardware for both non-human perception (e.g.,
drone/robot/autonomous car navigation) and immersive technologies like AR/VR,
requiring both structural accuracy and photorealism. However, there exists a
knowledge gap in how to apply geometric reconstruction and photorealism
modeling (novel view synthesis) in a unified framework. To address this gap and
promote the development of robust and immersive modeling and rendering with
consumer-grade devices, we propose a real-world Multi-Sensor Hybrid Room
Dataset (MuSHRoom). Our dataset presents exciting challenges and requires
state-of-the-art methods to be cost-effective, robust to noisy data and
devices, and can jointly learn 3D reconstruction and novel view synthesis
instead of treating them as separate tasks, making them ideal for real-world
applications. We benchmark several famous pipelines on our dataset for joint 3D
mesh reconstruction and novel view synthesis. Our dataset and benchmark show
great potential in promoting the improvements for fusing 3D reconstruction and
high-quality rendering in a robust and computationally efficient end-to-end
fashion. The dataset and code are available at the project website:
https://xuqianren.github.io/publications/MuSHRoom/.
