  Reflective and textureless surfaces such as windows, mirrors, and walls can
be a challenge for object and scene reconstruction. These surfaces are often
poorly reconstructed and filled with depth discontinuities and holes, making it
difficult to cohesively reconstruct scenes that contain these planar
discontinuities. We propose Echoreconstruction, an audio-visual method that
uses the reflections of sound to aid in geometry and audio reconstruction for
virtual conferencing, teleimmersion, and other AR/VR experience. The mobile
phone prototype emits pulsed audio, while recording video for RGB-based 3D
reconstruction and audio-visual classification. Reflected sound and images from
the video are input into our audio (EchoCNN-A) and audio-visual (EchoCNN-AV)
convolutional neural networks for surface and sound source detection, depth
estimation, and material classification. The inferences from these
classifications enhance scene 3D reconstructions containing open spaces and
reflective surfaces by depth filtering, inpainting, and placement of unmixed
sound sources in the scene. Our prototype, VR demo, and experimental results
from real-world and virtual scenes with challenging surfaces and sound indicate
high success rates on classification of material, depth estimation, and
closed/open surfaces, leading to considerable visual and audio improvement in
3D scenes (see Figure 1).
