  This paper introduces a method of structure inspection using mixed-reality
headsets to reduce the human effort in reporting accurate inspection
information such as fault locations in 3D coordinates. Prior to every
inspection, the headset needs to be localized. While external pose estimation
and fiducial marker based localization would require setup, maintenance, and
manual calibration; marker-free self-localization can be achieved using the
onboard depth sensor and camera. However, due to limited depth sensor range of
portable mixed-reality headsets like Microsoft HoloLens, localization based on
simple point cloud registration (sPCR) would require extensive mapping of the
environment. Also, localization based on camera image would face the same
issues as stereo ambiguities and hence depends on viewpoint. We thus introduce
a novel approach to Joint Point Cloud and Image-based Localization (JPIL) for
mixed-reality headsets that use visual cues and headset orientation to register
small, partially overlapped point clouds and save significant manual labor and
time in environment mapping. Our empirical results compared to sPCR show
average 10 fold reduction of required overlap surface area that could
potentially save on average 20 minutes per inspection. JPIL is not only
restricted to inspection tasks but also can be essential in enabling intuitive
human-robot interaction for spatial mapping and scene understanding in
conjunction with other agents like autonomous robotic systems that are
increasingly being deployed in outdoor environments for applications like
structural inspection.
