  Physical environment understanding is vital in delivering immersive and
interactive mobile augmented reality (AR) user experiences. Recently, we have
witnessed a transition in the design of environment understanding systems, from
visual data focused to centering on the concept of spatial context, including
user, device, and environment information. Even though spatial context can
benefit the environment understanding tasks, e.g., we demonstrate in a case
study that lighting estimation performance can be improved by as much as 59%,
not all environment understanding systems support spatial context. Furthermore,
even for the environment understanding systems that support spatial context,
not all useful spatial context has been leveraged; and the design and
implementation can differ vastly. In this paper, we advocate for the design of
a spatial context-aware and shared environment understanding system that can
effectively support multiple tasks simultaneously. Fundamentally, there are
many practical challenges in designing such a unified environment understanding
system. We discuss the challenges in the context of three open questions, (i)
how to fully leverage user mobility, (ii) how to design a unified data model,
and (iii) finally how to build a shared system for multiple tasks.
