  Despite the growing adoption of mixed reality and interactive AI agents, it
remains challenging for these systems to generate high quality 2D/3D scenes in
unseen environments. The common practice requires deploying an AI agent to
collect large amounts of data for model training for every new task. This
process is costly, or even impossible, for many domains. In this study, we
develop an infinite agent that learns to transfer knowledge memory from general
foundation models (e.g. GPT4, DALLE) to novel domains or scenarios for scene
understanding and generation in the physical or virtual world. The heart of our
approach is an emerging mechanism, dubbed Augmented Reality with Knowledge
Inference Interaction (ArK), which leverages knowledge-memory to generate
scenes in unseen physical world and virtual reality environments. The knowledge
interactive emergent ability (Figure 1) is demonstrated as the observation
learns i) micro-action of cross-modality: in multi-modality models to collect a
large amount of relevant knowledge memory data for each interaction task (e.g.,
unseen scene understanding) from the physical reality; and ii) macro-behavior
of reality-agnostic: in mix-reality environments to improve interactions that
tailor to different characterized roles, target variables, collaborative
information, and so on. We validate the effectiveness of ArK on the scene
generation and editing tasks. We show that our ArK approach, combined with
large foundation models, significantly improves the quality of generated 2D/3D
scenes, compared to baselines, demonstrating the potential benefit of
incorporating ArK in generative AI for applications such as metaverse and
gaming simulation.
