  We introduce RealitySummary, a mixed reality reading assistant that can
enhance any printed or digital document using on-demand text extraction,
summarization, and augmentation. While augmented reading tools promise to
enhance physical reading experiences with overlaid digital content, prior
systems have typically required pre-processed documents, which limits their
generalizability and real-world use cases. In this paper, we explore on-demand
document augmentation by leveraging large language models. To understand
generalizable techniques for diverse documents, we first conducted an
exploratory design study which identified five categories of document
enhancements (summarization, augmentation, navigation, comparison, and
extraction). Based on this, we developed a proof-of-concept system that can
automatically extract and summarize text using Google Cloud OCR and GPT-4, then
embed information around documents using a Microsoft Hololens 2 and Apple
Vision Pro. We demonstrate real-time examples of six specific document
augmentations: 1) summaries, 2) comparison tables, 3) timelines, 4) keyword
lists, 5) summary highlighting, and 6) information cards. Results from a
usability study (N=12) and in-the-wild study (N=11) highlight the potential
benefits of on-demand MR document enhancement and opportunities for future
research.
