  This paper introduces the concept of augmented conversation, which aims to
support co-located in-person conversations via embedded speech-driven
on-the-fly referencing in augmented reality (AR). Today computing technologies
like smartphones allow quick access to a variety of references during the
conversation. However, these tools often create distractions, reducing eye
contact and forcing users to focus their attention on phone screens and
manually enter keywords to access relevant information. In contrast, AR-based
on-the-fly referencing provides relevant visual references in real-time, based
on keywords extracted automatically from the spoken conversation. By embedding
these visual references in AR around the conversation partner, augmented
conversation reduces distraction and friction, allowing users to maintain eye
contact and supporting more natural social interactions. To demonstrate this
concept, we developed \system, a Hololens-based interface that leverages
real-time speech recognition, natural language processing and gaze-based
interactions for on-the-fly embedded visual referencing. In this paper, we
explore the design space of visual referencing for conversations, and describe
our our implementation -- building on seven design guidelines identified
through a user-centered design process. An initial user study confirms that our
system decreases distraction and friction in conversations compared to
smartphone searches, while providing highly useful and relevant information.
