  Voice assistants help users make phone calls, send messages, create events,
navigate, and do a lot more. However, assistants have limited capacity to
understand their users' context. In this work, we aim to take a step in this
direction. Our work dives into a new experience for users to refer to phone
numbers, addresses, email addresses, URLs, and dates on their phone screens.
Our focus lies in reference understanding, which becomes particularly
interesting when multiple similar texts are present on screen, similar to
visual grounding. We collect a dataset and propose a lightweight
general-purpose model for this novel experience. Due to the high cost of
consuming pixels directly, our system is designed to rely on the extracted text
from the UI. Our model is modular, thus offering flexibility, improved
interpretability, and efficient runtime memory utilization.
