  We introduce Reagent, a technology that readily converts ordinary webpages
containing structured data into software agents with which one can interact
naturally, via a combination of speech and pointing. Previous efforts to make
webpage content manipulable by third-party software components in browsers or
desktop applications have generally relied upon specialized instrumentation
included in the webpages -- a practice that neither scales well nor applies to
pre-existing webpages. In contrast, Reagent automatically captures semantic
details and semantically-meaningful mouse events from arbitrary webpages that
contain no pre-existing special instrumentation. Reagent combines these events
with text transcriptions of user speech to derive and execute parameterized
commands representing human intent. Thus, users may request various
visualization or analytic operations to be performed on data displayed on a
page by speaking to it and/or pointing to elements within it. When unable to
infer translations between event labels and human terminology, Reagent
proactively asks users for definitions and adds them to its dictionary. We
demonstrate Reagent in the context of a collection of pre-existing webpages
that contain football team and player statistics.
