  By combining voice and touch interactions, multimodal interfaces can surpass
the efficiency of either modality alone. Traditional multimodal frameworks
require laborious developer work to support rich multimodal commands where the
user's multimodal command involves possibly exponential combinations of
actions/function invocations. This paper presents ReactGenie, a programming
framework that better separates multimodal input from the computational model
to enable developers to create efficient and capable multimodal interfaces with
ease. ReactGenie translates multimodal user commands into NLPL (Natural
Language Programming Language), a programming language we created, using a
neural semantic parser based on large-language models. The ReactGenie runtime
interprets the parsed NLPL and composes primitives in the computational model
to implement complex user commands. As a result, ReactGenie allows easy
implementation and unprecedented richness in commands for end-users of
multimodal apps. Our evaluation showed that 12 developers can learn and build a
nontrivial ReactGenie application in under 2.5 hours on average. In addition,
compared with a traditional GUI, end-users can complete tasks faster and with
less task load using ReactGenie apps.
