  This paper presents a new approach to form-filling by reformulating the task
as multimodal natural language Question Answering (QA). The reformulation is
achieved by first translating the elements on the GUI form (text fields,
buttons, icons, etc.) to natural language questions, where these questions
capture the element's multimodal semantics. After a match is determined between
the form element (Question) and the user utterance (Answer), the form element
is filled through a pre-trained extractive QA system. By leveraging pre-trained
QA models and not requiring form-specific training, this approach to
form-filling is zero-shot. The paper also presents an approach to further
refine the form-filling by using multi-task training to incorporate a
potentially large number of successive tasks. Finally, the paper introduces a
multimodal natural language form-filling dataset Multimodal Forms (mForms), as
well as a multimodal extension of the popular ATIS dataset to support future
research and experimentation. Results show the new approach not only maintains
robust accuracy for sparse training conditions but achieves state-of-the-art F1
of 0.97 on ATIS with approximately 1/10th of the training data.
