  Most popular goal-oriented dialogue agents are capable of understanding the
conversational context. However, with the surge of virtual assistants with
screen, the next generation of agents are required to also understand screen
context in order to provide a proper interactive experience, and better
understand users' goals. In this paper, we propose a novel multimodal
conversational framework, where the dialogue agent's next action and their
arguments are derived jointly conditioned both on the conversational and the
visual context. Specifically, we propose a new model, that can reason over the
visual context within a conversation and populate API arguments with visual
entities given the user query. Our model can recognize visual features such as
color and shape as well as the metadata based features such as price or star
rating associated with a visual entity. In order to train our model, due to a
lack of suitable multimodal conversational datasets, we also propose a novel
multimodal dialog simulator to generate synthetic data and also collect
realistic user data from MTurk to improve model robustness. The proposed model
achieves a reasonable 85% model accuracy, without high inference latency. We
also demonstrate the proposed approach in a prototypical furniture shopping
experience for a multimodal virtual assistant.
