  We devise a multimodal conversation system for dialogue utterances composed
of text, image or both modalities. We leverage Auxiliary UnsuperviseD vIsual
and TExtual Data (AUDITED). To improve the performance of text-based task, we
utilize translations of target sentences from English to French to form the
assisted supervision. For the image-based task, we employ the DeepFashion
dataset in which we seek nearest neighbor images of positive and negative
target images of the MMD data. These nearest neighbors form the nearest
neighbor embedding providing an external context for target images. We form two
methods to create neighbor embedding vectors, namely Neighbor Embedding by Hard
Assignment (NEHA) and Neighbor Embedding by Soft Assignment (NESA) which
generate context subspaces per target image. Subsequently, these subspaces are
learnt by our pipeline as a context for the target data. We also propose a
discriminator which switches between the image- and text-based tasks. We show
improvements over baselines on the large-scale Multimodal Dialogue Dataset
(MMD) and SIMMC.
