  Adversarial robustness evaluates the worst-case performance scenario of a
machine learning model to ensure its safety and reliability. This study is the
first to investigate the robustness of visually grounded dialog models towards
textual attacks. These attacks represent a worst-case scenario where the input
question contains a synonym which causes the previously correct model to return
a wrong answer. Using this scenario, we first aim to understand how multimodal
input components contribute to model robustness. Our results show that models
which encode dialog history are more robust, and when launching an attack on
history, model prediction becomes more uncertain. This is in contrast to prior
work which finds that dialog history is negligible for model performance on
this task. We also evaluate how to generate adversarial test examples which
successfully fool the model but remain undetected by the user/software
designer. We find that the textual, as well as the visual context are important
to generate plausible worst-case scenarios.
