  Our goal is to explore how the abilities brought in by a dialogue manager can
be included in end-to-end visually grounded conversational agents. We make
initial steps towards this general goal by augmenting a task-oriented visual
dialogue model with a decision-making component that decides whether to ask a
follow-up question to identify a target referent in an image, or to stop the
conversation to make a guess. Our analyses show that adding a decision making
component produces dialogues that are less repetitive and that include fewer
unnecessary questions, thus potentially leading to more efficient and less
unnatural interactions.
