  Though image-to-sequence generation models have become overwhelmingly popular
in human-computer communications, they suffer from strongly favoring safe
generic questions ("What is in this picture?"). Generating uninformative but
relevant questions is not sufficient or useful. We argue that a good question
is one that has a tightly focused purpose --- one that is aimed at expecting a
specific type of response. We build a model that maximizes mutual information
between the image, the expected answer and the generated question. To overcome
the non-differentiability of discrete natural language tokens, we introduce a
variational continuous latent space onto which the expected answers project. We
regularize this latent space with a second latent space that ensures clustering
of similar answers. Even when we don't know the expected answer, this second
latent space can generate goal-driven questions specifically aimed at
extracting objects ("what is the person throwing"), attributes, ("What kind of
shirt is the person wearing?"), color ("what color is the frisbee?"), material
("What material is the frisbee?"), etc. We quantitatively show that our model
is able to retain information about an expected answer category, resulting in
more diverse, goal-driven questions. We launch our model on a set of real world
images and extract previously unseen visual concepts.
