  As language and visual understanding by machines progresses rapidly, we are
observing an increasing interest in holistic architectures that tightly
interlink both modalities in a joint learning and inference process. This trend
has allowed the community to progress towards more challenging and open tasks
and refueled the hope at achieving the old AI dream of building machines that
could pass a turing test in open domains. In order to steadily make progress
towards this goal, we realize that quantifying performance becomes increasingly
difficult. Therefore we ask how we can precisely define such challenges and how
we can evaluate different algorithms on this open tasks? In this paper, we
summarize and discuss such challenges as well as try to give answers where
appropriate options are available in the literature. We exemplify some of the
solutions on a recently presented dataset of question-answering task based on
real-world indoor images that establishes a visual turing challenge. Finally,
we argue despite the success of unique ground-truth annotation, we likely have
to step away from carefully curated dataset and rather rely on 'social
consensus' as the main driving force to create suitable benchmarks. Providing
coverage in this inherently ambiguous output space is an emerging challenge
that we face in order to make quantifiable progress in this area.
