  An image related question defines a specific visual task that is required in
order to produce an appropriate answer. The answer may depend on a minor detail
in the image and require complex reasoning and use of prior knowledge. When
humans perform this task, they are able to do it in a flexible and robust
manner, integrating modularly any novel visual capability with diverse options
for various elaborations of the task. In contrast, current approaches to solve
this problem by a machine are based on casting the problem as an end-to-end
learning problem, which lacks such abilities.
  We present a different approach, inspired by the aforementioned human
capabilities. The approach is based on the compositional structure of the
question. The underlying idea is that a question has an abstract representation
based on its structure, which is compositional in nature. The question can
consequently be answered by a composition of procedures corresponding to its
substructures. The basic elements of the representation are logical patterns,
which are put together to represent the question. These patterns include a
parametric representation for object classes, properties and relations. Each
basic pattern is mapped into a basic procedure that includes meaningful visual
tasks, and the patterns are composed to produce the overall answering
procedure.
  The UnCoRd (Understand Compose and Respond) system, based on this approach,
integrates existing detection and classification schemes for a set of object
classes, properties and relations. These schemes are incorporated in a modular
manner, providing elaborated answers and corrections for negative answers. In
addition, an external knowledge base is queried for required common-knowledge.
We performed a qualitative analysis of the system, which demonstrates its
representation capabilities and provide suggestions for future developments.
