  Approaches to Grounded Language Learning typically focus on a single
task-based final performance measure that may not depend on desirable
properties of the learned hidden representations, such as their ability to
predict salient attributes or to generalise to unseen situations. To remedy
this, we present GROLLA, an evaluation framework for Grounded Language Learning
with Attributes with three sub-tasks: 1) Goal-oriented evaluation; 2) Object
attribute prediction evaluation; and 3) Zero-shot evaluation. We also propose a
new dataset CompGuessWhat?! as an instance of this framework for evaluating the
quality of learned neural representations, in particular concerning attribute
grounding. To this end, we extend the original GuessWhat?! dataset by including
a semantic layer on top of the perceptual one. Specifically, we enrich the
VisualGenome scene graphs associated with the GuessWhat?! images with abstract
and situated attributes. By using diagnostic classifiers, we show that current
models learn representations that are not expressive enough to encode object
attributes (average F1 of 44.27). In addition, they do not learn strategies nor
representations that are robust enough to perform well when novel scenes or
objects are involved in gameplay (zero-shot best accuracy 50.06%).
