  Abstraction is a core tenet of human cognition and communication. When
composing natural language instructions, humans naturally evoke abstraction to
convey complex procedures in an efficient and concise way. Yet, interpreting
and grounding abstraction expressed in NL has not yet been systematically
studied in NLP, with no accepted benchmarks specifically eliciting abstraction
in NL. In this work, we set the foundation for a systematic study of processing
and grounding abstraction in NLP. First, we deliver a novel abstraction
elicitation method and present Hexagons, a 2D instruction-following game. Using
Hexagons we collected over 4k naturally-occurring visually-grounded
instructions rich with diverse types of abstractions. From these data, we
derive an instruction-to-execution task and assess different types of neural
models. Our results show that contemporary models and modeling practices are
substantially inferior to human performance, and that models' performance is
inversely correlated with the level of abstraction, showing less satisfying
performance on higher levels of abstraction. These findings are consistent
across models and setups, confirming that abstraction is a challenging
phenomenon deserving further attention and study in NLP/AI research.
