  Text generation under constraints have seen increasing interests in natural
language processing, especially with the rapidly improving capabilities of
large language models. However, existing benchmarks for constrained generation
usually focus on fixed constraint types (e.g.,generate a sentence containing
certain words) that have proved to be easy for state-of-the-art models like
GPT-4. We present COLLIE, a grammar-based framework that allows the
specification of rich, compositional constraints with diverse generation levels
(word, sentence, paragraph, passage) and modeling challenges (e.g.,language
understanding, logical reasoning, counting, semantic planning). We also develop
tools for automatic extraction of task instances given a constraint structure
and a raw text corpus. Using COLLIE, we compile the COLLIE-v1 dataset with 2080
instances comprising 13 constraint structures. We perform systematic
experiments across five state-of-the-art instruction-tuned language models and
analyze their performances to reveal shortcomings. COLLIE is designed to be
extensible and lightweight, and we hope the community finds it useful to
develop more complex constraints and evaluations in the future.
