  Recent work shows Large Language Models (LLMs) struggle to understand natural
language constraints for various text generation tasks in zero- and few-shot
settings. While, in the code domain, there is wide usage of constraints in code
format to maintain the integrity of code written in Domain-Specific Languages
(DSLs), yet there has been no work evaluating LLMs with these constraints. We
propose two novel tasks to assess the controllability of LLMs using hard and
soft constraints represented as code across five representations. Our findings
suggest that LLMs struggle to comprehend constraints in all representations
irrespective of their portions in the pre-training data. While models are
better at comprehending constraints in JSON, YAML, and natural language
representations, they struggle with constraints represented in XML and the
resource-rich language Python.
