  Specialized lexicons are collections of words with associated constraints
such as special definitions, specific roles, and intended target audiences.
These constraints are necessary for content generation and documentation tasks
(e.g., writing technical manuals or children's books), where the goal is to
reduce the ambiguity of text content and increase its overall readability for a
specific group of audience. Understanding how large language models can capture
these constraints can help researchers build better, more impactful tools for
wider use beyond the NLP community. Towards this end, we introduce SpeciaLex, a
benchmark for evaluating a language model's ability to follow specialized
lexicon-based constraints across 18 diverse subtasks with 1,285 test instances
covering core tasks of Checking, Identification, Rewriting, and Open
Generation. We present an empirical evaluation of 15 open and closed-source
LLMs and discuss insights on how factors such as model scale, openness, setup,
and recency affect performance upon evaluating with the benchmark.
