  The adeptness of Large Language Models (LLMs) in comprehending and following
natural language instructions is critical for their deployment in sophisticated
real-world applications. Existing evaluations mainly focus on fragmented
constraints or narrow scenarios, but they overlook the comprehensiveness and
authenticity of constraints from the user's perspective. To bridge this gap, we
propose CFBench, a large-scale Comprehensive Constraints Following Benchmark
for LLMs, featuring 1,000 curated samples that cover more than 200 real-life
scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from
real-world instructions and constructs an innovative systematic framework for
constraint types, which includes 10 primary categories and over 25
subcategories, and ensures each constraint is seamlessly integrated within the
instructions. To make certain that the evaluation of LLM outputs aligns with
user perceptions, we propose an advanced methodology that integrates
multi-dimensional assessment criteria with requirement prioritization, covering
various perspectives of constraints, instructions, and requirement fulfillment.
Evaluating current leading LLMs on CFBench reveals substantial room for
improvement in constraints following, and we further investigate influencing
factors and enhancement strategies. The data and code are publicly available at
https://github.com/PKU-Baichuan-MLSystemLab/CFBench
