  The efficacy of large language models (LLMs) is heavily dependent on the
quality of the underlying data, particularly within specialized domains. A
common challenge when fine-tuning LLMs for domain-specific applications is the
potential degradation of the model's generalization capabilities. To address
these issues, we propose a two-stage approach for the construction of
production prompts designed to yield high-quality data. This method involves
the generation of a diverse array of prompts that encompass a broad spectrum of
tasks and exhibit a rich variety of expressions. Furthermore, we introduce a
cost-effective, multi-dimensional quality assessment framework to ensure the
integrity of the generated labeling data. Utilizing a dataset comprised of
service provider and customer interactions from the real estate sector, we
demonstrate a positive correlation between data quality and model performance.
Notably, our findings indicate that the domain-specific proficiency of general
LLMs can be enhanced through fine-tuning with data produced via our proposed
method, without compromising their overall generalization abilities, even when
exclusively domain-specific data is employed for fine-tuning.
