  In domain-specific applications, GPT-4, augmented with precise prompts or
Retrieval-Augmented Generation (RAG), shows notable potential but faces the
critical tri-lemma of performance, cost, and data privacy. High performance
requires sophisticated processing techniques, yet managing multiple agents
within a complex workflow often proves costly and challenging. To address this,
we introduce the PEER (Plan, Execute, Express, Review) multi-agent framework.
This systematizes domain-specific tasks by integrating precise question
decomposition, advanced information retrieval, comprehensive summarization, and
rigorous self-assessment. Given the concerns of cost and data privacy,
enterprises are shifting from proprietary models like GPT-4 to custom models,
striking a balance between cost, security, and performance. We developed
industrial practices leveraging online data and user feedback for efficient
model tuning. This study provides best practice guidelines for applying
multi-agent systems in domain-specific problem-solving and implementing
effective agent tuning strategies. Our empirical studies, particularly in the
financial question-answering domain, demonstrate that our approach achieves
95.0% of GPT-4's performance, while effectively managing costs and ensuring
data privacy.
