  Online Large Language Model (LLM) services such as ChatGPT and Claude 3 have
transformed business operations and academic research by effortlessly enabling
new opportunities. However, due to data-sharing restrictions, sectors such as
healthcare and finance prefer to deploy local LLM applications using costly
hardware resources. This scenario requires a balance between the effectiveness
advantages of LLMs and significant financial burdens. Additionally, the rapid
evolution of models increases the frequency and redundancy of benchmarking
efforts. Existing benchmarking toolkits, which typically focus on
effectiveness, often overlook economic considerations, making their findings
less applicable to practical scenarios. To address these challenges, we
introduce CEBench, an open-source toolkit specifically designed for
multi-objective benchmarking that focuses on the critical trade-offs between
expenditure and effectiveness required for LLM deployments. CEBench allows for
easy modifications through configuration files, enabling stakeholders to
effectively assess and optimize these trade-offs. This strategic capability
supports crucial decision-making processes aimed at maximizing effectiveness
while minimizing cost impacts. By streamlining the evaluation process and
emphasizing cost-effectiveness, CEBench seeks to facilitate the development of
economically viable AI solutions across various industries and research fields.
The code and demonstration are available in
\url{https://github.com/amademicnoboday12/CEBench}.
