  The assessment of advanced generative large language models (LLMs) poses a
significant challenge, given their heightened complexity in recent
developments. Furthermore, evaluating the performance of LLM-based applications
in various industries, as indicated by Key Performance Indicators (KPIs), is a
complex undertaking. This task necessitates a profound understanding of
industry use cases and the anticipated system behavior. Within the context of
the automotive industry, existing evaluation metrics prove inadequate for
assessing in-car conversational question answering (ConvQA) systems. The unique
demands of these systems, where answers may relate to driver or car safety and
are confined within the car domain, highlight the limitations of current
metrics. To address these challenges, this paper introduces a set of KPIs
tailored for evaluating the performance of in-car ConvQA systems, along with
datasets specifically designed for these KPIs. A preliminary and comprehensive
empirical evaluation substantiates the efficacy of our proposed approach.
Furthermore, we investigate the impact of employing varied personas in prompts
and found that it enhances the model's capacity to simulate diverse viewpoints
in assessments, mirroring how individuals with different backgrounds perceive a
topic.
