  Large Language Models have found application in various mundane and
repetitive tasks including Human Resource (HR) support. We worked with the
domain experts of SAP SE to develop an HR support chatbot as an efficient and
effective tool for addressing employee inquiries. We inserted a
human-in-the-loop in various parts of the development cycles such as dataset
collection, prompt optimization, and evaluation of generated output. By
enhancing the LLM-driven chatbot's response quality and exploring alternative
retrieval methods, we have created an efficient, scalable, and flexible tool
for HR professionals to address employee inquiries effectively. Our experiments
and evaluation conclude that GPT-4 outperforms other models and can overcome
inconsistencies in data through internal reasoning capabilities. Additionally,
through expert analysis, we infer that reference-free evaluation metrics such
as G-Eval and Prometheus demonstrate reliability closely aligned with that of
human evaluation.
