  Since OpenAI's release of ChatGPT, generative AI has received significant
attention across various domains. These AI-based chat systems have the
potential to enhance the productivity of knowledge workers in diverse tasks.
However, the use of free public services poses a risk of data leakage, as
service providers may exploit user input for additional training and
optimization without clear boundaries. Even subscription-based alternatives
sometimes lack transparency in handling user data. To address these concerns
and enable Fraunhofer staff to leverage this technology while ensuring
confidentiality, we have designed and developed a customized chat AI called
FhGenie (genie being a reference to a helpful spirit). Within few days of its
release, thousands of Fraunhofer employees started using this service. As
pioneers in implementing such a system, many other organizations have followed
suit. Our solution builds upon commercial large language models (LLMs), which
we have carefully integrated into our system to meet our specific requirements
and compliance constraints, including confidentiality and GDPR. In this paper,
we share detailed insights into the architectural considerations, design,
implementation, and subsequent updates of FhGenie. Additionally, we discuss
challenges, observations, and the core lessons learned from its productive
usage.
