  The widespread adoption of large language models (LLMs) has created a
pressing need for an efficient, secure and private serving infrastructure,
which allows researchers to run open source or custom fine-tuned LLMs and
ensures users that their data remains private and is not stored without their
consent. While high-performance computing (HPC) systems equipped with
state-of-the-art GPUs are well-suited for training LLMs, their batch scheduling
paradigm is not designed to support real-time serving of AI applications. Cloud
systems, on the other hand, are well suited for web services but commonly lack
access to the computational power of HPC clusters, especially expensive and
scarce high-end GPUs, which are required for optimal inference speed. We
propose an architecture with an implementation consisting of a web service that
runs on a cloud VM with secure access to a scalable backend running a multitude
of LLM models on HPC systems. By offering a web service using our HPC
infrastructure to host LLMs, we leverage the trusted environment of local
universities and research centers to offer a private and secure alternative to
commercial LLM services. Our solution natively integrates with the HPC batch
scheduler Slurm, enabling seamless deployment on HPC clusters, and is able to
run side by side with regular Slurm workloads, while utilizing gaps in the
schedule created by Slurm. In order to ensure the security of the HPC system,
we use the SSH ForceCommand directive to construct a robust circuit breaker,
which prevents successful attacks on the web-facing server from affecting the
cluster. We have successfully deployed our system as a production service, and
made the source code available at \url{https://github.com/gwdg/chat-ai}
