  Prior Authorization delivers safe, appropriate, and cost-effective care that
is medically justified with evidence-based guidelines. However, the process
often requires labor-intensive manual comparisons between patient medical
records and clinical guidelines, that is both repetitive and time-consuming.
Recent developments in Large Language Models (LLMs) have shown potential in
addressing complex medical NLP tasks with minimal supervision. This paper
explores the application of Multi-Agent System (MAS) that utilize specialized
LLM agents to automate Prior Authorization task by breaking them down into
simpler and manageable sub-tasks. Our study systematically investigates the
effects of various prompting strategies on these agents and benchmarks the
performance of different LLMs. We demonstrate that GPT-4 achieves an accuracy
of 86.2% in predicting checklist item-level judgments with evidence, and 95.6%
in determining overall checklist judgment. Additionally, we explore how these
agents can contribute to explainability of steps taken in the process, thereby
enhancing trust and transparency in the system.
