  Large language models (LLMs) present a valuable technology for various
applications in healthcare, but their tendency to hallucinate introduces
unacceptable uncertainty in critical decision-making situations. Human-AI
collaboration (HAIC) can mitigate this uncertainty by combining human and AI
strengths for better outcomes. This paper presents a novel guided deferral
system that provides intelligent guidance when AI defers cases to human
decision-makers. We leverage LLMs' verbalisation capabilities and internal
states to create this system, demonstrating that fine-tuning small-scale LLMs
with data from large-scale LLMs greatly enhances performance while maintaining
computational efficiency and data privacy. A pilot study showcases the
effectiveness of our proposed deferral system.
