  Nowadays, large-scale foundation models are being increasingly integrated
into numerous safety-critical applications, including human-autonomy teaming
(HAT) within transportation, medical, and defence domains. Consequently, the
inherent 'black-box' nature of these sophisticated deep neural networks
heightens the significance of fostering mutual understanding and trust between
humans and autonomous systems. To tackle the transparency challenges in HAT,
this paper conducts a thoughtful study on the underexplored domain of
Explainable Interface (EI) in HAT systems from a human-centric perspective,
thereby enriching the existing body of research in Explainable Artificial
Intelligence (XAI). We explore the design, development, and evaluation of EI
within XAI-enhanced HAT systems. To do so, we first clarify the distinctions
between these concepts: EI, explanations and model explainability, aiming to
provide researchers and practitioners with a structured understanding. Second,
we contribute to a novel framework for EI, addressing the unique challenges in
HAT. Last, our summarized evaluation framework for ongoing EI offers a holistic
perspective, encompassing model performance, human-centered factors, and group
task objectives. Based on extensive surveys across XAI, HAT, psychology, and
Human-Computer Interaction (HCI), this review offers multiple novel insights
into incorporating XAI into HAT systems and outlines future directions.
