  Explainable AI (XAI) refers to techniques that provide human-understandable
insights into the workings of AI models. Recently, the focus of XAI is being
extended towards Large Language Models (LLMs) which are often criticized for
their lack of transparency. This extension calls for a significant
transformation in XAI methodologies because of two reasons. First, many
existing XAI methods cannot be directly applied to LLMs due to their complexity
advanced capabilities. Second, as LLMs are increasingly deployed across diverse
industry applications, the role of XAI shifts from merely opening the "black
box" to actively enhancing the productivity and applicability of LLMs in
real-world settings. Meanwhile, unlike traditional machine learning models that
are passive recipients of XAI insights, the distinct abilities of LLMs can
reciprocally enhance XAI. Therefore, in this paper, we introduce Usable XAI in
the context of LLMs by analyzing (1) how XAI can benefit LLMs and AI systems,
and (2) how LLMs can contribute to the advancement of XAI. We introduce 10
strategies, introducing the key techniques for each and discussing their
associated challenges. We also provide case studies to demonstrate how to
obtain and leverage explanations. The code used in this paper can be found at:
https://github.com/JacksonWuxs/UsableXAI_LLM.
