  In this survey, we address the key challenges in Large Language Models (LLM)
research, focusing on the importance of interpretability. Driven by increasing
interest from AI and business sectors, we highlight the need for transparency
in LLMs. We examine the dual paths in current LLM research and eXplainable
Artificial Intelligence (XAI): enhancing performance through XAI and the
emerging focus on model interpretability. Our paper advocates for a balanced
approach that values interpretability equally with functional advancements.
Recognizing the rapid development in LLM research, our survey includes both
peer-reviewed and preprint (arXiv) papers, offering a comprehensive overview of
XAI's role in LLM research. We conclude by urging the research community to
advance both LLM and XAI fields together.
