  Knowledge tagging for questions plays a crucial role in contemporary
intelligent educational applications, including learning progress diagnosis,
practice question recommendations, and course content organization.
Traditionally, these annotations are always conducted by pedagogical experts,
as the task requires not only a strong semantic understanding of both question
stems and knowledge definitions but also deep insights into connecting
question-solving logic with corresponding knowledge concepts. With the recent
emergence of advanced text encoding algorithms, such as pre-trained language
models, many researchers have developed automatic knowledge tagging systems
based on calculating the semantic similarity between the knowledge and question
embeddings. In this paper, we explore automating the task using Large Language
Models (LLMs), in response to the inability of prior encoding-based methods to
deal with the hard cases which involve strong domain knowledge and complicated
concept definitions. By showing the strong performance of zero- and few-shot
results over math questions knowledge tagging tasks, we demonstrate LLMs' great
potential in conquering the challenges faced by prior methods. Furthermore, by
proposing a reinforcement learning-based demonstration retriever, we
successfully exploit the great potential of different-sized LLMs in achieving
better performance results while keeping the in-context demonstration usage
efficiency high.
