  Multi-agent reinforcement learning (MARL) is employed to develop autonomous
agents that can learn to adopt cooperative or competitive strategies within
complex environments. However, the linear increase in the number of agents
leads to a combinatorial explosion of the action space, which may result in
algorithmic instability, difficulty in convergence, or entrapment in local
optima. While researchers have designed a variety of effective algorithms to
compress the action space, these methods also introduce new challenges, such as
the need for manually designed prior knowledge or reliance on the structure of
the problem, which diminishes the applicability of these techniques. In this
paper, we introduce Evolutionary action SPAce Reduction with Knowledge
(eSpark), an exploration function generation framework driven by large language
models (LLMs) to boost exploration and prune unnecessary actions in MARL. Using
just a basic prompt that outlines the overall task and setting, eSpark is
capable of generating exploration functions in a zero-shot manner, identifying
and pruning redundant or irrelevant state-action pairs, and then achieving
autonomous improvement from policy feedback. In reinforcement learning tasks
involving inventory management and traffic light control encompassing a total
of 15 scenarios, eSpark consistently outperforms the combined MARL algorithm in
all scenarios, achieving an average performance gain of 34.4% and 9.9% in the
two types of tasks respectively. Additionally, eSpark has proven to be capable
of managing situations with a large number of agents, securing a 29.7%
improvement in scalability challenges that featured over 500 agents. The code
can be found in https://github.com/LiuZhihao2022/eSpark.git.
