  Instead of making behavioral decisions directly from the exponentially
expanding joint observational-action space, subtask-based multi-agent
reinforcement learning (MARL) methods enable agents to learn how to tackle
different subtasks. Most existing subtask-based MARL methods are based on
hierarchical reinforcement learning (HRL). However, these approaches often
limit the number of subtasks, perform subtask recognition periodically, and can
only identify and execute a specific subtask within the predefined fixed time
period, which makes them inflexible and not suitable for diverse and dynamic
scenarios with constantly changing subtasks. To break through above
restrictions, a \textbf{S}liding \textbf{M}ultidimensional t\textbf{A}sk window
based m\textbf{U}ti-agent reinforcement learnin\textbf{G} framework (SMAUG) is
proposed for adaptive real-time subtask recognition. It leverages a sliding
multidimensional task window to extract essential information of subtasks from
trajectory segments concatenated based on observed and predicted trajectories
in varying lengths. An inference network is designed to iteratively predict
future trajectories with the subtask-oriented policy network. Furthermore,
intrinsic motivation rewards are defined to promote subtask exploration and
behavior diversity. SMAUG can be integrated with any Q-learning-based approach.
Experiments on StarCraft II show that SMAUG not only demonstrates performance
superiority in comparison with all baselines but also presents a more prominent
and swift rise in rewards during the initial training stage.
