  Agent forecasting systems have been explored to investigate agent patterns
and improve decision-making in various domains, e.g., pedestrian predictions
and marketing bidding. Badminton represents a fascinating example of a
multifaceted turn-based sport, requiring both sophisticated tactic developments
and alternate-dependent decision-making. Recent deep learning approaches for
player tactic forecasting in badminton show promising performance partially
attributed to effective reasoning about rally-player interactions. However, a
critical obstacle lies in the unclear functionality of which features are
learned for simulating players' behaviors by black-box models, where existing
explainers are not equipped with turn-based and multi-output attributions. To
bridge this gap, we propose a turn-based feature attribution approach,
ShuttleSHAP, for analyzing forecasting models in badminton based on variants of
Shapley values. ShuttleSHAP is a model-agnostic explainer that aims to quantify
contribution by not only temporal aspects but also player aspects in terms of
multifaceted cues. Incorporating the proposed analysis tool into the
state-of-the-art turn-based forecasting model on the benchmark dataset reveals
that it is, in fact, insignificant to reason about past strokes, while
conventional sequential models have greater impacts. Instead, players' styles
influence the models for the future simulation of a rally. On top of that, we
investigate and discuss the causal analysis of these findings and demonstrate
the practicability with local analysis.
