  Skeletal motion plays a pivotal role in human activity recognition (HAR).
Recently, attack methods have been proposed to identify the universal
vulnerability of skeleton-based HAR(S-HAR). However, the research of
adversarial transferability on S-HAR is largely missing. More importantly,
existing attacks all struggle in transfer across unknown S-HAR models. We
observed that the key reason is that the loss landscape of the action
recognizers is rugged and sharp. Given the established correlation in prior
studies~\cite{qin2022boosting,wu2020towards} between loss landscape and
adversarial transferability, we assume and empirically validate that smoothing
the loss landscape could potentially improve adversarial transferability on
S-HAR. This is achieved by proposing a new post-train Dual Bayesian strategy,
which can effectively explore the model posterior space for a collection of
surrogates without the need for re-training. Furthermore, to craft adversarial
examples along the motion manifold, we incorporate the attack gradient with
information of the motion dynamics in a Bayesian manner. Evaluated on benchmark
datasets, e.g. HDM05 and NTU 60, the average transfer success rate can reach as
high as 35.9\% and 45.5\% respectively. In comparison, current state-of-the-art
skeletal attacks achieve only 3.6\% and 9.8\%. The high adversarial
transferability remains consistent across various surrogate, victim, and even
defense models. Through a comprehensive analysis of the results, we provide
insights on what surrogates are more likely to exhibit transferability, to shed
light on future research.
