  Skeletal motions have been heavily replied upon for human activity
recognition (HAR). Recently, a universal vulnerability of skeleton-based HAR
has been identified across a variety of classifiers and data, calling for
mitigation. To this end, we propose the first black-box defense method for
skeleton-based HAR to our best knowledge. Our method is featured by full
Bayesian treatments of the clean data, the adversaries and the classifier,
leading to (1) a new Bayesian Energy-based formulation of robust discriminative
classifiers, (2) a new adversary sampling scheme based on natural motion
manifolds, and (3) a new post-train Bayesian strategy for black-box defense. We
name our framework Bayesian Energy-based Adversarial Training or BEAT. BEAT is
straightforward but elegant, which turns vulnerable black-box classifiers into
robust ones without sacrificing accuracy. It demonstrates surprising and
universal effectiveness across a wide range of skeletal HAR classifiers and
datasets, under various attacks. Code is available at
https://github.com/realcrane/RobustActionRecogniser.
