  It has been a significant challenge to portray intraclass disparity precisely
in the area of activity recognition, as it requires a robust representation of
the correlation between subject-specific variation for each activity class. In
this work, we propose a novel end-to-end knowledge directed adversarial
learning framework, which portrays the class-conditioned intraclass disparity
using two competitive encoding distributions and learns the purified latent
codes by denoising learned disparity. Furthermore, the domain knowledge is
incorporated in an unsupervised manner to guide the optimization and further
boosts the performance. The experiments on four HAR benchmark datasets
demonstrate the robustness and generalization of our proposed methods over a
set of state-of-the-art. We further prove the effectiveness of automatic domain
knowledge incorporation in performance enhancement.
