  Activity classification has observed great success recently. The performance
on small dataset is almost saturated and people are moving towards larger
datasets. What leads to the performance gain on the model and what the model
has learnt? In this paper we propose identity preserve transform (IPT) to study
this problem. IPT manipulates the nuisance factors (background, viewpoint,
etc.) of the data while keeping those factors related to the task (human
motion) unchanged. To our surprise, we found popular models are using highly
correlated information (background, object) to achieve high classification
accuracy, rather than using the essential information (human motion). This can
explain why an activity classification model usually fails to generalize to
datasets it is not trained on. We implement IPT in two forms, i.e. image-space
transform and 3D transform, using synthetic images. The tool will be made
open-source to help study model and dataset design.
