  Human activity understanding is of widespread interest in artificial
intelligence and spans diverse applications like health care and behavior
analysis. Although there have been advances in deep learning, it remains
challenging. The object recognition-like solutions usually try to map pixels to
semantics directly, but activity patterns are much different from object
patterns, thus hindering success. In this work, we propose a novel paradigm to
reformulate this task in two stages: first mapping pixels to an intermediate
space spanned by atomic activity primitives, then programming detected
primitives with interpretable logic rules to infer semantics. To afford a
representative primitive space, we build a knowledge base including 26+ M
primitive labels and logic rules from human priors or automatic discovering.
Our framework, the Human Activity Knowledge Engine (HAKE), exhibits superior
generalization ability and performance upon canonical methods on challenging
benchmarks. Code and data are available at http://hake-mvig.cn/.
