  Skeleton-based motion representations are robust for action localization and
understanding for their invariance to perspective, lighting, and occlusion,
compared with images. Yet, they are often ambiguous and incomplete when taken
out of context, even for human annotators. As infants discern gestures before
associating them with words, actions can be conceptualized before being
grounded with labels. Therefore, we propose the first unsupervised pre-training
framework, Boundary-Interior Decoding (BID), that partitions a skeleton-based
motion sequence into discovered semantically meaningful pre-action segments. By
fine-tuning our pre-training network with a small number of annotated data, we
show results out-performing SOTA methods by a large margin.
