  Human activity recognition (HAR) with wearables is one of the serviceable
technologies in ubiquitous and mobile computing applications. The
sliding-window scheme is widely adopted while suffering from the multi-class
windows problem. As a result, there is a growing focus on joint segmentation
and recognition with deep-learning methods, aiming at simultaneously dealing
with HAR and time-series segmentation issues. However, obtaining the full
activity annotations of wearable data sequences is resource-intensive or
time-consuming, while unsupervised methods yield poor performance. To address
these challenges, we propose a novel method for joint activity segmentation and
recognition with timestamp supervision, in which only a single annotated sample
is needed in each activity segment. However, the limited information of sparse
annotations exacerbates the gap between recognition and segmentation tasks,
leading to sub-optimal model performance. Therefore, the prototypes are
estimated by class-activation maps to form a sample-to-prototype contrast
module for well-structured embeddings. Moreover, with the optimal transport
theory, our approach generates the sample-level pseudo-labels that take
advantage of unlabeled data between timestamp annotations for further
performance improvement. Comprehensive experiments on four public HAR datasets
demonstrate that our model trained with timestamp supervision is superior to
the state-of-the-art weakly-supervised methods and achieves comparable
performance to the fully-supervised approaches.
