  We propose a novel conditional GAN (cGAN) model for continuous fine-grained
human action segmentation, that utilises multi-modal data and learned scene
context information. The proposed approach utilises two GANs: termed Action GAN
and Auxiliary GAN, where the Action GAN is trained to operate over the current
RGB frame while the Auxiliary GAN utilises supplementary information such as
depth or optical flow. The goal of both GANs is to generate similar `action
codes', a vector representation of the current action. To facilitate this
process a context extractor that incorporates data and recent outputs from both
modes is used to extract context information to aid recognition. The result is
a recurrent GAN architecture which learns a task specific loss function from
multiple feature modalities. Extensive evaluations on variants of the proposed
model to show the importance of utilising different information streams such as
context and auxiliary information in the proposed network; and show that our
model is capable of outperforming state-of-the-art methods for three widely
used datasets: 50 Salads, MERL Shopping and Georgia Tech Egocentric Activities,
comprising both static and dynamic camera settings.
