  We present a robotic system that watches a human using a Kinect v2 RGB-D
sensor, detects what he forgot to do while performing an activity, and if
necessary reminds the person using a laser pointer to point out the related
object. Our simple setup can be easily deployed on any assistive robot.
  Our approach is based on a learning algorithm trained in a purely
unsupervised setting, which does not require any human annotations. This makes
our approach scalable and applicable to variant scenarios. Our model learns the
action/object co-occurrence and action temporal relations in the activity, and
uses the learned rich relationships to infer the forgotten action and the
related object. We show that our approach not only improves the unsupervised
action segmentation and action cluster assignment performance, but also
effectively detects the forgotten actions on a challenging human activity RGB-D
video dataset. In robotic experiments, we show that our robot is able to remind
people of forgotten actions successfully.
