  Determining when people are struggling from video enables a finer-grained
understanding of actions and opens opportunities for building intelligent
support visual interfaces. In this paper, we present a new dataset with three
assembly activities and corresponding performance baselines for the
determination of struggle from video. Three real-world problem-solving
activities including assembling plumbing pipes (Pipes-Struggle), pitching
camping tents (Tent-Struggle) and solving the Tower of Hanoi puzzle
(Tower-Struggle) are introduced. Video segments were scored w.r.t. the level of
struggle as perceived by annotators using a forced choice 4-point scale. Each
video segment was annotated by a single expert annotator in addition to
crowd-sourced annotations. The dataset is the first struggle annotation dataset
and contains 5.1 hours of video and 725,100 frames from 73 participants in
total. We evaluate three decision-making tasks: struggle classification,
struggle level regression, and struggle label distribution learning. We provide
baseline results for each of the tasks utilising several mainstream deep neural
networks, along with an ablation study and visualisation of results. Our work
is motivated toward assistive systems that analyze struggle, support users
during manual activities and encourage learning, as well as other video
understanding competencies.
