  We propose an effective framework for the temporal action segmentation task,
namely an Action Segment Refinement Framework (ASRF). Our model architecture
consists of a long-term feature extractor and two branches: the Action
Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The
long-term feature extractor provides shared features for the two branches with
a wide temporal receptive field. The ASB classifies video frames with action
classes, while the BRB regresses the action boundary probabilities. The action
boundaries predicted by the BRB refine the output from the ASB, which results
in a significant performance improvement. Our contributions are three-fold: (i)
We propose a framework for temporal action segmentation, the ASRF, which
divides temporal action segmentation into frame-wise action classification and
action boundary regression. Our framework refines frame-level hypotheses of
action classes using predicted action boundaries. (ii) We propose a loss
function for smoothing the transition of action probabilities, and analyze
combinations of various loss functions for temporal action segmentation. (iii)
Our framework outperforms state-of-the-art methods on three challenging
datasets, offering an improvement of up to 13.7% in terms of segmental edit
distance and up to 16.1% in terms of segmental F1 score. Our code will be
publicly available soon.
