  Integrating deep learning and causal discovery has increased the
interpretability of Temporal Action Segmentation (TAS) tasks. However,
frame-level causal relationships exist many complicated noises outside the
segment-level, making it infeasible to directly express macro action semantics.
Thus, we propose Causal Abstraction Segmentation Refiner (CASR), which can
refine TAS results from various models by enhancing video causality in
marginalizing frame-level casual relationships. Specifically, we define the
equivalent frame-level casual model and segment-level causal model, so that the
causal adjacency matrix constructed from marginalized frame-level causal
relationships has the ability to represent the segmnet-level causal
relationships. CASR works out by reducing the difference in the causal
adjacency matrix between we constructed and pre-segmentation results of
backbone models. In addition, we propose a novel evaluation metric Causal Edit
Distance (CED) to evaluate the causal interpretability. Extensive experimental
results on mainstream datasets indicate that CASR significantly surpasses
existing various methods in action segmentation performance, as well as in
causal explainability and generalization.
