  Assessing action quality is challenging due to the subtle differences between
videos and large variations in scores. Most existing approaches tackle this
problem by regressing a quality score from a single video, suffering a lot from
the large inter-video score variations. In this paper, we show that the
relations among videos can provide important clues for more accurate action
quality assessment during both training and inference. Specifically, we
reformulate the problem of action quality assessment as regressing the relative
scores with reference to another video that has shared attributes (e.g.,
category and difficulty), instead of learning unreferenced scores. Following
this formulation, we propose a new Contrastive Regression (CoRe) framework to
learn the relative scores by pair-wise comparison, which highlights the
differences between videos and guides the models to learn the key hints for
assessment. In order to further exploit the relative information between two
videos, we devise a group-aware regression tree to convert the conventional
score regression into two easier sub-problems: coarse-to-fine classification
and regression in small intervals. To demonstrate the effectiveness of CoRe, we
conduct extensive experiments on three mainstream AQA datasets including AQA-7,
MTL-AQA and JIGSAWS. Our approach outperforms previous methods by a large
margin and establishes new state-of-the-art on all three benchmarks.
