  Automated vision-based score estimation models can be used as an alternate
opinion to avoid judgment bias. In the past works the score estimation models
were learned by regressing the video representations to the ground truth score
provided by the judges. However such regression-based solutions lack
interpretability in terms of giving reasons for the awarded score. One solution
to make the scores more explicable is to compare the given action video with a
reference video. This would capture the temporal variations w.r.t. the
reference video and map those variations to the final score. In this work, we
propose a new action scoring system as a two-phase system: (1) A Deep Metric
Learning Module that learns similarity between any two action videos based on
their ground truth scores given by the judges; (2) A Score Estimation Module
that uses the first module to find the resemblance of a video to a reference
video in order to give the assessment score. The proposed scoring model has
been tested for Olympics Diving and Gymnastic vaults and the model outperforms
the existing state-of-the-art scoring models.
