  Current benchmarks for optical flow algorithms evaluate the estimation
quality by comparing their predicted flow field with the ground truth, and
additionally may compare interpolated frames, based on these predictions, with
the correct frames from the actual image sequences. For the latter comparisons,
objective measures such as mean square errors are applied. However, for
applications like image interpolation, the expected user's quality of
experience cannot be fully deduced from such simple quality measures.
Therefore, we conducted a subjective quality assessment study by crowdsourcing
for the interpolated images provided in one of the optical flow benchmarks, the
Middlebury benchmark. We used paired comparisons with forced choice and
reconstructed absolute quality scale values according to Thurstone's model
using the classical least squares method. The results give rise to a re-ranking
of 141 participating algorithms w.r.t. visual quality of interpolated frames
mostly based on optical flow estimation. Our re-ranking result shows the
necessity of visual quality assessment as another evaluation metric for optical
flow and frame interpolation benchmarks.
