  The diversity of video delivery pipeline poses a grand challenge to the
evaluation of adaptive bitrate (ABR) streaming algorithms and objective
quality-of-experience (QoE) models. Here we introduce so-far the largest
subject-rated database of its kind, namely WaterlooSQoE-IV, consisting of 1350
adaptive streaming videos created from diverse source contents, video encoders,
network traces, ABR algorithms, and viewing devices. We collect human opinions
for each video with a series of carefully designed subjective experiments.
Subsequent data analysis and testing/comparison of ABR algorithms and QoE
models using the database lead to a series of novel observations and
interesting findings, in terms of the effectiveness of subjective experiment
methodologies, the interactions between user experience and source content,
viewing device and encoder type, the heterogeneities in the bias and preference
of user experiences, the behaviors of ABR algorithms, and the performance of
objective QoE models. Most importantly, our results suggest that a better
objective QoE model, or a better understanding of human perceptual experience
and behaviour, is the most dominating factor in improving the performance of
ABR algorithms, as opposed to advanced optimization frameworks, machine
learning strategies or bandwidth predictors, where a majority of ABR research
has been focused on in the past decade. On the other hand, our performance
evaluation of 11 QoE models shows only a moderate correlation between
state-of-the-art QoE models and subjective ratings, implying rooms for
improvement in both QoE modeling and ABR algorithms. The database is made
publicly available at: \url{https://ece.uwaterloo.ca/~zduanmu/waterloosqoe4/}.
