Modeling Rate and Perceptual Quality of Scalable Video %as Functions of Quantization and Frame Rate and Its Application in Scalable Video Adaptation
Yen-Fu Ou
Wenzhi Lin
Huiqi Zeng
Yao Wang\thanks{Y.-F. Ou
W. Lin
H. Zeng
and Y. Wang are with the Polytechnic Institute of New York University
Brooklyn
NY 11201 USA (e-mail: you01@students.poly.edu; mosquitolwz@gmail.com; yoyoapply@gmail.com; yao@poly.edu).
In networked video applications, the frame rate (FR) and quantization stepsize (QS) of a compressed video are often adapted in response to the changes of the available bandwidth. It is important to understand how do the variation of FR and QS and their variation pattern affect the video quality. In this paper, we investigate the impact of temporal variation of FR and QS on the perceptual video quality. Among all possible variation patterns, we focus on videos in which two FR's (or QS's) alternate over a fixed interval. We explore the human responses to such variation by conducting subjective evaluation of test videos with different variation magnitudes and frequencies. We further analyze statistical significance of the impact of variation magnitude, variation frequency, video content, and their interactions. By analyzing the subjective ratings, we propose two models for predicting the quality of video with alternating FR and QS, respectively, The proposed models have simple mathematical forms with a few content-dependent parameters. The models fit the measured data very well using parameters determined by least square fitting with the measured data. We further propose some guidelines for adaptation of FR and QS based on trends observed from subjective test results. %Such quality assessment and modeling are essential for helping video adaptation decisions when delivering video over dynamically changing wireless links. %(including the FR (or QS) variation difference, FR (or QS) variation ratio, changing intervals, and video content) %further propose several models for predicting the quality of video with alternating FR or QS accurately. These models have different tradeoffs between accuracy and complexity.
main.tex

%\documentclass[one column, draftcls]{IEEEtran}
%\documentclass[one column, draftcls]{IEEEtran}
%\documentclass[final,9pt,technote]{IEEEtran}
\documentclass[journal]{IEEEtran}
%\usepackage{spconf}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage[cmex10]{amsmath}
%\usepackage[T1]{fontenc}
\usepackage{subfigure}
\usepackage{times}
\usepackage{comment}
\usepackage{multirow}
%\usepackage{ifpdf}
\usepackage{amsfonts}
\usepackage{txfonts}
\usepackage{cite}
\usepackage{listings}
\usepackage{xcolor}
%\usepackage[section] {placeins}
%\usepackage{hyperref}
\usepackage{url}
%\makeatletter
%\def\url@leostyle{%
%  \@ifundefined{selectfont}{\def\UrlFont{\sf}}{\def\UrlFont{\small\ttfamily}}}
%\makeatother

% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

%% Now actually use the newly defined style.
%\urlstyle{leo}
%
%\lstset{keywordstyle=\color{blue!70},
%commentstyle=\color{red!50!green!50!blue!50}, frame=single,
%rulesepcolor=\color{red!20!green!20!blue!20},escapeinside=``,
%xleftmargin=2em,xrightmargin=1em, aboveskip=1em,
%backgroundcolor=\color{yellow}}
%
%
%
%
%% correct bad hyphenation here
%\hyphenation{op-tical net-works semi-conduc-tor}

\begin{document}
%
% paper title
% can use linebreaks \\ within to get better formatting as desired
\title{Perceptual Quality of Video with Periodic Frame Rate and Quantization Variation--Subjective Studies and Analytical Modeling}

%
%\title{Modeling Rate and Perceptual Quality of  Scalable Video
%as Functions of Quantization and Frame Rate and Its Application in Scalable Video Adaptation}
%

%\name{{Yen-Fu Ou,~Yuanyi Xue,~Zhan Ma,~Yao Wang}
%\thanks{This work is supported in part by the National Science Foundation under Grant No. 0430145.}
%\address{Department of Electrical and Computer Engineering\\
%        Polytechnic Institute of NYU, Brooklyn, NY 11201, U.S.A \\
%        Email: \{you01,~yxue01,~zma03\}@students.poly.edu,~yao@poly.edu}}

\author{Yen-Fu Ou,
	    Wenzhi Lin,
	    Huiqi Zeng,
	    Yao Wang\thanks{Y.-F. Ou, W. Lin, H. Zeng, and Y. Wang are with the Polytechnic Institute of New York
University, Brooklyn, NY 11201 USA (e-mail: you01@students.poly.edu;
mosquitolwz@gmail.com; yoyoapply@gmail.com; yao@poly.edu).}
}

%
\maketitle
%
%\small{

\begin{abstract}
In networked video applications, the frame rate (FR) and quantization stepsize (QS) of a compressed video are often adapted in response to the changes of the available bandwidth. It is important to understand how do the variation of FR and QS and their variation pattern affect the video quality. In this paper, we investigate the impact of temporal variation of FR and QS on the perceptual video quality. Among all possible variation patterns, we focus on videos in which two FR's (or QS's) alternate over a fixed interval. We explore the human responses to such variation by conducting subjective evaluation of test videos with different variation magnitudes and frequencies. We further analyze statistical significance of the impact of variation magnitude, variation frequency, video content, and their interactions. By analyzing the subjective ratings, we propose two models for predicting the quality of video with alternating FR and QS, respectively, The proposed models have simple mathematical forms with a few content-dependent parameters. The models fit the measured data very well using parameters determined by least square fitting with the measured data.
We further propose some guidelines for adaptation of FR and QS based on trends observed from subjective test results.
%Such quality assessment and modeling are essential for helping video adaptation decisions when delivering video over dynamically changing wireless links.
%(including the FR (or QS) variation difference, FR (or QS) variation ratio, changing intervals, and video content)
%further propose several models for predicting the  quality of video with alternating FR or QS accurately. These models have different tradeoffs between accuracy and complexity.
\end{abstract}
%
\begin{keywords}
perceptual video quality, frame rate, QS, temporal variation, quality metrics.
\end{keywords}
%

\section{Introduction}\label{sec:Intro}

%\PARstart{T}{he} development of stat-of-art mobile networking
%utilizes the multimedia system for pervasive entertainment demand on
%personal portable devices.

In wireless video streaming, due to the limited sustainable bandwidth of a receiver, a video often has to be coded (or transcoded or extracted from a scalable stream) at reduced frame rate (FR), reduced frame size (FS), and/or increased quantization stepsize (QS), so that each coded frame has adequate quality. A critical issue is how to choose the appropriate spatial, temporal and amplitude resolutions (STAR), so as to achieve the best trade-off between picture quality and motion fluidity in the delivered video (Note that amplitude resolution is inversely related to QS). Another challenging problem is that the sustainable bandwidth of a network link often fluctuates in time, calling for adaptation of STAR. One naive approach would be to find the STAR that optimizes the perceptual quality over each short time duration based on the available instantaneous bandwidth. This may however create a video with rapidly fluctuating STAR, which may be annoying to the viewer. For example, variation in frame rate can cause visually annoying jitter artifacts.  It is important to understand how does the variation of the  STAR, individually and collectively, affect the perceived quality. Such understanding would enable us to impose proper constraints on the variation of the STAR, when adapting the STAR based on the time-varying  bandwidth.

Take for example a hypothetical case where the available bandwidth alternates between $R_l$ and $R_h$, and the frame rates that can lead to the best perceived quality for constant rate video at $R_l$ and $R_h$ are $t_l$ and $t_h$, respectively. In this situation, is it better to  code the video with alternating FR's of $t_l$ and $t_h$, or would it be better to stay at $t_l$? More generally, one may want to vary not only the FR, but also the FS and QS to meet the instantaneous rate constraints.

There have been several studies regarding the influence of temporal and amplitude resolutions, individually or jointly, on the perceptual quality \cite{G1070, QualityMetricTQuantBitRate, yenfu_csvt, TemporalMobileVideoBroadcasting, PerceptualTemporalQualityMetrics, Baroncini_2006, Moorty_Mobile_VQA_2012,yenfu_icassp, yenfu_QPVar_icip}. Some of these works (e.g.~\cite{G1070, QualityMetricTQuantBitRate, yenfu_csvt}) consider the cases where the FR and QS are fixed in the entire video, whereas some  (e.g.~\cite{TemporalMobileVideoBroadcasting, PerceptualTemporalQualityMetrics}) consider the impact of FR variation, due to non-uniform and bursty packet losses. Authors in both ~\cite{TemporalMobileVideoBroadcasting, PerceptualTemporalQualityMetrics} proposed quality models based on their subjective quality assessment, however, the model in~\cite{TemporalMobileVideoBroadcasting} involves too many parameters (a total of 4 fixed constants) and the other~\cite{PerceptualTemporalQualityMetrics} predict the quality index by non-linearly combining several sub functions, which can be used to measure different temporal features of video contents. Authors in~\cite{Baroncini_2006} proposed a variable frame rate
control scheme, which adapts the frame rate under fluctuating bandwidth environment. Although this scheme is developed by observing the human preference between fixed and variable frame rate videos they did not provide any mathematical function forms regarding the quality metrics. They provided guidelines for video adaptation instead. In~\cite{Moorty_Mobile_VQA_2012}, authors reported subjective test results for test videos with periodic variations of QS, while the FR and FS are fixed. However, they only considered QS variation under a fixed, relatively slow variation frequency (change every 5 sec), and no analytical models are proposed.
%they don't explore the quality impact while varying the frame rate.

We have conducted subjective tests evaluating the impact of FR variations when QS and spatial resolution are fixed.
Among the many possible patterns of temporal variations, we consider the simple case where the FR alternates between $t_l$ and $t_h$, with each FR staying over a constant time duration Fz. See Fig.~\ref{fig:QPFRVar_test} for an illustration of how the FR and the corresponding video rate changes in time in our test sequences. We study the effect of $t_h$, $t_l$, their difference (variation magnitude) as well as the effect of Fz (inversely related to variation frequency) on the perceived quality. This study directly addresses the questions we raised for the hypothetical example given earlier. But it can also shed lights for more complicated cases where the FR may vary among more than two levels and the variation may not follow a periodic pattern. We also conducted a parallel study of videos with periodic QS variation  between $q_l$ and $q_h$, and investigated the impact of $q_l$, $q_l$, and Fz on the perceived quality. Preliminary results of these studies have been reported in~\cite{yenfu_icassp, yenfu_QPVar_icip}.

In this paper, we report our subjective test results as well as analytical quality models for videos with FR and QS variations, respectively. Section~\ref{sec:SQmeasurement} describes our subjective test configurations. Section~\ref{sec:Test_results_FRVar} presents our study for videos with periodic variation of FR, including subjective test results, and proposed model that predicts the perceived quality based on $t_l$, $t_h$ and Fz. Section~\ref{sec:Test_results_QPVar} presents the similar set of results, but for videos with QS variation. Combining the two subjective test results in aforementioned two sections, Sec.~\ref{sec:FRQPVar_cmp} compares quality of videos with FR and QS variations, respectively, under the same bit rate variation.
Section~\ref{sec:ANOVA_test} investigates the statistical significance of impact of FR/QS variation and video content on perceptual quality using the ANOVA technique. Finally Section~\ref{sec:Conclusion} summarizes our main findings and propose several guidelines for video adaptation based on our findings.

% You must have at least 2 lines in the paragraph with the drop letter
% (should never be an issue)

%\subsection{Subsection Heading Here}
%Subsection text here.
\vspace{-.in}
\section{Subjective Test Setup}\label{sec:SQmeasurement}
\subsection{Testing Material}
%Specifically, 7 video sequences, ``akiyo'', ``city'', ``crew'',
% ``football'', ``foreman'', ``ice'' and ``waterfall'', all in CIF ($352\times288$) resolution, are
%encoded into 5 temporal layers using dyadic hierarchical prediction
%structure, with frame rates 1.875, 3.75, 7.5, 15, and 30 Hz,
%respectively, and each temporal layer contains 5 CGS~\cite{SVC_FD} layers obtained
%with QP of 44, 40, 36, 32, 28.
%To derive the appropriate functional forms for $Q_q(q; t_{\max} )$
%and $Q_t(t; q)$, we conducted subjective tests to obtain mean
%opinion scores (MOS) for the same  test sequences used for deriving
%the rate model, but the subjective tests were performed only for 84
%decoded sequences,  at frame rates of 30, 15, 7.5, 3.75 Hz, and  QP
%equals to 28, 36, and 40 (corresponding to quantization stepsize
%of 16, 40, 64, respectively). The subjective test data are obtained
%in two separate sessions. The first session contains four test
%sequences (``akiyo'', ``city'', ``crew'', and ``football'') as reported
%in~\cite{yenfu_csvt}. The second session includes 5 sequences (``akiyo'', ``football'', ``foreman'',
%``ice'' and ``waterfall''), with two common sequences (``akiyo'', ``football'') as in the first data set.  We used ratings for the two common sequences in these two sessions to determine a mapping function between the scores obtained in these two data sets, following the algorithm described in~\cite{CombiningDataSets}.

Our experiment is conducted using five video source sequences,
Akiyo, Foreman, Football, Ice, Waterfall, all in CIF $(352\times 288)$
resolution and at frame rate 30 fps originally 10 seconds long, which are chosen from
JVT (Joint Video Team) test sequence pool~\cite{JVT_site}.
All these sequences are coded using JVT scalable video model
(JSVM912)~\cite{JSVM}. For each sequence, one bitstream is generated with five temporal layers, with corresponding FR of 1.875, 3.75, 7.5, 15, and 30Hz , and each
temporal layer in turn has five quality layers created with QP equal
to 28, 32, 36, 40 and 44 (with corresponding to QS = 16, 25, 40, 64, 102), respectively, using the coarse grain scalability (CGS) without QS cascading. The GOP size is set to 16 with only the first frame coded as in the I mode. Hierarchical-B structure is used to provide temporal scalability. For motion estimation, the FastSearch mode is enabled with maximum search range of 16 for full-pel search. We use SAD (Sum of Absolute Difference) as cost function for both full-pel and sub-pel. The entropy coding method is CAVLC. The other encoding configurations follow the default settings in JSVM.
% -------------------------------------------------------------------------
% The Testing points of frame rate variation
\begin{table}[t]
\centering
\caption{Testing configuration for frame rate variation}
\label{tab:testing_points_FRVar}
\begin{tabular}{c|c|c|c}
\hline
%\multicolumn{5}{|c|}{Proposed Model}\\
%\hline
%$\rm RMSE$ & 1.73\% & 3.61\% & 2.1\% & 2.7\% \\
%$\rm PC$ & 0.99 & 0.98 & 0.99 & 0.99\\
%\hline
QS & Fz&  $t_h$(Hz) & $t_l$(Hz)\\
\hline
\hline
\multirow{4}{*}{16} & \multirow{4}{*}{1/2/3 sec} & 30 & 30/15/7.5 \\
~ & ~ & 15 & 15/7.5 \\
~ & ~ & 7.5 & 7.5 \\
\hline
\end{tabular}
\end{table}
%
% -------------------------------------------------------------------------
% The Testing points of frame rate variation
%\begin{table}[htp]
%\centering
%\caption{Testing configuration for QS variation}
%\label{tab:testing_points_QPVar}
%\begin{tabular}{c|c|c|c}
%\hline
%%\multicolumn{5}{|c|}{Proposed Model}\\
%%\hline
%%$\rm RMSE$ & 1.73\% & 3.61\% & 2.1\% & 2.7\% \\
%%$\rm PC$ & 0.99 & 0.98 & 0.99 & 0.99\\
%%\hline
%FR & Fz&  ${\rm QP}_b$ & ${\rm QP}_v$\\
%\hline
%\hline
%\multirow{3}{*}{30} & \multirow{2}{*}{1/2/3 sec} & 28 & 28/32/36/40/44 \\
%~ & ~ & 36 & 32/36/40/44 \\

%\cline{2-4}
%~ & 3 sec & 44 & 32/40/44 \\
%\hline
%\end{tabular}
%\end{table}
%
\begin{table}[htp]
\centering
\caption{Testing configuration for QS variation}
\label{tab:testing_points_QPVar}
\begin{tabular}{c|c|c|c}
\hline
FR & Fz&  ${\rm QS}_b$ & ${\rm QS}_v$\\
\hline
\hline
\multirow{3}{*}{30} & \multirow{2}{*}{1/2/3 sec} & 16 & 16/25/40/64/102 \\
~ & ~ & 40 & 25/40/64/102 \\
\cline{2-4}
~ & 3 sec & 102 & 25/64/102 \\
\hline
\multicolumn{4}{p{6 cm}}{*Note that QS values of 16, 25, 40, 64, and 102, correspond to QP levels of 28, 32, 36, 40, and 44, respectively.}\\
\end{tabular}
\end{table}



%% -------------------------------------------------------------------------
%% The Testing points of frame rate variation
%\begin{table}[t]
%\centering
%\caption{Number of viewers before/after screening}
%\label{tab:viewer_number}
%\begin{tabular}{c|c|c|c|c|c}
%\hline
% & Football & Foreman &  Ice & Waterfall & Akiyo\\
%\hline
%&\multicolumn{5}{c}{Test QP Variation}\\
%\hline
%After & 20 & 22 & 22 & 20 & 21 \\
%\hline
%Before & 17  & 19 & 20  & 17  & 19 \\
%\hline
%\end{tabular}
%\end{table}
%

Two different experiments, examining quality impact of FR and QS variation, were conducted. For temporal variation, as shown in Fig.~\ref{fig:QPFRVar_test}, we generate videos in which two frame rates
switch back and forth periodically through the entire video with changing interval (Fz) of 1, 2, and 3 seconds. The QS is fixed at 16. Let $t_h$ and $t_l$ denote the higher and lower FR of the video.
Table \ref{tab:testing_points_FRVar} details all the test configurations, which leads to a total of 90 processed (encoded and decoded) video sequences (PVS).
For QS variation, we fix FR to 30Hz but allow QS to switch back and forth periodically through the entire video with Fz of 1, 2, and 3 seconds. ${\rm QS}_b$ and ${\rm QS}_v$ denote the base (beginning) QS and deviated QS and the combinations of ${\rm QS}_b$ and ${\rm QS}_v$ are summarized in Tab.~\ref{tab:testing_points_QPVar}. As a result, there are a total of 130 PVS's.

We choose to examine the variation intervals of 1s, 2s, and 3s only, because this range represents most interesting cases to be studied, at least for an initial investigation. It is worth noting that, in video streaming, through the use of buffers, video rate variation can typically be limited to be no faster than changing every second. It is our hope that this initial study can provide important insight regarding the impact of the variation frequency and frequency magnitude on the perceptual quality.

%As shown in Tab.~\ref{tab:testing_points_QPVar},
\subsection{Subjective Test Configuration}
% Reminder: the "draftcls", not "draft", class option should be used if
% it is desired that the figures are to be displayed while in draft mode.
The subjective quality assessment is carried out by using a protocol similar to
ACR (Absolute Category Rating) described in
\cite{RecP910}. Basically, each viewer is presented a series of video in a random order, and the viewer is asked to give overall rating of each video in the range of 0 to 100. Each test for one subject consists of two sessions, a training session and a test session. The training session (about 2 minutes) is used for the subject to accustom him/herself to the rating procedure and ask questions if any. The PVS's in the test session (about 12 minutes) are ordered randomly so that each subject sees the video clips in a different order. Each test session contains only a subset of PVS's with either FR or QS variations only. Most of the viewers are engineering students from Polytechnic Institute of New York University, with age 21 to 33. There are on average 22 ratings for each PVS with QS variations, and 20 ratings for each PVS with FR variations. Details regarding each experiment are can be found in~\cite{yenfu_icassp, yenfu_QPVar_icip}.

%For Fz=1 and 2, the PVS's are 8 second long, taking the first 8 second of the original 10 sec long video. For Fz=3, in order for the total durations of different QS (or FR) to be the same, we produce 12 sec. long video. All these 12 second long PVSs start from the last 6 seconds of the original 10 sec. long sequences, and then the first 6 seconds for the sake of eliminating the annoying effect of scene change while looping back from the start point of video. By doing this, the scene change occurs at the same time as the QS (or FR) switches.


% -------------------------------------------------------------------------------------
% ---------------------------- periodic FR/QS Variation -------------------------------
\begin{figure}[t]
\vspace{-.in}
\centering
  \includegraphics[scale=0.3]{figure1_mod.pdf}
\caption{The Variations of QS or FR for a video}\label{fig:QPFRVar_test}
\vspace{-.in}
\end{figure}
% -------------------------------------------------------------------------------------
%% -------------------------- FR Variation -------------------------------
%\begin{figure}[t]
%\vspace{-.in}
%\centering
%  \includegraphics[scale=0.5]{QPVar_mapping1.pdf}
%  \includegraphics[scale=0.5]{mapping_data2_and_data3.pdf}
%\caption{The mapping plot for mapping Z-scores between two tests. Horizontal axis is the Z-scores from Test 2 and vertical axis is the Z-score from Test 1. Lines are the linear mapping function. Left: QS variation, where a single mapping function is found for all source sequences, Right: FR variation, where a best mapping function is found for each source sequence.}\label{fig:mapping_function}
%\vspace{-.in}
%\end{figure}
% -------------------------------------------------------------------------------------
%\begin{figure}[t]
%\vspace{-.in}
%\centering
%  \includegraphics[scale=0.29]{TQuant_MOS_FR_QP28_CI95.pdf}
%  \includegraphics[scale=0.29]{TQuant_p_MOS_ratio_FR_QP28.pdf}
%\caption{{\small Left (a): measured MOS; Right (b): normalized MOS where points are measured data and curves are based on the model in~(\ref{eq:T_fittingModel}). The error bars are 95\% confidence interval.}}\label{fig:m_MOS_FR_sameQP}
%\vspace{-.in}
%\end{figure}

\subsection{Data Post Processing}
The raw ratings are converted to Z-scores~\cite{AM_zscore} based on the mean and standard deviation of all the scores of each viewer.
In order to remove ``noisy'' ratings or outliers, we adopted, two post screening methods in concatenation. We first perform BT.500-11 post screening method~\cite{BT500} in Z-score domain to remove all ratings by certain viewers because their ratings are outside the range of the majority of the viewers. On average, one viewer is eliminated for each PVS. We then conduct the second step to the remaining ratings in the raw score domain using a ratio/averaging method. Basically we make use of the fact that a video coded at a lower FR (or higher QS) would not have a rating higher than a video coded at a higher FR (or lower QS), if the viewer's judgement is consistent. Therefore, we remove all the ratings by a viewer for the same source video if this viewer violated this expected consistency test for more than a certain number of PVS's for this source. More details can be found in~\cite{yenfu_icassp, yenfu_QPVar_icip}. After this step, on average, 17 ratings remain for each PVS.


% ------------------------------------------------------------------------------------
% measured MOS vs. average FR for different sequence.
% -------------------------------------------------------------------------------------
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.4]{FRVar_MNQT_avgFR_akiyo.pdf}
  \includegraphics[scale=0.4]{FRVar_MNQT_avgFR_foreman.pdf}
  \includegraphics[scale=0.4]{FRVar_MNQT_avgFR_football.pdf}
  \includegraphics[scale=0.4]{FRVar_MNQT_avgFR_ice.pdf}
  \includegraphics[scale=0.4]{FRVar_MNQT_avgFR_waterfall.pdf}
 \caption{Constant FR case: Q($t,t$) vs. normalized FR ($t/t_{\max}$). Points are the measured data and curves are obtained using Eq.~(\ref{eq:FRVar_MNQT_model}) with PCC=$0.995$, RMSE=$0.013$. Parameter $\alpha_t$ for each sequence is obtained by least squares fitting.}
 \label{fig:FRVar_pMOS_NQT_avgFR}
\end{figure*}

%% ------------------------------------------------------------------------------------
%% measured MOS vs. Normalized FR for different sequence.
%% -------------------------------------------------------------------------------------
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.37]{NMOS_avgFR_Akiyo.pdf}
%  \includegraphics[scale=0.37]{NMOS_avgFR_Foreman.pdf}
%  \includegraphics[scale=0.37]{NMOS_avgFR_Football.pdf}
%  \includegraphics[scale=0.37]{NMOS_avgFR_Ice.pdf}
%  \includegraphics[scale=0.37]{NMOS_avgFR_Waterfall.pdf}
%  \includegraphics[scale=0.4]{FRVar_NMOS_raw_Legend.pdf}
% \caption{Q($t_h,t_l$) vs. average FR. Points are the measured data with different markers and colors corresponding to different $t_h$  and Fz, as indicated in the legend to the right. For example, the points connected by the red line indicated by $30_1$ indicate data points corresponding to different $t_l$, with $t_h$=30, Fz=1 sec. Curves are predicted quality by (\ref{eq:FRVar_MNQT_model}).}
% \label{fig:FRVar_NMOS_avgFR}
%\end{figure*}

% ------------------------------------------------------------------------------------
% measured MOS vs. Normalized FR for different sequence.
% -------------------------------------------------------------------------------------
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.4]{NMOS_movingAvg_avgFR_akiyo.pdf}
  \includegraphics[scale=0.4]{NMOS_movingAvg_avgFR_foreman.pdf}
  \includegraphics[scale=0.4]{NMOS_movingAvg_avgFR_football.pdf}
  \includegraphics[scale=0.4]{NMOS_movingAvg_avgFR_ice.pdf}
  \includegraphics[scale=0.4]{NMOS_movingAvg_avgFR_waterfall.pdf}
  \includegraphics[scale=0.35]{FRVar_NMOS_raw_Legend.pdf}
 \caption{Q($t_h,t_l$) vs. average FR $t_{avg}$. Points are the measured data with different markers and colors corresponding to different $t_h$  and Fz, as indicated in the legend to the right. For example, the points connected by the red line indicated by $30_1$ indicate data points corresponding to different $t_l$, with $t_h$=30, Fz=1 sec. The cyan curves are predicted quality by (\ref{eq:FRVar_MNQT_model}) for constant FR's. The navy solid and blue dotted lines are obtained from (${\rm MNQT}_c$($t_h$)+${\rm MNQT}_c$($t_l$))/2 for $t_h$=30 and 15, respectively.}
 \label{fig:FRVar_NMOS_moving_avg_avgFR}
\end{figure*}

% ------------------------------------------------------------------------------------
% measured MOS vs. FR ratio for different sequence.
% -------------------------------------------------------------------------------------
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.37]{FRVar_NMOS_ratioFR_fl_akiyo.pdf}
  \includegraphics[scale=0.37]{FRVar_NMOS_ratioFR_fl_foreman.pdf}
  \includegraphics[scale=0.37]{FRVar_NMOS_ratioFR_fl_football.pdf}
  \includegraphics[scale=0.37]{FRVar_NMOS_ratioFR_fl_ice.pdf}
  \includegraphics[scale=0.37]{FRVar_NMOS_ratioFR_fl_waterfall.pdf}
  \includegraphics[scale=0.35]{FRVar_NMOS_ratioFR_fl_Legend.pdf}
 \caption{Q($t_h,t_l$)$/$Q($t_l,t_l$) vs. FR ratio ($t_h/t_l$) when $t_l$ is fixed. Lines with different markers and colors correspond to different $t_l$ and Fz, as indicated by the legends on the right. For example, the brown solid line as indicated by $7.5_3$ corresponds to data obtained with $t_l$=7.5, $t_h$=7.5, 15, 30, and Fz=3.}
 \label{fig:FRVar_NMOS_ratioFR_fl}
\end{figure*}

% -------------------------------------------------------------------------------------------------
% -------------------------------------------------------------------------------------------------
%\subsubsection{Datasets Combining}\label{sssec_Comb}
We use MOS($t_1$, $t_2$) to represent the mean opinion score (MOS) for a PVS with two oscillating frame rates $t_1$ and $t_2$.  In most practical applications, it is the relative MOS compared with some reference maximum MOS  that is of importance. Therefore, we report and model the relative quality ratings defined by Q($t_1,t_2$) = MOS($t_1,t_2$)/MOS($t_{\max}, t_{\max}$), with $t_{\max}$=30Hz.  Similarly, MOS($q_1, q_2$) denotes the MOS for a PVS with two oscillating QS's $q_1$ and $q_2$, and Q($q_1,q_2$) = MOS($q_1,q_2$)/MOS($q_{\min},q_{\min}$) with $q_{\min}$=16. The test video and test results (in terms of MOS) are available at~\cite{QPFRVar_dataset}.

% -------------------------------------------------------------------------------------------------
% -------------------------------------------------------------------------------------------------
\section{Video Quality under Frame Rate Variation}\label{sec:Test_results_FRVar}
This section considers the impact of periodic variation of FR on the perceptual quality, while the QS is fixed (QS=16). We first present results for videos with constant FR and show that the quality v.s. FR relation can be captured accurately by a model presented in~(\ref{eq:FRVar_MNQT_model}). We then report subjective ratings for videos with periodic FR variations under different variation magnitudes and frequencies. Based on these results, we further propose a model that relate the quality with both the low and high frame rates.

% -------------------------------------------------------------------------------------
% -------------------------------------------------------------------------------------
% -------------------------- FR Variation -------------------------------
\subsection{Impact of Constant Frame Rate}\label{ssec:Imp_FR}
Figure ~\ref{fig:FRVar_pMOS_NQT_avgFR} shows Q($t_l, t_h$) v.s. $t/t_{\max}$ (here $t_{\max}$=30), of all the testing sequences with $t_l$=$t_h$=$t$. As expected, the MOS reduces as the frame rate decreases. In our prior work~\cite{yenfu_csvt} that examined the effect of FR and QS on the perceptual quality, when each is held constant over the entire test sequence, we have found that given a fixed QS, the impact of FR can be captured by the following model, known as model for normalized quality vs. temporal resolution (MNQT),
\begin{equation} \label{eq:FRVar_MNQT_model}
{\rm MNQT}_c(t) = \frac{1- e^{-\alpha_t \cdot(\frac{t}{t_{\max}})}}{1-e^{-\alpha_t}}.
\end{equation}
The model parameter $\alpha_t$ characterizes how fast the quality drops as the frame rate reduces with smaller value corresponding to faster dropping rate.
%For this reason, in this model as well as following proposed models using the same functional form, the parameter $\alpha$ varies with the actual video content. During the model regression to find the best value of $\alpha$, we choose Root Mean Square Error (RMSE) and Pearson Correlation Coefficient (PCC) to validate the goodness of prediction.
As can be seen from Fig.~\ref{fig:FRVar_pMOS_NQT_avgFR}, this model fits with the subset of measured data ($t_h=t_l$) very well with Pearson Correlation Coefficient (PCC)=$0.995$ and Root Mean Square Error (RMSE)=$0.013$. %\footnote{Note that although we found that $\beta_t$ = 1 is the best fitting with PCC=$0.995$, RMSE=$0.013$, the dataset reported here is only the subset of the entire test sequence pool}.

% ------------------------------------------------------------------------------------
% measured MOS vs. FR ratio for different sequence.
% -------------------------------------------------------------------------------------
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.4]{FRVar_NMOS_ratioFR_fh_akiyo.pdf}
  \includegraphics[scale=0.4]{FRVar_NMOS_ratioFR_fh_foreman.pdf}
  \includegraphics[scale=0.4]{FRVar_NMOS_ratioFR_fh_football.pdf}
  \includegraphics[scale=0.4]{FRVar_NMOS_ratioFR_fh_ice.pdf}
  \includegraphics[scale=0.4]{FRVar_NMOS_ratioFR_fh_waterfall.pdf}
%  \includegraphics[scale=0.4]{FRVar_NMOS_DeltaFR_Legend.pdf}
 \caption{Q($t_h,t_l$)$/$Q($t_h,t_h$) vs. FR ratio ($t_l/t_h$)  when $t_h$ is fixed. Lines with different markers and colors correspond to different $t_h$ and Fz, as indicated by the legend in Fig.~\ref{fig:FRVar_NMOS_moving_avg_avgFR}}
 \label{fig:FRVar_NMOS_ratioFR_FR_h}
\end{figure*}


%% ------------------------------------------------------------------------------------
%% Predicted  MOS vs. FR ratio for t_h base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_3a_pow_Akiyo.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_3a_pow_Foreman.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_3a_pow_Football.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_3a_pow_Ice.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_3a_pow_Waterfall.pdf}
% \caption{Predicted Q($t_h,t_l$)$/$Q($t_h,t_h$) vs. FR ratio ($t_l/t_h$) by a power law model given in Eq.~(\ref{eq:FRVar_NMOS_ratioQS_pow_model}) with PCC = $0.980$, RMSE = $0.025$. Different model parameters are used for different $t_h$ and Fz.}\label{fig:FRVar_PrdNMOS_ratioQS_6a_pow}
%\end{figure*}

%% ------------------------------------------------------------------------------------
%% Predicted  MOS vs. FR ratio for t_h base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a_pow_Akiyo.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a_pow_Foreman.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a_pow_Football.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a_pow_Ice.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a_pow_Waterfall.pdf}
% \caption{Predicted quality vs. FR ratio ($t_l/t_h$) for high FR base using Eq.~(\ref{eq:FRVar_NMOS_ratioQS_pow_model}) with PCC = $0.958$, RMSE = $0.036$. Points are measured data and curves with same marker are the same $t_h$.}\label{fig:FRVar_PrdNMOS_ratioQS_2a_pow}
%\end{figure*}



%% ------------------------------------------------------------------------------------
%% Predicted  MOS vs. FR ratio for t_h base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_6a1k_exp_Akiyo.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_6a1k_exp_Foreman.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_6a1k_exp_Football.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_6a1k_exp_Ice.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_6a1k_exp_Waterfall.pdf}
% \caption{Q($t_h,t_l$)$/$Q($t_h,t_h$) vs. FR ratio ($t_l/t_h$) by an exponential model given in Eq.~(\ref{eq:FRVar_NMOS_ratioQS_invexp_model}) with PCC = $0.991$, RMSE = $0.016$. Different model parameters are used for different $t_h$ and Fz.}\label{fig:FRVar_NMOS_ratioQS_invexp_6a1k}
%\end{figure*}

% ------------------------------------------------------------------------------------
% Predicted  MOS vs. FR ratio for t_h base for different sequence.
% -------------------------------------------------------------------------------------
%
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a1k_exp_akiyo.pdf}
  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a1k_exp_foreman.pdf}
  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a1k_exp_football.pdf}
  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a1k_exp_ice.pdf}
  \includegraphics[scale=0.4]{FRVar_PrdNMOS_fl_fh_2a1k_exp_waterfall.pdf}
%  \includegraphics[scale=0.4]{FRVar_NMOS_DeltaFR_Legend.pdf}
 \caption{Predicted Q($t_h,t_l$)$/$Q($t_h,t_h$) vs. FR ratio ($t_l/t_h$) by an exponential model given in Eq.~(\ref{eq:FRVar_NMOS_ratioQS_invexp_model}) with PCC = $0.971$, RMSE = $0.029$. The model parameter $\alpha_{tv}$ depends on $t_h$, but not Fz.}\label{fig:FRVar_NMOS_ratioQS_invexp_2a1k}
\end{figure*}
% ------------------------------------------------------------------------------------
% ------------------------------------------------------------------------------------


\subsection{Impact of Frame Rate Variation}\label{ssec:Imp_FRVar}
%The test video and subjective test results will be available at \url{http://vision.poly.edu/}.
We now consider sequences in which the frame rate alternates between $t_h$ and $t_l$. We first discuss, under the same average frame rate, $t_{avg}$ = ($t_h$+$t_l$)/2,  how does frame rate variation magnitude $\Delta t$=$t_h$-$t_l$ affects the perceived quality. Figure~\ref{fig:FRVar_NMOS_moving_avg_avgFR} shows that, when the $t_{avg}$ is the same, the MOS for a video with a constant frame rate (as indicated as cyan lines) is higher than that with frame rate variation. The degradation due to frame rate change is more severe when $\Delta t$ is higher (e.g., quality difference between (30,7.5) and constant frame rate of (30+7.5)/2=18.75 is greater than quality difference between (30,15) and constant frame rate of (30+15)/2=22.5.).  This result is as expected, as large frame rate variation induces noticeable jitter. It is interesting to note that points corresponding to $t_l$=$t_h$/2 are relatively close to the quality-frame rate curves achievable by using constant frame rates, for most of the sequences. But those with $t_l$ lower than $t_h$/2 are much below.

Figure~\ref{fig:FRVar_NMOS_moving_avg_avgFR} shows that using the quality model derived for constant FR, i.e., ${\rm MNQT}_c (t_{avg})$ with $t_{avg}$=($t_h$+$t_h$)/2,  to represent the quality of video with alternating FR's $t_h$ and $t_l$ is not very accurate. Another intuitive approach would be to model the perceived quality of a video with alternating FR between $t_h$ and $t_l$ by the average of the quality of a video with constant FR=$t_h$, predicted by ${\rm MNQT}_c(t_h)$, and that of a video with constant FR=$t_l$, predicted by ${\rm MNQT}_c(t_l)$. The results are also illustrated in Fig.~\ref{fig:FRVar_NMOS_moving_avg_avgFR}, where the navy solid lines and blue dotted lines are the predicted quality using this method when $t_h$ equals 30 and 15 Hz, respectively. The results indicate that this method is  more accurate than using ${\rm MNQT}_c(t_{avg})$, but is still not very accurate especially when $t_h$ is much larger than $t_l$.

To examine whether alternating between
$t_l$ and $t_h$ leads to better quality than staying at $t_l$, we plots Q($t_h, t_l$)/Q($t_l, t_l$) against the ratio $t_h/t_l$ in Figure~\ref{fig:FRVar_NMOS_ratioFR_fl}, where it shows that alternating between $t_l$ and $t_h$ generally leads to a quality that is better than or similar to that obtained by staying at $t_l$ (except for Foreman when $t_l$=7.5 and Fz=1, and for Ice when $t_l$=15 and Fz=2). The slope of improvement depends on Fz, $t_l$ and the source sequence.
%reduces as $t_l$ increases, and the degree of improvement is inconsistent (e.g., Football and Waterfall have higher slope in some Fz cases). When $t_h$ is more than double of $t_l$, the quality improvement is also inconsistent.
However, the quality improvement becomes saturated as $t_h/t_l>$2. Furthermore, when $t_h>$2$t_l$, the quality reduces compared to when $t_h$=2$t_l$, for some cases, possibly due to the noticeable frame rate variation.

We next look at when $t_h$ is fixed, how the quality changes with different $t_l$. Figure~\ref{fig:FRVar_NMOS_ratioFR_FR_h} shows how Q($t_h, t_l$)/Q($t_h, t_h$) decreases with the frame rate ratio $t_l/t_h$.
%It is interesting that the dropping trends under different $t_h$ are similar. This implies that the impact of frame rate variation can be well captured by the frame rate ratio, independent of $t_h$.
We can see that the dropping trend depends on $t_h$, with a higher $t_h$ leading to a slower dropping rate. The dropping rate is also influenced by Fz, although the trend is inconsistent.


%% ------------------------------------------------------------------------------------
%% Predicted  MOS vs. normalized FR for t_h base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_2a_pow_Akiyo.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_2a_pow_Foreman.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_2a_pow_Football.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_2a_pow_Ice.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_2a_pow_Waterfall.pdf}
% \caption{Predicted quality vs. normalized FR ($t/t_{\max}$) for high FR base using Eq.~(\ref{eq:FRVar_NMOS_ratioQS_pow_model}) as $\overline{Q}_v$ in (\ref{eq:FRVar_overallQ}), with PCC = $0.960$, RMSE = $0.035$. Points are measured data and curves with same marker are the same $t_h$.}\label{fig:FRVar_PrdNMOS_NFR_2a_pow}
%\end{figure*}

%% ------------------------------------------------------------------------------------
%% Predicted  MOS vs. normalized FR for t_h base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_6a1k_exp_Akiyo.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_6a1k_exp_Foreman.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_6a1k_exp_Football.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_6a1k_exp_Ice.pdf}
%  \includegraphics[scale=0.4]{FRVar_PrdNMOS_NFR_6a1k_exp_Waterfall.pdf}
% \caption{Predicted quality vs. normalized FR ($t/t_{\max}$) for high FR base using Eq.~(\ref{eq:FRVar_NMOS_ratioQS_invexp_model}) as $\overline{Q}_v$ in (\ref{eq:FRVar_overallQ}), with PCC = $0.990$, RMSE = $0.019$. Points are measured data and curves with same marker are the same $t_h$.}\label{fig:FRVar_PrdNMOS_NFR_6a1k_invexp}
%\end{figure*}

% ------------------------------------------------------------------------------------
% Predicted  MOS vs. normalized FR for t_h base for different sequence.
% -------------------------------------------------------------------------------------
%
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.43]{FRVar_PrdNMOS_NFR_2a1k_exp_akiyo.pdf}
  \includegraphics[scale=0.43]{FRVar_PrdNMOS_NFR_2a1k_exp_foreman.pdf}
  \includegraphics[scale=0.43]{FRVar_PrdNMOS_NFR_2a1k_exp_football.pdf}
  \includegraphics[scale=0.43]{FRVar_PrdNMOS_NFR_2a1k_exp_ice.pdf}
  \includegraphics[scale=0.43]{FRVar_PrdNMOS_NFR_2a1k_exp_waterfall.pdf}
 \caption{Predicted Q($t_h,t_l$) vs. $t_{avg}/t_{\max}$ using Eq.~(\ref{eq:FRVar_overallQ_invexp_invexp}) with PCC = $0.971$, RMSE = $0.029$.}\label{fig:FRVar_PrdNMOS_NFR_2a1k_invexp}
\end{figure*}


We have found that the quality drop with the frame rate ratio can be modeled well
%using a power law function, i.e., $(t_l/t_h)^{-\alpha_{tv}({\rm Fz}, t_h)}$, where $\alpha_{tv}$ is the model parameter. The fitting curve capture the measured data well with PCC = $0.980$, RMSE = $0.025$, when each model curve is used for different $t_h$ and Fz. We also plot the fitting curve as $\alpha_{tv}$ is independent of Fz with high PCC = $0.98$ in Fig.~\ref{fig:FRVar_PrdNMOS_ratioQS_2a_pow}. Besides the power law function, we proposed to model the Q($t_h, t_l$)/Q($t_h, t_h$) using the
using an inverse exponential function of the form
\begin{equation}\label{eq:FRVar_NMOS_ratioQS_invexp_model}
{\rm MNQT}_v(t_h,t_l) = \frac{1-e^{-\alpha_{tv}(t_h)\cdot(\frac{t_l}{t_h})}}{1-e^{-\alpha_{tv}(t_h)}},
\end{equation}
where the parameter $\alpha_{tv}$ depends on $t_h$. This is based on our earlier observation that the influence of Fz on the dropping rate is inconsistent, and also based on the ANOVA analysis described in Tab.~\ref{tab:FRVar_ANOVA_Fz} and Sec.~\ref{ssec:FRVar_ANOVA_test}. Fig.~\ref{fig:FRVar_NMOS_ratioQS_invexp_2a1k} presents the fitting curves using~(\ref{eq:FRVar_NMOS_ratioQS_invexp_model}).

%As can be seen from the figure that the dropping rates corresponding to different Fz do not follow a consistent trend. This suggests that the difference in the dropping rates when Fz varies may be due to viewer inconsistency and are statistically insignificant.
%Given this observation, and based on the ANOVA analysis described in Tab.~\ref{tab:FRVar_ANOVA_Fz} and Sec.~\ref{ssec:FRVar_ANOVA_test}, we choose to use the same model parameter for all different Fz's. The model curves shown in Fig.~\ref{fig:FRVar_NMOS_ratioQS_invexp_2a1k} for each source video are obtained with separate model parameter $\alpha_{tv}$ for each $t_h$, determined by least squares fitting.
%For better understanding and quantifying the performance of these two models in (\ref{eq:FRVar_NMOS_ratioQS_pow_model},\ref{eq:FRVar_NMOS_ratioQS_invexp_model}), we compare with all the models and parameter dependency in Tab.~\ref{tab:FRVar_models_performance}. The model in Eq.~(\ref{eq:FRVar_NMOS_ratioQS_invexp_model}) with $\alpha_{tv}$ depending on $t_h$ may give less model parameters without sacrificing its performance.

\subsection{Overall Quality Model}
In order to predict the overall quality of a video with frame rates oscillating between $t_h$ and $t_l$, we make use of the fact that Q($t_h$, $t_l$) can be written as
\begin{equation}\label{eq:FRVar_def_form}
Q(t_h,t_l)=Q(t_h, t_h)\frac{Q(t_h,t_l)}{Q(t_h,t_h)}.
\end{equation}

Then we can use the model in~(\ref{eq:FRVar_MNQT_model}) to predict the first term and use the model in~(\ref{eq:FRVar_NMOS_ratioQS_invexp_model}) to estimate the second term. This yields the proposed quality model for videos with FR variation (to be called ${\rm QTV}$) given below
%with the impact of both constant FR and FR variation for a known pair of $t_h$ and $t_l$, we propose to use the product of two models, ${\rm MNQT}_c(t_h)$ and ${\rm MNQT}_v(t_l, t_h)$, i.e.,
\begin{align}\label{eq:FRVar_overallQ}
{\rm QTV}(t_h,t_l) &= {\rm MNQT}_c(t_h) {\rm MNQT}_v(t_l, t_h).\\
&=\frac{1- e^{-\alpha_t \cdot(\frac{t}{t_{\max}})}}{1-e^{-\alpha_t}}\frac{1- e^{-\alpha_{tv}(t_h)\cdot(\frac{t_l}{t_h})}}{1-e^{-\alpha_{tv}(t_h)}}.\label{eq:FRVar_overallQ_invexp_invexp}
\end{align}

The first term in~(\ref{eq:FRVar_overallQ}) predicts the quality achievable with a constant high FR $t_h$, and the second term estimates the quality reduction when the FR fluctuates between $t_h$ and $t_l$.
%in~(\ref{eq:FRVar_NMOS_ratioQS_invexp_model}), i.e.,
%\begin{equation}\label{eq:FRVar_overallQ_invexp_invexp}
%{\rm QTV}_1(t_h,t_l) =\frac{1- e^{-\alpha_t \cdot(\tilde{t})^{\beta_t}}}{1-e^{-\alpha_t}}\frac{1- e^{-\alpha_{tv}(t_h)\cdot(t_r)^{\beta_{tv}}}}{1-e^{-\alpha_{tv}}},
%\end{equation} where $\alpha_{tv}$ is independent of Fz but dependent on $t_h$ and $\beta_{tv}$ is a constant value of 1. with PCC=0.978, RMSE= 0.027 for beta = 1, PCC = 0.972, 0.029
Figure~\ref{fig:FRVar_PrdNMOS_NFR_2a1k_invexp} shows the model curves fit the measured data very well.
We note that these highly accurate results are obtained by finding best fitting model parameters for each sequence. Nonetheless, the fact that all sequences can be modeled well using the same model form is very encouraging. It suggests that the proposed simple model form reveals the right trend of how does $t_l$ and $t_h$ each affects the overall quality. Additional study is needed to investigate the relation between $\alpha_{tv}$ and $t_h$ and to develop a closed-form function possibly with content-dependent parameters. For the model to be useful for predicting quality of other sequences outside our test set, one must also investigate how to predict the model parameters from the video content.

%illustrates the predicted Q($t_h,t_l$) v.s. $t_{avg}/t_{\max}$. It can be seen that the predicted curve fits the measured data very well.

%Besides studying the quality impact directly from FR variation strength of $t_h$ and $t_l$, Note that we normalize the MOS by the MOS at $t_{\max}$ instead of the quality at $t_h$ so that the quality effect is independent of $t_h$.
%Figure~\ref{fig:FRVar_PrdNMOS_tl_tmax_1a1k_invexp} illustrates the measured Q($t_h,t_l$) against $t_l/t_{\max}$.
%It is interesting to observe that the dropping trends of videos with different $t_h$ and Fz's are clustered together. This implies that the human eyes are mostly dominated by the effect of $t_l$ in a video with FR alternating between $t_l$ and $t_h$. We further find that Q($t_l,t_h$) can be approximated well by an inverted exponential function against $t_l/t_{\min}$, i.e.,
%\begin{equation}\label{eq:FRVar_tl_tmax_invexp_model}
%{\rm QTV}_2(t_l) = \frac{1- e^{-\alpha_{tv}\cdot (\frac{t_l}{t_{\max}})^{\beta_{tv}}}}{1-e^{-\alpha_{tv}}},
%\end{equation}
%where $\beta_{tv}$ is a constant value of 0.8 and $\alpha_{tv}$ is a model parameter, characterizing the dropping trend of the curve with smaller value corresponding to higher dropping trend. As shown in Fig.~\ref{fig:FRVar_PrdNMOS_tl_tmax_1a1k_invexp}, this simple model fits all the measured data quite well with PCC=$0.970$.
%Although the PCC is not as high as the model in (\ref{eq:FRVar_overallQ_invexp_invexp}), the advantage of this simple model (\ref{eq:FRVar_tl_tmax_invexp_model}) require only one parameter while it uses two models with two parameters in (\ref{eq:FRVar_overallQ_invexp_invexp}).
%For better understanding of the tradeoff between model complexity and model accuracy, Tab.~\ref{tab:FRVar_models_performance} summarizes the model accuracy of ${\rm QTV}_1$ under different number of parameters.
%We also provide the model accuracy obtained by the naive attempt to predict the quality of FR variation using ${\rm MNQT}_c$ at $t_{avg}$ or the average of ${\rm MNQT}_c$ at $t_h$ and $t_l$ in first two rows.
%It is interesting that ${\rm QTV}_2$ model using $t_l$ only  provides much higher accuracy than using $t_{avg}$, or average of Q($t_h$) and Q($t_l$). It is only slightly worse than ${\rm QTV}_1$ with three parameters.
%Although it requires only one parameter, the prediction accuracy is not quite well. The third and fourth rows are very similar regrading the accuracy so that we should choose one with fewest model parameters or a better implementation for existing applications.

%-------------------------------------------------------------
% The accuracy of different model form and model parameters
% --------------------------------------------------------------------------
%\newcommand{\newfootnotemark}[1]{\addtocounter{footnote}{#1} \footnotemark[\value{footnote}]}
%\newcommand{\newfootnotetext}[2]{\addtocounter{footnote}{#1} \footnotetext[\value{footnote}]{#2}}
%
%\begin{table}[!htp]
%\centering
%\caption{The performance of different models for quality under FR variation}\label{tab:FRVar_models_performance}
%\begin{tabular}{p{1.5cm}|c|c|c|c}
%\hline
%Function & Assumption & \# par\newfootnotemark{1}& PCC & RMSE\\
%%\hline
%\hline
%\hline
%${\rm MNQT}_c(t_{avg})$& --& 1 & 0.73 & 0.089\\ %0.704$, RMSE = $0.105$
%\hline
%$({\rm MNQT}_c(t_h)+{\rm MNQT}_c(t_l))/2$& --& 1 & 0.898 & 0.076\\
%\hline
%\multirow{4}{*}{${\rm QTV}_1(t_h,t_l)$}& $\alpha_{tv}$$\succ$$t_h$, Fz; $\beta_{tv}$$\succ$$t_h$ &9& 0.991 & 0.010\\
%&  $\alpha_{tv}$$\succ$$t_h$, Fz & 7 & 0.981 & 0.016 \\
%&  $\alpha_{tv}$$\succ$$t_h$ & 3 & 0.971 & 0.029 \\
%%\hline
%%${\rm QTV}_2(t_h,t_l)$&$\alpha_{tv}$ not $\succ$$t_h$, Fz&1& 0.970 & 0.032\\
%\hline
%\hline
%\multicolumn{5}{l}{$\succ$:Depends on}
%\end{tabular}
%\end{table}
%
%\newfootnotetext{0}{The number of parameters for ${\rm QTV}_1$ depends on the number of $t_h$ considered. Here, we only consider two $t_h$'s.}

% -------------------------------------------------------------------------------------
% -------------------------- QS Variation -------------------------------

\section{Video Quality under QS Variation}\label{sec:Test_results_QPVar}
This section presents the impact of periodic variation of QS on the perceptual quality, while the FR is fixed. We first demonstrate results for videos with constant QS and show that the quality v.s. QS relation can be approximated well by a model presented in~(\ref{eq:QPVar_MNQQ_model}). We then report subjective ratings for videos with periodic QS variations under different variation magnitudes and frequencies. Based on these results, we further propose a model that relate the quality with both the low and high QS.

\subsection{Impact of constant QS}\label{ssec:Imp_QP}
First we investigate the influence of the QS on the perceptual quality of a video when QS is constant. Figure~\ref{fig:QPVar_MNQQ} shows Q($q, q$) v.s. $q_{\min}/q$ for all the testing sequences with $q_h=q_l=q$. As expected, the quality reduces as $q$ increases (or $q_{min}/q$ decreases). In our prior work~\cite{yenfu_csvt}, we have examined the impact of QS on the video quality, when the FR and FS are fixed. There we have found that the impact of QS can be captured by the following model, known as model for normalized quality vs. quantization (MNQQ),
\begin{equation} \label{eq:QPVar_MNQQ_model}
{\rm MNQQ}_c(q) = \frac{1- e^{-\alpha_q (\frac{q_{\min}}{q})}}{1-e^{-\alpha_q}}.
\end{equation}
As can be seen from Fig.~\ref{fig:QPVar_MNQQ}, this model fits with the subset of measured data ($q_l=q_h$) very well, with PCC=$0.990$, RMSE=$0.045$.
Note that the parameter $\alpha_q$ characterizes how fast the quality drops as the QS increases, with a smaller value corresponding to faster drop.


% ------------------------------------------------------------------------------------
% measured MOS vs. Normalized QS for different sequence.
% -------------------------------------------------------------------------------------
%
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.4]{QPVar_MNQQ_invexp_Akiyo.pdf}
  \includegraphics[scale=0.4]{QPVar_MNQQ_invexp_Foreman.pdf}
  \includegraphics[scale=0.4]{QPVar_MNQQ_invexp_Football.pdf}
  \includegraphics[scale=0.4]{QPVar_MNQQ_invexp_Ice.pdf}
  \includegraphics[scale=0.4]{QPVar_MNQQ_invexp_Waterfall.pdf}
 \caption{Constant QS case:Q($q_h, q_l$) vs. $q_{\min}/q$ for videos with $q_l=q_h=q$. Points are the measured data and curves are obtained using Eq.~(\ref{eq:QPVar_MNQQ_model}) with PCC=$0.990$, RMSE=$0.045$.}
 \label{fig:QPVar_MNQQ}
\end{figure*}

% ------------------------------------------------------------------------------------
% measured MOS vs. Normalized QS for different sequence.
% -------------------------------------------------------------------------------------
%
\begin{figure*}[!htp]
\vspace{-.1in}
\centering
  \includegraphics[scale=0.37]{QPVar_NMOS_qmin_q_raw_Akiyo.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_qmin_q_raw_Foreman.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_qmin_q_raw_Football.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_qmin_q_raw_Ice.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_qmin_q_raw_Waterfall.pdf}
  \includegraphics[scale=0.3]{QPVar_qql_Legend.pdf}
 \caption{Q($q_l, q_h$) vs. $q_{\min}/q_{avg}$ when $q_l$ is fixed. Each line connects points with the same $q_l$ and Fz. For example, $16_3$ indicates $q_h$=16, Fz=3. The cyan and yellow dash curve are obtained from (${\rm MNQQ}_c$($q_l$=16)+${\rm MNQQ}_c$($q_h$))/2, and (${\rm MNQQ}_c$($q_l$=40)+${\rm MNQQ}_c$($q_h$))/2, respectively.}
 \label{fig:QPVar_MNQQ_moving_avg}
\end{figure*}
% ------------------------------------------------------------------------------------
% measured MOS vs. Ratio qstep for QP_h base for different sequence.
% -------------------------------------------------------------------------------------
%
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS_qqh_Akiyo.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS_qqh_Foreman.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS_qqh_Football.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS_qqh_Ice.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS_qqh_Waterfall.pdf}
  \includegraphics[scale=0.34]{QPVar_qqh_Legend.pdf}
 \caption{Q($q_h, q_l$)/Q($q_h, q_h$) vs. $q_l/q_h$ when $q_h$ is fixed. Each line connects points with the same $q_h$ and Fz. For example, $102_3$ indicates $q_h$=102, Fz=3.}
 \label{fig:NMOS_ql_qh_QP_h}
\end{figure*}


% ----------------------------------------------------------
\subsection{Impact of QS Variation}\label{ssec:Imp_QPVar}
We now consider video clips in which the QS alternates between $q_h$ and $q_l$. We first discuss, under the same average QS, $q_{avg}$ = ($q_h$+$q_l$)/2,  how does QS variation affects the perceived quality. Figure~\ref{fig:QPVar_MNQQ_moving_avg} shows that, when $q_{avg}$ is the same, the quality for a video with a constant QS, Q($q_{avg}, q_{avg}$), is higher than that with QS variation, Q($q_h$, $q_l$). Note that the solid grayish blue curve in Fig.~\ref{fig:QPVar_MNQQ_moving_avg} are predicted quality for a video with constant QS based on (\ref{eq:QPVar_MNQQ_model}). It shows that using the quality model derived for constant QS, i.e., ${\rm MNQQ}_c (q_{avg})$  to represent the quality of video with alternating QS's $q_h$ and $q_l$ is not very accurate. Another natural attempt would be to model the perceived quality of a video with alternating QS between $q_l$ and $q_h$ by the average of the quality of a video with constant QS=$q_l$, predicted by ${\rm MNQQ}_c(q_l)$, and that of a video with constant QS=$q_h$, predicted by ${\rm MNQQ}_c(q_h)$. The result is also included in Fig.~\ref{fig:QPVar_MNQQ_moving_avg}; the cyan solid lines and yellow dash lines indicate the predicted quality using this method when $q_l$ equals 16 and 40, respectively. It can be seen that the prediction is not accurate especially when $q_l$ is low, e.g., 16.

We next examine when $q_h$ is fixed due to the lowest available bandwidth, whether alternating between $q_l$ and $q_h$ leads to better quality than staying at $q_h$. We plot Q($q_l, q_h$)/Q($q_h, q_h$) against the ratio $q_h/q_l$ in Fig.~\ref{fig:NMOS_ql_qh_QP_h}. For $q_h$=102, alternating between $q_l$ and $q_h$, is consistently better than staying at $q_h$=102, and the degree of improvement depends on Fz and the texture details of the video (e.g., Waterfall and Foreman have higher improvement ratio, e.g., up to 2). It can be observed that shorter Fz leads to less improvement. This is as expected as shorter Fz corresponds to more rapid QS switching, which can be more annoying to the human viewers. Interestingly, the slope of improvement saturates and becomes inconsistent as $q_l$ further decreases as $q_h/q_l$ becomes higher than 2.5. When $q_h$ is already low (e.g. $q_h$=40), switching to $q_l$ provides inconsistent gain for most sequences. For Waterfall, we see a consistent quality drop when Fz=1 and 2. Our ANOVA analysis (described in Sec.~\ref{sec:ANOVA_test}) shows that the quality variation observed for both $q_l$=25 and $q_l$=16 is statistically insignificant (see entries for ${\rm P_2}$ and ${\rm P_3}$ in Tab.~\ref{tab:ANOVA_QPVar}).


%are illustrated in Fig.~\ref{fig:QPVar_NMOS_ratioQS_qmin_qavg_invexp} with PCC=$0.942$, RMSE=$0.067$. This is not an accurate way since we learn that the quality impact is quite dependent of $q_h$ and $q_l$.



%% ------------------------------------------------------------------------------------
%% measured MOS vs. Delta qstep for QP_l base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.4]{QPVar_NMOS_deltaQS_qql_Akiyo.pdf}
%  \includegraphics[scale=0.4]{QPVar_NMOS_deltaQS_qql_Foreman.pdf}
%  \includegraphics[scale=0.4]{QPVar_NMOS_deltaQS_qql_Football.pdf}
%  \includegraphics[scale=0.4]{QPVar_NMOS_deltaQS_qql_Ice.pdf}
%  \includegraphics[scale=0.4]{QPVar_NMOS_deltaQS_qql_Waterfall.pdf}
%%  \includegraphics[scale=0.4]{QPVar_NMOS_delQP_qql_Legend.pdf}
% \caption{Measured normalized MOS vs. $\Delta$QS ($q_l-q_h$)  for low QP base.}\label{fig:QPVar_NMOS_deltaq_QP_l}
%\end{figure*}

%% ------------------------------------------------------------------------------------
%% measured MOS vs. ratio qstep for QP_l base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS1_qql_Akiyo.pdf}
%  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS1_qql_Foreman.pdf}
%  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS1_qql_Football.pdf}
%  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS1_qql_Ice.pdf}
%  \includegraphics[scale=0.37]{QPVar_NMOS_ratioQS1_qql_Waterfall.pdf}
%  \includegraphics[scale=0.36]{QPVar_NMOS_delq_Legend.pdf}
% \caption{Q($q_h, q_l$)/Q($q_l, q_l$) vs. QS ratio ($q_l/q_h$) when $q_l$ is fixed. Points are measured data and urves with different markers and colors are corresponding to different $q_l$ and Fz}\label{fig:QPVar_NMOS_ql_qh_QP_l}
%\end{figure*}


%% ------------------------------------------------------------------------------------
% Predicted  MOS vs. ratio qstep for QP_l base for different sequence.
% -------------------------------------------------------------------------------------
%
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ql_qh_invexp_new_Akiyo_b1.pdf}
  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ql_qh_invexp_new_Foreman_b1.pdf}
  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ql_qh_invexp_new_Football_b1.pdf}
  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ql_qh_invexp_new_Ice_b1.pdf}
  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ql_qh_invexp_new_Waterfall_b1.pdf}
%  \includegraphics[scale=0.36]{QPVar_NMOS_delq_Legend.pdf}
 \caption{$Q(q_h,q_l)/Q(q_l, q_l)$ vs. QS ratio ($q_l/q_h$). The model curves are based on the model in (\ref{eq:QPVar_invexp}) with PCC = $0.983$, RMSE = $0.035$. $\alpha_{qv}$ depends on $t_h$ and Fz but is the same for Fz=1 and 2.}\label{fig:QPVar_NMOS_ratioQS_invexp_4par}
\end{figure*}
% using an exponential model given in~(\ref{eq:QPVar_invexp}) with PCC = $0.983$, RMSE = $0.035$. Points are measured data. The model curves are used for different $q_l$ and Fz but is the same at Fz=1,2 , $\beta_{qv}$ for $q_l=$28 and 36 is 0.92, respectively.
%% ------------------------------------------------------------------------------------
%% Predicted  MOS vs. ratio qstep for QP_l base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ratioQS_qql_invexp_1a1k_Akiyo.pdf}
%  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ratioQS_qql_invexp_1a1k_Foreman.pdf}
%  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ratioQS_qql_invexp_1a1k_Football.pdf}
%  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ratioQS_qql_invexp_1a1k_Ice.pdf}
%  \includegraphics[scale=0.4]{QPVar_PrdNMOS_ratioQS_qql_invexp_1a1k_Waterfall.pdf}
%%  \includegraphics[scale=0.36]{QPVar_NMOS_delq_Legend.pdf}
% \caption{${\rm NQQ}_v$ vs. QS ratio ($q_l/q_h$) using an exponential model given in (\ref{eq:QPVar_invexp}) with PCC = $0.982$, RMSE = $0.028$. Points are measured data. Each model curve is for same $q_l$ but all different Fz's.}\label{fig:QPVar_NMOS_ratioQS_invexp_1a1k}
%\end{figure*}

%Instead of the quality variation in terms of Q($q_l, q_h$)/Q($q_h, q_h$) , we further investigates Q($q_l, q_h$)/Q($q_l, q_l$) (to be denoted as ${\rm NQQ}_v$), which
Next, we examine how the quality reduces when the QS varies between $q_l$ and $q_h$, compared to the quality achievable with a constant QS=$q_l$. Figure~\ref{fig:QPVar_NMOS_ratioQS_invexp_4par} plots Q($q_l, q_h$)/Q($q_l, q_l$) v.s. $q_l/q_h$. Here we can see that the falling rate depends on the video content and $q_l$. The dropping rate is very similar for Fz=1 and Fz=2,  but is typically slower when Fz=3.

We have found that Q($q_l, q_h$)/Q($q_h, q_h$) can be modeled quite accurately by the inverted exponential function of the QS ratio $q_l/q_h$, i.e.,
\begin{align}\label{eq:QPVar_invexp}
{\rm MNQQ}_v(q_h,q_l) &= \frac{1- e^{-\alpha_{qv}({\rm Fz}, q_l)\cdot(\frac{q_l}{q_h})}}{1-e^{-\alpha_{qv}({\rm Fz}, q_l)}},
\end{align}
where $\alpha_{qv}$ is a model parameter reflecting the dropping rate.
%As is clear from Fig.~\ref{fig:QPVar_NMOS_ratioQS_invexp_4par}, the parameter $\alpha_{qv}$ depends on $q_l$ and video content. Besides, the dropping trends with Fz=1 and Fz=2 are very similar, but they are different from when Fz=3.
Based on the observations noted previously, and the ANOVA analysis described in Sec.~\ref{ssec:QPVar_ANOVA_test} and Tab.~\ref{tab:QPVar_ANOVA_Fz}, we choose to use the same model parameter $\alpha_{qv}$ for Fz=1 and 2, but a different one for Fz=3. Figure.~\ref{fig:QPVar_NMOS_ratioQS_invexp_4par} shows that the model predicts the measured data very well.

\subsection{Overall Video Quality}
In order to predict the overall quality of a video with alternating QS, we again recognize that Q($q_h,q_l$) can be written as
\begin{equation}\label{eq:QPVar_def_form}
Q(q_h,q_l)=Q(q_l, q_l)\frac{Q(q_h,q_l)}{Q(q_l,q_l)}
\end{equation}

Then we can use the model in~(\ref{eq:QPVar_MNQQ_model}) to predict the term and use the model in~(\ref{eq:QPVar_invexp}) to approximate the second term. This yields the proposed quality model for videos with QS variation (to be called ${\rm QQV}$) given below
\begin{align}\label{eq:QPVar_overallQ}
{\rm QQV}(q_h,q_l) &= {\rm MNQQ}_c(q_l) {\rm MNQQ}_v(q_l, q_h),\\
&=\frac{1- e^{-\alpha_q (\frac{q_{\min}}{q_l})}}{1-e^{-\alpha_q}} \frac{1- e^{-\alpha_{qv}({\rm Fz}, q_l)\cdot(\frac{q_l}{q_h})}}{1-e^{-\alpha_{qv}({\rm Fz}, q_l)}}.\label{eq:overallQ_invexp_invexp}
\end{align}
Figure~\ref{fig:QPVar_NMOS_NQS_invexp_2par} illustrates that the predicted quality fits with the measured data very well. Note that the first term in~(\ref{eq:QPVar_MNQQ_model}) is responsible for predicting the quality at a constant QS $q_l$, whereas the second term accounts for the quality degradation due to QS variation.
%Here, $\alpha_{qv}$ depends on $q_l$ and Fz's but is the same for both Fz=1,2, while $\beta_{qv}$ is a constant value of 0.91.
% ------------------------------------------------------------------------------------
% Predicted MOS vs. avg QS for different sequence. including constant QP and QP variation
% -------------------------------------------------------------------------------------
%
\begin{figure*}[ht]
\centering
  \includegraphics[scale=0.37]{QPVar_NMOS_NQS_invexp_overall_new_Akiyo.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_NQS_invexp_overall_new_Foreman.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_NQS_invexp_overall_new_Football.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_NQS_invexp_overall_new_Ice.pdf}
  \includegraphics[scale=0.37]{QPVar_NMOS_NQS_invexp_overall_new_Waterfall.pdf}
  \includegraphics[scale=0.34]{QPVar_qql_Legend.pdf}
\caption{Q($q_h, q_l$) vs. $q_{\min}/q_{avg}$. Points are measured data and curves are based on the model in~(\ref{eq:overallQ_invexp_invexp}) with PCC = $0.978$, RMSE = $0.042$.}\label{fig:QPVar_NMOS_NQS_invexp_2par}
\end{figure*}


%% ------------------------------------------------------------------------------------
%% measured MOS vs. Normalized FR for different sequence.
%% -------------------------------------------------------------------------------------
%\begin{figure*}[!ht]
%\centering
%  \includegraphics[scale=0.43]{FRVar_NMOS_NBR_Akiyo.pdf}
%  \includegraphics[scale=0.43]{FRVar_NMOS_NBR_Foreman.pdf}
%  \includegraphics[scale=0.43]{FRVar_NMOS_NBR_Football.pdf}
%  \includegraphics[scale=0.43]{FRVar_NMOS_NBR_Ice.pdf}
%  \includegraphics[scale=0.43]{FRVar_NMOS_NBR_Waterfall.pdf}
%%  \includegraphics[scale=0.5]{FRVar_NMOS_NBR_Legend.pdf}
% \caption{Q($r_{th},r_{tl}$) vs. $r_{avg}/r_{\max}$. Points are the measured data and curves with different markers and colors corresponding to different $t_h$ and Fz. The fitting curves are predicted quality obtained from (\ref{eq:MNQR_BR_model}) with PCC = $0.992$, RMSE = $0.035$.}
% \label{fig:FRVar_NMOS_NBR}
%\end{figure*}
%
%
%%% ------------------------------------------------------------------------------------
%% measured MOS vs. normalized BR for QP_l base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.43]{QPVar_NMOS_ratioBR_raw_Akiyo.pdf}
%  \includegraphics[scale=0.43]{QPVar_NMOS_ratioBR_raw_Foreman.pdf}
%  \includegraphics[scale=0.43]{QPVar_NMOS_ratioBR_raw_Football.pdf}
%  \includegraphics[scale=0.43]{QPVar_NMOS_ratioBR_raw_Ice.pdf}
%  \includegraphics[scale=0.43]{QPVar_NMOS_ratioBR_raw_Waterfall.pdf}
%%  \includegraphics[scale=0.4]{QPVar_NMOS_NBR_Legend.pdf}
% \caption{normalized MOS vs. normalized BR ($r_{avg}/r_{\max}$) for low QS base. Points are measured MOS and curves are predicted quality obtained from (\ref{eq:MNQR_BR_model}) with same marker are the same $q_l$. PCC = $0.987$, RMSE = $0.042$.}\label{fig:QPVar_NMOS_NBR}
%\end{figure*}
% ------------------------------------------------------------------------------------
% measured MOS vs. Normalized FR for different sequence.
% -------------------------------------------------------------------------------------
\begin{figure*}[!ht]
\centering
  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz3_SVC_akiyo.pdf}
  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz3_SVC_foreman.pdf}
  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz3_SVC_football.pdf}
  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz3_SVC_ice.pdf}
  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz3_SVC_waterfall.pdf}
 \caption{Q($r_{h},r_{l}$) vs. $r_{l}/r_{h}$ when videos are coded using H.264/SVC. Blue points are for videos obtained by varying FR, whereas red points for obtained by varying FR. The high rate point is obtained with QS $q_l$=$16$, $t_h$=$30$. Points are the measured data with different markers and colors corresponding to different $r_h$ at $q=16$ or $t=30$Hz.}
 \label{fig:FRQPVar_cmp_SVC}
\end{figure*}


%% -----------------------%-------------------------------------------------------------
%% measured MOS vs. normalized BR for QP_l base for different sequence.
%% -------------------------------------------------------------------------------------
%%
%\begin{figure*}[ht]
%\centering
%  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz2_SL_Akiyo.pdf}
%  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz2_SL_Foreman.pdf}
%  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz2_SL_Football.pdf}
%  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz2_SL_Ice.pdf}
%  \includegraphics[scale=0.41]{NMOS_rate_FRQPVar_cmp_Fz2_SL_Waterfall.pdf}
% \caption{Q($r_{h},r_{l}$) vs. $r_{l}/r_{h}$ when videos are coded using single layer coding. Points are the measured data with different markers and colors corresponding to different $r_h$ at $q=16$ or $t=30$Hz.}\label{fig:FRQPVar_cmp_SL}
%\end{figure*}



\section{Adaptation of FR and QS under the Same Rate Variation}\label{sec:FRQPVar_cmp}
Recall that the motivation for this study is how to adapt a video when the available bandwidth changes. A natural question to ask is, given the rate variation pattern,  which encoding parameter should we change, QS or FR? To answer this question, we find the bit rates corresponding to different QS and FR variations, and replot the quality in terms of the same bit rate variation.

%For the network video delivery, the video quality may variate due to the heavily-trafficked transmission even without considered packet loss. This is essentially caused by dynamically bandwidth exploitation by heterogeneous networked users and unstable wireless capacities. The exploration of this phenomenon will be discuss in this section base on our subjective results in this paper. With a given reduced rate, wether we should change QS or FR in order to optimally minimize the impact of quality degradation? In this section we present the quality reduction when QS increases, v.s. quality reduction when FR decreases.
%Two scenarios are investigated, one is when videos are coded using scalable extension of H.264/AVC (SVC)~\cite{H264SVC}, and the other is using single layer H.264/AVC.

Let $r_{l}$ and $r_{h}$ denote the bitrate at $t_l$ (or $q_h$) and $t_h$ (or $q_l$), respectively. We plot the measured data, defined as Q($r_{h},r_{l}$) = MOS($r_{h},r_{l}$)/MOS($r_{\max},r_{\max}$) in terms of bitrate ratio $r_{l}/r_{h}$ in Figs.~\ref{fig:FRQPVar_cmp_SVC}  , where the high rate point ($r_h$) is achieved with FR=30 and QS=16. The rates corresponding to different FR and QS combinations are the actual bit rates when the test videos are coded using the H.264/SVC encoder using the configuration described in Sec.~\ref{sec:SQmeasurement}. Because the quality ratings between different Fz's are quite similar, so we only present the cases with Fz=3 only. Figure.~\ref{fig:FRQPVar_cmp_SVC} demonstrates the quality degradation v.s. rate ratio. It is clear that for the same periodic rate variation, videos with QS variation are more preferable than FR variation.
We would like to stress that this result is for a particular  high rate point when the video is coded at a very high quality. When the video is already at a relatively low rate (coded using a lower FR or higher QS), it is possible that lowering FR may be better than increasing QS, if one has to further reduce the bit rate periodically. Unfortunately, we don't have subjective ratings for videos starting at a common encoding configuration at a low bit rate. Acquiring such data is beyond the scope of this paper.


% ---------------------------------------------------------------------
\section{Statistic Analysis}\label{sec:ANOVA_test}
The results presented in Sec.~\ref{sec:Test_results_FRVar} and~\ref{sec:Test_results_QPVar} show that FR/QS variation magnitude, variation frequency and video content all affect the perceptual quality. To evaluate whether the changes in quality ratings due to these factors and their interactions are statistically significant, we perform three-way Analysis of Variance (ANOVA)~\cite{Snedecor_Statistics}. With ANOVA, we compute the probability ($p$-value, which is derived from the cumulative distribution function of F based on the $F$-value) of the event that the difference in MOS when a particular variable is changed is due to chance. If this probability is low ($p$-value $<$ 0.05), we consider this variable as having statistical significance on MOS.

%For each considered case, we evaluate the $p$-value due to $\Delta$QS/FR, Fz (frequency), video content (i.e. across five video sources), and their interactions, respectively.
%, listed in Table~\ref{tab:ANOVA_QPVar} and~\ref{tab:ANOVA_FRVar}.
% --------------------------------------------------------------------------
% TABLE ANOVA TEST RESULT For FR Variation
% --------------------------------------------------------------------------
%\begin{table}[!htp]
%\centering
%\caption{Three-way ANOVA results for FR variation using Q($t_h$,$t_l$) data}\label{tab:ANOVA_FRVar}
%\begin{tabular}{c|c|c}
%\hline
%Variables & $F$-value & $p$-value\\
%%\hline
%\hline
%\hline
%\multicolumn{3}{c}{${\rm P_1}$ : All Data}\\
%\hline
%$\Delta$t &  26.3  & 0 \\
%Content &  83.83  & 0  \\
%Frequency &   20.62 & 0  \\
%%Content*frequency &  5.32  &  0\\
%%$\Delta$t*frequency &  33.71 &  0 \\
%%$\Delta$t*Content &  0.85 &  0.595  \\
%%$\Delta$t*Content*frequency &  0.75 &  0.798 \\
%\hline
%\multicolumn{3}{c}{${\rm P_2}$ : (30, 15) and (30, 30)}\\
%\hline
%$\Delta t$ & 1.74  & 0.187 \\
%%Frequency &  21.22 & 0 \\
%%Content &  10.58 & 0 \\
%%Content*frequency & 2.56  & 0.009  \\
%%$\Delta t$*frequency &  44.95  & 0  \\
%%$\Delta t$*Content & 1.65  & 0.15  \\
%\hline
%\multicolumn{3}{c}{${\rm P_3}$ : (15, 7.5) and (15, 15)}\\
%\hline
%$\Delta t$ & 45.02  & 0 \\
%%Frequency &  3.19 & 0.042 \\
%%Content & 11.34  & 0 \\
%%Content*frequency &  2.6  & 0.008  \\
%%$\Delta t$*frequency &  8.21  & 0  \\
%%$\Delta t$*Content & 0.52  & 0.72  \\
%\hline
%\multicolumn{3}{c}{${\rm P_4}$ : (30, 15) and (15, 15)}\\
%\hline
%$\Delta t$ & 44.52  & 0 \\
%%Frequency &  21.69 & 0 \\
%%Content &  14.24 & 0 \\
%%Content*frequency &  3.28  & 0.001  \\
%%$\Delta t$*frequency &  35.1  & 0  \\
%%$\Delta t$*Content & 0.47  & 0.75  \\
%\hline
%\multicolumn{3}{c}{${\rm P_5}$ : (15, 7.5) and (7.5, 7.5)}\\
%\hline
%$\Delta t$ & 31.36  & 0 \\
%%Frequency & 2.48  & 0.08 \\
%%Content &  7.59 & 0 \\
%%Content*frequency &  2.06  &  0.03 \\
%%$\Delta t$*frequency &  7.35  & 0.0007  \\
%%$\Delta t$*Content &  1.69 & 0.15  \\
%\hline
%% $Content \ast f_{max} \ast \Delta f$ & - & - & - \\
%\hline
%\end{tabular}
%\end{table}


% --------------------------------------------------------------------------
% --------------------------------------------------------------------------
\begin{table}[!htp]
\centering
\caption{Three-way ANOVA results for FR variation using Q($t_h$,$t_l$) data}\label{tab:ANOVA_FRVar}
\begin{tabular}{@{~}c@{ } p{1.5 cm}|c|c|c}
\hline
& Cases & Factors & $F$-value & $p$-value\\
%\hline
\hline
\hline
\multirow{3}{*}{${\rm P_1}$:} & \multirow{3}{*}{All Data}&$\Delta$t &  26.3  & 0 \\
&&Content &  83.83  & 0  \\
&&Frequency &   20.62 & 0  \\
%Content*frequency &  5.32  &  0\\
%$\Delta$t*frequency &  33.71 &  0 \\
%$\Delta$t*Content &  0.85 &  0.595  \\
%$\Delta$t*Content*frequency &  0.75 &  0.798 \\
\hline
\hline
${\rm P_2}$: & (30, 15) and (30, 30) &$\Delta t$ & 1.74  & 0.187 \\
%Frequency &  21.22 & 0 \\
%Content &  10.58 & 0 \\
%Content*frequency & 2.56  & 0.009  \\
%$\Delta t$*frequency &  44.95  & 0  \\
%$\Delta t$*Content & 1.65  & 0.15  \\
\hline
\hline
${\rm P_3}$: & (15, 7.5) and (15, 15) &$\Delta t$ & 45.02  & 0 \\
%Frequency &  3.19 & 0.042 \\
%Content & 11.34  & 0 \\
%Content*frequency &  2.6  & 0.008  \\
%$\Delta t$*frequency &  8.21  & 0  \\
%$\Delta t$*Content & 0.52  & 0.72  \\
\hline
\hline
${\rm P_4}$: & (30, 15) and (15, 15) &$\Delta t$ & 44.52  & 0 \\
%Frequency &  21.69 & 0 \\
%Content &  14.24 & 0 \\
%Content*frequency &  3.28  & 0.001  \\
%$\Delta t$*frequency &  35.1  & 0  \\
%$\Delta t$*Content & 0.47  & 0.75  \\
\hline
\hline
${\rm P_5}$: & (15, 7.5) and (7.5, 7.5) &$\Delta t$ & 31.36  & 0 \\
%Frequency & 2.48  & 0.08 \\
%Content &  7.59 & 0 \\
%Content*frequency &  2.06  &  0.03 \\
%$\Delta t$*frequency &  7.35  & 0.0007  \\
%$\Delta t$*Content &  1.69 & 0.15  \\
\hline
% $Content \ast f_{max} \ast \Delta f$ & - & - & - \\
\hline
\end{tabular}
\end{table}

% --------------------------------------------------------------------------
% TABLE ANOVA TEST RESULT
% --------------------------------------------------------------------------
\begin{table}[!htp]
\centering
\caption{The ANOVA test for the FR variation frequency using Q($t_h,t_l$)/Q($t_h,t_h$) data}\label{tab:FRVar_ANOVA_Fz}
\begin{tabular}{c|c|c|c}
\hline
Target pair & Factors & $F$-value & $p$-value\\
\hline
\hline
\multicolumn{4}{c}{all Fz's and $t_h$'s}\\
\hline
% & $\Delta t$ &  166.87 & 0  \\
%-& Fz &  0.92 & 0.40  \\
 & $\Delta t$*Fz &  0.88 & 0.54  \\
\hline
\multicolumn{4}{c}{when $t_h$ = 30Hz, $t_l$=30, 15, 7.5}\\
\hline
Fz = 1 and 2 & $\Delta t$*Fz & 1.76 & 0.19 \\
Fz = 1 and 3 & $\Delta t$*Fz & 1.44  & 0.25  \\
Fz = 2 and 3 & $\Delta t$*Fz &  0.49 &  0.61 \\
Fz =1, 2, 3 & $\Delta t$*Fz & 1.28  & 0.29  \\
\hline
\hline
\multicolumn{4}{c}{when $t_h$ = 15Hz, , $t_l$=15, 7.5}\\
\hline
Fz = 1 and 2 & $\Delta t$*Fz &  3.43 & 0.08  \\
Fz = 1 and 3 & $\Delta t$*Fz & 3.41  & 0.08  \\
Fz = 2 and 3 & $\Delta t$*Fz & 0.001  & 0.96  \\
Fz =1, 2, 3 & $\Delta t$*Fz & 2.44 & 0.10  \\
\hline
\end{tabular}
\end{table}
% -------------------------------------------------------------------------


\subsection{Analysis for Frame Rate Variation}\label{ssec:FRVar_ANOVA_test}
In Tab.~\ref{tab:ANOVA_FRVar}, results given under ${\rm P_1}$ are obtained by considering all test conditions together. We can see that FR variation magnitude (indicated by $\Delta t$=$t_h-t_l$), variation frequency (inversely related to Fz), and video content all have statistically significant impact on the subjective ratings. %Furthermore, there is significant interaction between variation magnitude and frequency, between content and variation frequency, but not between variation magnitude and content. The three way interaction among them is insignificant. This implies that viewers perceive similar quality degradation under the FR switching regardless the video contents. We note that the same phenomenon is also discovered in constant FR cases ($p$-value = 0.49 for interaction of constant FR and video contents using two way ANOVA on raw MOS data for videos with constant FR), where changes in viewers' ratings with the FR is insignificantly affected by video content.
In addition to conducting ANOVA over all data, we also looked at the significance of variation magnitude for a few specific cases. The impact of FR variation is significant in all the cases except ${\rm P_2}$. This means that the quality rating difference between a video with constant FR of 30 Hz, and that alternating between 30 and 15Hz, is mostly due to viewers' inconsistency.
%p-value for \Delta t * content is 0, and F-value is 3.25 using two ANOVA on normalized MOS data
%p-value for constant FR * content is 0.49, and F-value is 0.93 using two ANOVA on MOS data


% what you are saying would suggest that alpha_tv does not depend on content? But they do. I think to justify what you say would need to do ANOVA on the Q(th,tl)/Q(tl,tl) data, but evaluating deltat * content. do you have that already? What does it indicate?  I am not sure whether your interpretation here is correct. I think the means the absolute quality reduction, not the reduction relative to Q(th,th). I would just delete it now if you do not have a better explanation

%But it is interesting to observe that the effect of Fz does not happen in ${\rm P_5}$. This indicates that when the constant FR is relatively low, the changes in variation interval do not significantly influence the perceptual quality.
%The effect of video content is significant for ${\rm P_2}$-${\rm P_5}$. This suggests that the video content does affect viewer ratings for all the cases, but it may slightly influence viewer ratings at lower $t_l$ (smaller $F$-value) . As we next look at the three-way ANOVA interaction between $\Delta t$ and frequency, the all the cases are showing significant differences ($p$-value of all the cases are $< 0.05$).  In addition, the interaction between content and frequency also has significant difference for all the cases ${\rm P_2}$ - ${\rm P_5}$, while there is no significant impact of interaction between content and $\Delta t$.

Although Tab.~\ref{tab:ANOVA_FRVar} shows that Fz has a significant impact on the absolute quality rating Q($t_h,t_l$),
%and that there are significant interaction between $\delta t$ and Fz in terms of Q($t_h,t_l$),
we are interested to know the statistical significance of the influence of Fz on the dropping trend of Q($t_h,t_l$)/Q($t_h,t_h$) data, as mentioned in Sec.~\ref{ssec:Imp_FRVar}. Toward this goal, we conduct a two-way repetition ANOVA test on the Q($t_h,t_l$)/Q($t_h,t_h$) data. Results in Tab.~\ref{tab:FRVar_ANOVA_Fz} suggest that there is no significant interaction between FR variation magnitude and Fz on the Q($t_h,t_l$)/Q($t_h,t_h$) data, with $p$-value $>$ 0.05 under all combinations of $t_h$ and $t_l$ examined.

\subsection{Statistical Significance for QS Variation}\label{ssec:QPVar_ANOVA_test}
Results given under ${\rm P_1}$ in Tab.~\ref{tab:ANOVA_QPVar} are obtained by considering all test conditions together. We can see that QS variation magnitude (indicated by $\Delta q$=$q_h-q_l$), variation frequency, and video content all have significant impact on the subjective ratings.
%Unlike the cases with FR variation, there is no significant interaction between each pair of these three factors.
In addition to conducting ANOVA over all data, we also looked at a few specific cases, each comparing ratings for two ($q_l$, $q_h$) pairs. The ANOVA results show that, when $q_l$ and $q_h$ are similar, e.g., ${\rm P_2}$ and ${\rm P_3}$, the quality difference due to QS variation is insignificant, which is as expected. In particular, this result confirms that the observed change when switching from $q_h$=40 to $q_l$=25 or 16 in Fig.~\ref{fig:NMOS_ql_qh_QP_h} is not statistically significant.

%Table~\ref{tab:ANOVA_QPVar} shows that, for most cases, QP variation has significant impact on the subjective ratings except ${\rm P_4}$, which may indicate that viewers tend to ignore the quality effect when the perceived quality is already bad (where $t_l$ is quite low). This is a reasonable results since it is mostly consistent with the test results demonstrated in those figures above. Regarding how Fz affects the quality: the impact of frequency is not significant for all the cases except in ${\rm P_2}$. It addresses the fact that when $\Delta$QS is larger, the viewers will be more sensitive to the frequency change of QS variation.
%The effect of video content is significant for ${\rm P_1}$-${\rm P_5}$. This suggests that the video content does affect viewer ratings for all the cases, especially it significantly influences viewer ratings for larger $\Delta$QS. We next look at the three-way ANOVA interaction between QS variation and frequency, and it is shown that the effects are significant (all $p$-values are larger than 0.05) in ${\rm P_1}$ - ${\rm P_4}$ cases. We can learn that the perceived degradation due to QS variation is quite independent of frequency, but not for case ${\rm P_2}$. In addition, the interaction between content and frequency has no significant difference for all the cases ${\rm P_1}$ - ${\rm P_4}$, while on the contrary, we can see opposite statistics results for cases in FR variation; especially for ${\rm P_2}$, the individual factor for either frequency or content is significant, but there is no sign showing the interaction between these two factors as well as the the interaction between $\Delta$QS and content. This result is somewhat surprising and should be verified with subjective testing involving more test sequences and viewers.

To further investigate the dependency of the dropping trend of Q($q_l,q_h$)/Q($q_l,q_l$) on Fz, we conducted a two-way ANOVA test on Q($q_l,q_h$)/Q($q_l,q_l$) data. From Tab.~\ref{tab:QPVar_ANOVA_Fz}, we learn that there is no significant difference between Fz=1 and 2 nor between Fz=2 and 3, but the difference between Fz=1 and Fz=3 is significant. This results is quite consistent with the observations from Fig.~\ref{fig:QPVar_NMOS_ratioQS_invexp_4par}, and this is the basis we choose to use the same model parameter $\alpha_{qv}$ for Fz=1 and Fz=2, but a different value for Fz=3 in the ${\rm QQV}$ model in~(\ref{eq:QPVar_overallQ}).
% --------------------------------------------------------------------------
% TABLE ANOVA TEST RESULT
% --------------------------------------------------------------------------
%\begin{table}[!ht]
%\centering
%\caption{Three-way ANOVA results for QS Variation using Q($q_l,q_h$) data}\label{tab:ANOVA_QPVar}
%\begin{tabular}{c|c|c}
%\hline
%Factors & $F$-value & $p$-value\\
%%\hline
%\hline
%\hline
%\multicolumn{3}{c}{${\rm P_1}$ : All Data}\\
%\hline
%$\Delta q$ &  26.8 & 1e-5  \\
%Content &  5.25  & 9e-4  \\
%Frequency &  4.28  & 0.01  \\
%%Content*frequency &  0.07 & 0.99 \\
%%$\Delta q$*frequency & 0.54  & 0.82  \\
%%$\Delta q$*Content & 0.45  & 0.96  \\
%%$\Delta q$*Content*frequency & 0.05  & 1  \\
%\hline
%\multicolumn{3}{c}{${\rm P_2}$ : Q(40, 40) and Q(40, 25)}\\
%\hline
%$\Delta q$ & 0.02  & 0.886  \\
%%Content & 8.16  & 6.3e-3 \\
%%Frequency &  2.89  & 0.11  \\
%%Content*frequency & 1.62  & 0.25 \\
%%$\Delta$q*frequency &  2.11 & 0.18 \\
%\hline
%\multicolumn{3}{c}{${\rm P_3}$ : Q(40, 40) and Q(40, 16)}\\
%\hline
%$\Delta q$ & 2.53  & 0.15  \\
%%Content & 11.1 & 0.002 \\
%%Frequency &  3.15 &  0.09  \\
%%Content*frequency & 1& 0.5  \\
%%$\Delta$q*frequency & 3.15  &  0.09 \\
%\hline
%\multicolumn{3}{c}{${\rm P_4}$ : Q(16, 16) and Q(16, 102)}\\
%\hline
%$\Delta q$ & 3041  & 0  \\
%% Content & 20.41  & 3e-4  \\
%% Frequency &  5.52  & 0.03  \\
%% Content*frequency & 1  & 0.5\\
%% $\Delta$q*frequency & 5.52  & 0.03 \\
%\hline
%\multicolumn{3}{c}{${\rm P_5}$ : Q(102, 102) and Q(16, 102)} \\
%\hline
%$\Delta q$ & 368.79  & 0  \\
%%Content &  243.21 & 0 \\
%%Frequency &  5.52  & 0.031 \\
%%Content*frequency &  1  & 0.5 \\
%%$\Delta$q*frequency & 5.52  & 0.03  \\
%\hline
%\hline
%\end{tabular}
%\end{table}


% ------------------------------------------------------------------------
% ------------------------------------------------------------------------
\begin{table}[!ht]
\centering
\caption{Three-way ANOVA results for QS Variation using Q($q_l,q_h$) data}\label{tab:ANOVA_QPVar}
\begin{tabular}{@{~}c@{ } p{1.9 cm}|c|c|c}
\hline
& Cases & Factors & $F$-value & $p$-value\\
%\hline
\hline
\hline
%\multicolumn{3}{c}{${\rm P_1}$ : All Data}\\
\multirow{3}{*}{${\rm P_1}$:} & \multirow{3}{*}{All Data}&$\Delta q$ &  26.8 & 1e-5  \\
 &&Content &  5.25  & 9e-4  \\
 &&Frequency &  4.28  & 0.01  \\
%Content*frequency &  0.07 & 0.99 \\
%$\Delta q$*frequency & 0.54  & 0.82  \\
%$\Delta q$*Content & 0.45  & 0.96  \\
%$\Delta q$*Content*frequency & 0.05  & 1  \\
\hline
%\multicolumn{3}{c}{${\rm P_2}$ : Q(40, 40) and Q(40, 25)}\\
\hline
${\rm P_2}$:& Q(40, 40) and Q(40, 25)&$\Delta q$ & 0.02  & 0.886  \\
%Content & 8.16  & 6.3e-3 \\
%Frequency &  2.89  & 0.11  \\
%Content*frequency & 1.62  & 0.25 \\
%$\Delta$q*frequency &  2.11 & 0.18 \\
\hline
%\multicolumn{3}{c}{${\rm P_3}$ : Q(40, 40) and Q(40, 16)}\\
\hline
${\rm P_3}$:& Q(40, 40) and Q(40, 16)&$\Delta q$ & 2.53  & 0.15  \\
%Content & 11.1 & 0.002 \\
%Frequency &  3.15 &  0.09  \\
%Content*frequency & 1& 0.5  \\
%$\Delta$q*frequency & 3.15  &  0.09 \\
\hline
%\multicolumn{3}{c}{${\rm P_4}$ : Q(16, 16) and Q(16, 102)}\\
\hline
${\rm P_4}$:& Q(16, 16) and Q(16, 102)&$\Delta q$ & 3041  & 0  \\
% Content & 20.41  & 3e-4  \\
% Frequency &  5.52  & 0.03  \\
% Content*frequency & 1  & 0.5\\
% $\Delta$q*frequency & 5.52  & 0.03 \\
\hline
%\multicolumn{3}{c}{${\rm P_5}$ : Q(102, 102) and Q(16, 102)} \\
\hline
${\rm P_5}$:& Q(102, 102) and Q(16, 102)&$\Delta q$ & 368.79  & 0  \\
%Content &  243.21 & 0 \\
%Frequency &  5.52  & 0.031 \\
%Content*frequency &  1  & 0.5 \\
%$\Delta$q*frequency & 5.52  & 0.03  \\
\hline
\hline
\end{tabular}
\end{table}

% --------------------------------------------------------------------------
% TABLE ANOVA TEST RESULT
% --------------------------------------------------------------------------
\begin{table}[!htp]
\centering
\caption{The ANOVA test for the QS variation frequency using Q($q_h,q_l$)/Q($q_l,q_l$) data}\label{tab:QPVar_ANOVA_Fz}
\begin{tabular}{c|c|c|c}
\hline
Target pair & Factors & $F$-value & $p$-value\\
\hline
%\hline
%\multicolumn{4}{c}{all Data, at $q_l$=16, 40}\\
%\hline
% & $\Delta$QS &  209.86 & 0  \\
% & Fz &  3.83 & 0.02  \\
% & $q_l$ &  60.33 & 0  \\
%-& $\Delta$QS*Fz &  2.79 & 0.03  \\
% & $\Delta$QS*$q_l$ &  19.75 & 0  \\
% & $q_l$*Fz &  0.67 & 0.51  \\
% & $\Delta$QS*Fz*$q_l$ &  0.43 & 0.78  \\
\hline
\multicolumn{4}{c}{all Fz's and $q_l$'s}\\
\hline
% & $\Delta q$ &  97.2 & 0  \\
%-& Fz &  1.77 & 0.17  \\
 & $\Delta q$*Fz &  1.22 & 0.27  \\
\hline
\multicolumn{4}{c}{when $q_l$ = 28}\\
\hline
Fz = 1 and 2 & $\Delta q$*Fz &  0.56 & 0.69  \\
Fz = 1 and 3 & $\Delta q$*Fz &  2.37 & 0.06  \\
Fz =  2 and 3 & $\Delta q$*Fz &  1.38 & 0.25  \\
Fz =1, 2, 3 & $\Delta q$*Fz & 1.61 & 0.19  \\

\hline
\hline
\multicolumn{4}{c}{when $q_l$ = 36}\\
\hline
Fz = 1 and 2 & $\Delta q$*Fz &  0.19 & 0.82  \\
Fz = 1 and 3 & $\Delta q$*Fz &  3.85 & 0.03  \\
Fz =  2 and 3 & $\Delta q$*Fz &  1.54 & 0.23  \\
Fz =1, 2, 3 & $\Delta q$*Fz & 1.61 & 0.19  \\
\hline
\end{tabular}
\end{table}

% ----------------------------------------------------------


% % ------------------ End ANOVA --------------------
%\begin{table}[htp]
%\centering
%\caption{Frame rate variation pairs}\label{tab:FRVar_pair}
%\begin{tabular}{c|c}
%\hline
%Pair index & description\\
%\hline
%\hline
%${\rm P_1}$ & (30, 15) and 30 fps, \\
%${\rm P_2}$ & (15, 7.5) and 15 fps \\
%${\rm P_3}$ & (7.5, 3.75) and 7.5 fps \\
%${\rm P_4}$ & (7.5, 3.75) and 3.75\\
%\hline
%\hline
%\end{tabular}
%\end{table}

%
%\begin{table}[htp]
%\centering
%\caption{Three-way ANOVA results}\label{tab:ANOVA}
%\begin{tabular}{c|c|c|c}
%\hline
%%\multicolumn{5}{|c|}{Proposed Model}\\
%%\hline
%%$\rm RMSE$ & 1.73\% & 3.61\% & 2.1\% & 2.7\% \\
%%$\rm PC$ & 0.99 & 0.98 & 0.99 & 0.99\\
%%\hline
%Variables & degree of freedom & F-value & p-value\\
%\hline
%\hline
%$f_{avg}$ & - & - & - \\
%$QP$ & - & - & - \\
%$Content$ & - & - & - \\
%\hline
%\multirow{4}{*}{$f_{avg}$} & 1 & 29.45 & 0.00001 (${\rm C_1}$) \\
%~ & 1 & 6.2 & 0.0138 (${\rm C_2}$) \\
%~ & 1 & 6.98 & 0.009 (${\rm C_3}$) \\
%~ & 1 & 2.62 & 0.108 (${\rm C_4}$) \\
%\hline
%% $Content \ast f_{max} \ast \Delta f$ & - & - & - \\
%\hline
%
%\end{tabular}
%\end{table}


\section{Conclusion}\label{sec:Conclusion}
In this paper, we report the results of our subjective experiments to investigate the impact
of periodic FR/QS variation on the perceived video quality. We observed following interesting trends.
Regarding the FR variation, firstly, under the same average FR the quality for a video with a constant frame rate is higher than that with FR variation, alternating between $t_l$ and $t_h$; and Secondly, the degradation due to FR change is more severe when $t_h/t_l$ ratio is higher, especially when $t_h>$ 2$t_l$;
%thirdly, points corresponding to $t_l$=$t_h$/2 are quite close to the operational quality-frame rate curves achievable by using constant FR for most of the sequences. But those with $t_l$ lower than $t_h$/2 are much below the curve.
Thirdly, alternating between $t_l$ and $t_h$ is generally better than staying at $t_l$. However, the quality improvement become saturated when $t_h/t_l > 2$. Finally the variation frequency does not have significant impact on the quality decay relative to the quality with a constant FR equal to $t_h$. The last two observations are   somewhat surprising, and may not be true when Fz $<$ 1 sec.
%We further find that the variation frequency does not have a significant impact on the perceived quality. This suggests that when a video switches between two frame rates, switching faster or slowly does not affect the perceptual quality significantly.

Similar phenomena have also been observed under QS variation. Firstly, under the same average QS, a video with a constant QS is perceptually more appealing than a video with variable QS under the same average QS. Secondly, the degradation due to QS change is more severe when constant low QS $q_l$ is relatively low and $q_h/q_l$ ratio is high. Thirdly, alternating between $q_h$ and $q_l$ is generally equal or better than a video with a constant high QS $q_h$. However, the improvement becomes saturated when $q_l/q_h<$ 0.4.
Unlike the case with FR variation, the variation frequency does affect the quality decay relative to the quality with a constant QS equal to $q_l$. Overall, slower variation leads to less quality decay, as expected.

These results provide several important guidelines for rate adaptation in video transmission over dynamically changing networks. For example, when the underlying application allows sufficiently long delay, it is better to code a video at constant FR and QS, and use a large buffer at the sender to deal with the bandwidth variation. When the delay constraint is stringent and one cannot afford to use large buffers, at least in the range of variation frequency considered here (Fz $>=$ 1 sec), it is better to allow the rate to vary with the available bandwidth (by adjusting FR or QS), rather than staying at the lower rate. This suggests that trying to keep a smooth but low video quality may not be the right strategy for video streaming in dynamic networks. Allowing  a certain amount of quality fluctuation may in fact lead to better quality.
However, the system should limit the amount of changes in FR or QS. Specifically, when the lowest bandwidth demands a FR=$t_l$ and QS=$q_h$, the system should limit the FR to at most 2$t_l$, and QS to at least 0.4$q_h$, even if the instantaneous bandwidth allows higher $t$ or lower $q$ occasionally. When the rate reduction can be made by changing either QS or FR, it is suggested that increasing QS is preferred than decreasing FR to achieve the least quality degradation. By comparing video quality under the same periodic rate variation, we further found that adapting QS is better than changing FR, when the high rate point allows the video to be coded at very good quality (e.g. with high FR and low QS). Somewhat surprising, this is observed for video with either low or high motion.

%, when $q_h$ is high and the improvement is more significant when Fz is longer; and thirdly, the quality degradation due to QS variation between $q_l$ and $q_h$, compared to the quality achievable with a constant low QS $q_l$ follows an inverse exponential function of the ratio $q_l/q_h$, and the dropping rate is slower with a longer Fz.

Based on the observed relation between the quality and FR or QS variations, we propose several quality models. For FR variation, the model ${\rm QTV}$ determines the quality of a video with alternating frames rates of $t_h$ and $t_l$ as the product of two terms. The first term reflects the quality of a video at a constant FR of $t_h$ and the second term indicates the degradation due to the switching between $t_h$ and $t_l$, and depends only on the ratio $t_l/t_h$. Similarly, we have developed ${\rm QQV}$ model for videos with QS variation. We note that these models are proposed based on the trends observed from the test sequences used in our subjective tests, and they need to be validated using other test sequences. In this study, the model parameters are derived using least square fitting of the model with the mean opinion scores for each sequence. For the model to be useful, one must also study how to predict the model parameters from the video content. Furthermore, rather than predicting the model parameter $\alpha_t$ for each $t_h$, it may be possible to find a simple functional form to model  $\alpha_t$($t_h$). Similarly, one may examine and model the underlying relation of $\alpha_q$ with $q_h$ and Fz. These are interesting directions for future studies.

%$Although the ${\rm QTV}_2$ and ${\rm QQV}_2$ models are less accurate than ${\rm QTV}_1$ and ${\rm QQV}_1$, respectively, they require only a single content-dependent parameter, and still provide high accuracy with PCC between 0.96 and 0.97.

%Regarding the analytical modeling, the quality degradation due to FR/QS variation follows a exponential function of FR/QS ratio $t_l/t_h$ (or $q_l/q_h$). The overall quality for a given $t_l/t_h$ (or $q_l/q_h$) can be modeled by either the product of two exponential functions or one exponential function. The former one consists of one sub-model for constant FR/QS and the other for FR/QS variation. The later one utilizes one simple function, which is dependent of only $t_l$ (or $q_h$), to approximate the quality degradation for both constant or variant FR/QS. Although the model performance of later one is not as good as the former one, it require less number of parameters (one content-dependent parameter, $\alpha_{tv}/\alpha_{qv}$) and acceptable prediction accuracy with PCC value of around 0.96-0.97. We also conducted three-way ANOVA to evaluate the statistical significance of the impact of FR/QS variation, changing interval and video content on the perceived quality.

This paper focuses only on the impact of periodic frame rate/QS changes on the perceptual quality. One challenging issue in modeling the impact of FR and QS variation on the quality is how to design the subjective test to help understand the effect of infinitely many likely variation patterns. Our subjective study that considers only periodic variations is only the first step towards this challenge, and is motivated by the fact that any pattern can be decomposed into periodic patterns with different frequencies. Our study so far considered only the effect of FR and QS variation, individually, and under a few variation frequencies. Future studies may examine a wider range of variation frequency, variation of spatial resolution, and joint impact of FR, QS and spatial resolution variation. %Regarding the modeling part, it seems that $\alpha_{tv}$, $\alpha_{qv}$ are quite similar among most sequences. It is worth to study the relationship between the parameters and video contents.  One can categorize the videos into a few categories, and train some fixed parameters into each category. So in practice, only a fixed set of parameters ($\alpha_{tv}$, $\alpha_{qv}$, etc) will be used for all sequences. In addition, even though $\alpha_{tv}$ depends on $t_h$ ($\alpha_{qv}$ depends on $q_l$), there seem to be some consistent relations between parameters under different $t_h$ (or $q_l$), which may be deduced with further investigation.

%We are currently examining the impact of other factors including frame rate variation frequency, variation of QS, and variation of both frame rate and QS. Based on findings from these as well as other subjective tests, we hope to model the video quality over a certain duration with STAR variation, as a function of the average STAR as well as some statistics characterizing their temporal variation.


% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEtran}
%\small{\bibliography{QP_FR_Variation}}
\begin{thebibliography}{1}

\bibitem{G1070}
{ITU-T Recommendation G.1070}, ``{Opinion Model for Video-Telephony
  Applications},'' Apr. 2007.

\bibitem{QualityMetricTQuantBitRate}
{R.~Feghali et al.}, ``{Video quality metric for bit rate control via joint
  adjustment of quantization and frame rate},'' \emph{IEEE Trans. on
  Broadcasting}, vol.~53, no.~1, pp. 441--446, Mar. 2007.

\bibitem{yenfu_csvt}
Y.-F. Ou, Z.~Ma, T.~Liu, and Y.~Wang, ``{Perceptual Quality Assessment of Video
  Considering both Frame Rate and Quantization Artifacts},'' \emph{IEEE Trans.
  On Circuit Syst. Video Technology (CSVT'11)}, vol.~21, no.~3, pp. 286--298,
  Mar. 2011.

\bibitem{TemporalMobileVideoBroadcasting}
{H.-T. Quan et al.}, ``{Temporal Aspect of Perceived Quality of Mobile Video
  Broadcasting},'' \emph{IEEE Trans. on Broadcasting}, vol.~54, no.~3, pp.
  641--651, Sept. 2008.

\bibitem{PerceptualTemporalQualityMetrics}
{K.-C.~Yang et al.}, ``{Perceptual Temporal Quality Metric for Compressed
  Video},'' \emph{IEEE Trans. on Multimedia}, vol.~9, pp. 1528--1535, Nov.
  2007.

\bibitem{Baroncini_2006}
V.~Baroncini, D.~Ciavatta, G.~Gaudino, R.~Felice, G.~Iacovoni, and F.~Ubaldi,
  ``Variable frame rate video for mobile devices,'' in \emph{Proceedings of the
  2nd international conference on Mobile multimedia communications}, ser.
  MobiMedia '06.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2006, pp. 1--6.

\bibitem{Moorty_Mobile_VQA_2012}
A.~K. Moorthy, L.~K. Choi, A.~Bovik, and G.~deVeciana, ``{Mobile Video Quality
  Assessment Database},'' in \emph{IEEE International Conference on
  Communications (ICC)}, Jun. 2012.

\bibitem{yenfu_icassp}
Y.-F. Ou, Y.~Zhou, and Y.~Wang, ``{Perceptual Quality of Video with Frame Rate
  Variation : a subjective study},'' in \emph{IEEE International Conference on
  Acoustics, Speech, and Signal Processing (ICASSP10')}, Mar. 2010, pp. 2446 --
  2449.

\bibitem{yenfu_QPVar_icip}
Y.-F. Ou, H.~Zeng, and Y.~Wang, ``Perceptual quality of video with quantization
  variation:a subjective study and analytical modeling,'' in \emph{IEEE
  International Conference on Image Processing (ICIP)}, 2012.

\bibitem{JVT_site}
 [Online]. Available: \url{http://wftp3.itu.int/av-arch/jvt-site/}


\bibitem{JSVM}
{\it Joint Scalable Video Model}, ``{Joint Video Team (JVT) of ISO/IEC MPEG
  $\&$ ITU-T VCEG, Doc. JVT-X202},'' Jul. 2007.

\bibitem{RecP910}
{ITU-R Rec. P.910}, ``{Subjective Video Quality Assessment Methods for
  Multimedia Applications},'' 1999.

\bibitem{AM_zscore}
A.~M. van Dijk, J.~B. Martens, and A.~B. Watson, ``{Quality Assessment of Coded
  Images using Numerical Category Scaling},'' in \emph{Proc. SPIE}, vol. 2451,
  1995, p. 90�101.

\bibitem{BT500}
{ ITU-R Rec. BT.500-11}, ``{Methodology for the subjective assessment of the
  quality of television pictures},'' 2002.

\bibitem{QPFRVar_dataset}
{NYUPoly Video Lab database:FR/QS Variation subjective quality evaluation}.
  [Online]. Available:
  \url{http://vision.poly.edu/index.html/index.php?n=HomePage.ScalableVideoQualityWithFrameRateAndQuantizationArtifacts}

\bibitem{Snedecor_Statistics}
{G. W. Snedecor et al.}, \emph{Statistical Methods, 8th ed Ames}.\hskip 1em
  plus 0.5em minus 0.4em\relax IA: Iowa State Univ. Press, 1989.

\end{thebibliography}
%}
\end{document}



1406.2018.extracted_bib
{G1070 {ITU-T Recommendation G.1070, ``{Opinion Model for Video-Telephony Applications,'' Apr. 2007.
1406.2018.extracted_bib
{QualityMetricTQuantBitRate {R.~Feghali et al., ``{Video quality metric for bit rate control via joint adjustment of quantization and frame rate,'' IEEE Trans. on Broadcasting, vol.~53, no.~1, pp. 441--446, Mar. 2007.
1406.2018.extracted_bib
{yenfu_csvt Y.-F. Ou, Z.~Ma, T.~Liu, and Y.~Wang, ``{Perceptual Quality Assessment of Video Considering both Frame Rate and Quantization Artifacts,'' IEEE Trans. On Circuit Syst. Video Technology (CSVT'11), vol.~21, no.~3, pp. 286--298, Mar. 2011.
1406.2018.extracted_bib
{TemporalMobileVideoBroadcasting {H.-T. Quan et al., ``{Temporal Aspect of Perceived Quality of Mobile Video Broadcasting,'' IEEE Trans. on Broadcasting, vol.~54, no.~3, pp. 641--651, Sept. 2008.
1406.2018.extracted_bib
{PerceptualTemporalQualityMetrics {K.-C.~Yang et al., ``{Perceptual Temporal Quality Metric for Compressed Video,'' IEEE Trans. on Multimedia, vol.~9, pp. 1528--1535, Nov. 2007.
1406.2018.extracted_bib
{Baroncini_2006 V.~Baroncini, D.~Ciavatta, G.~Gaudino, R.~Felice, G.~Iacovoni, and F.~Ubaldi, ``Variable frame rate video for mobile devices,'' in Proceedings of the 2nd international conference on Mobile multimedia communications, ser. MobiMedia '06.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2006, pp. 1--6.
1406.2018.extracted_bib
{Moorty_Mobile_VQA_2012 A.~K. Moorthy, L.~K. Choi, A.~Bovik, and G.~deVeciana, ``{Mobile Video Quality Assessment Database,'' in IEEE International Conference on Communications (ICC), Jun. 2012.
1406.2018.extracted_bib
{yenfu_icassp Y.-F. Ou, Y.~Zhou, and Y.~Wang, ``{Perceptual Quality of Video with Frame Rate Variation : a subjective study,'' in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP10'), Mar. 2010, pp. 2446 -- 2449.
1406.2018.extracted_bib
{yenfu_QPVar_icip Y.-F. Ou, H.~Zeng, and Y.~Wang, ``Perceptual quality of video with quantization variation:a subjective study and analytical modeling,'' in IEEE International Conference on Image Processing (ICIP), 2012.
1406.2018.extracted_bib
{JVT_site [Online]. Available: \url{http://wftp3.itu.int/av-arch/jvt-site/
1406.2018.extracted_bib
{JSVM {Joint Scalable Video Model, ``{Joint Video Team (JVT) of ISO/IEC MPEG $\&$ ITU-T VCEG, Doc. JVT-X202,'' Jul. 2007.
1406.2018.extracted_bib
{RecP910 {ITU-R Rec. P.910, ``{Subjective Video Quality Assessment Methods for Multimedia Applications,'' 1999.
1406.2018.extracted_bib
{AM_zscore A.~M. van Dijk, J.~B. Martens, and A.~B. Watson, ``{Quality Assessment of Coded Images using Numerical Category Scaling,'' in Proc. SPIE, vol. 2451, 1995, p. 90�101.
1406.2018.extracted_bib
{BT500 { ITU-R Rec. BT.500-11, ``{Methodology for the subjective assessment of the quality of television pictures,'' 2002.
1406.2018.extracted_bib
{QPFRVar_dataset {NYUPoly Video Lab database:FR/QS Variation subjective quality evaluation. [Online]. Available: \url{http://vision.poly.edu/index.html/index.php?n=HomePage.ScalableVideoQualityWithFrameRateAndQuantizationArtifacts
1406.2018.extracted_bib
{Snedecor_Statistics {G. W. Snedecor et al., Statistical Methods, 8th ed Ames.\hskip 1em plus 0.5em minus 0.4em\relax IA: Iowa State Univ. Press, 1989.