A comprehensive error rate for multiple testing
Djalel Eddine Meskaldji\footnote{Signal Processing Laboratory (LTS5)
Jean-Philippe Thiran*
Stephan Morgenthaler\footnote{FSB/MATHA
Ecole Polytechnique F\'{e}d\'{e}rale de Lausanne (EPFL), Lausanne, Switzerland.}
In multiple testing, a variety of control metrics for false positives have been introduced, among them the Family-Wise Error Rate (FWER), the Per-Family Error Rate (PFER), the False Discovery Rate (FDR), etc. In this paper we propose a comprehensive family of error rates together with a corresponding family of multiple testing procedures (MTP). Based on the needs of the problem at hand, the user can choose a particular member among these MTPs. The new error rate limits the number of false positives relative to an arbitrary non-decreasing function $s$ of the number of rejections $R$, that is, $\mathrm{FP}/s(R)$. An example is $\mathrm{FP}/(R\wedge \tau)$, the ratio of the number of false positives divided by the smaller among $\tau$ and all rejections. This particular error rate accepts a fixed percentage of false rejections among all rejections, but only up to $R=\tau$, then a stricter control takes over and for $R>\tau$, FP is limited to a percentage of $\tau$. We investigate both the control of quantiles and of expectations and describe the corresponding multiple testing procedures. In the above example, the expectation of the criterion leads to a family of multiple testing procedures that bridges the gap between the PFER $(\tau=1)$ and the FDR $(\tau=\infty)$.
Manuscript.tex
\documentclass{article}[12pt]

\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{natbib}
\usepackage{theorem}
\usepackage{setspace}
\usepackage{amssymb}
\usepackage{color}
\usepackage{lineno}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proof}{Proof}[theorem]
\newtheorem{acknowledgement}[theorem]{Acknowledgement}
\newtheorem{algorithm}[theorem]{Algorithm}
\newtheorem{axiom}[theorem]{Axiom}
\newtheorem{case}[theorem]{Case}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{conclusion}[theorem]{Conclusion}
\newtheorem{condition}[theorem]{Condition}
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{criterion}[theorem]{Criterion}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\newtheorem{exercise}[theorem]{Exercise}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{notation}[theorem]{Notation}
\newtheorem{problem}[theorem]{Problem}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{procedure}[theorem]{Procedure}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{solution}[theorem]{Solution}
\newtheorem{summary}[theorem]{Summary}

\newcommand{\V}{{\mathsf{FP}}}
\newcommand{\T}{{\mathsf{TP}}}
\newcommand{\E}{E}
\newcommand{\Pro}{P}
\newcommand{\pro}{P}
\newcommand{\prob}[1]{\mathbb{P}\paren{#1}}
\newcommand{\Rj}{{\mathcal{R}}}
\newcommand{\fwer}{\mbox{FWER}}
\newcommand{\pfer}{\mbox{PFER}}
\newcommand{\fdr}{{\mbox{FDR}}}
\newcommand{\lsu}{\mbox{LSU}}
\newcommand{\tlsu}{\tau\mbox{LSU}}
\newcommand{\fdp}{{\mbox{FDP}}}
\newcommand{\fer}{{\mbox{FER}}}
\newcommand{\Hyp}{\mathcal{H}}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\ind}[1]{{\mathbf{1}\set{#1}}}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\st}{\;|\;}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\rbar}{\overline r}
\newcommand{\cbar}{\overline c}
\newcommand{\sfdp}{\mbox{SFDP}}
\newcommand{\sev}{\mbox{SEV}}
\newcommand{\stp}{\mbox{STP}}
\newcommand{\Hs}{H_i, i\in I = \{1, \ldots,m\}}
\newcommand{\pvalues}{p_1,\ldots,p_m}
\newcommand{\opvalues}{p_{(1)},\ldots,p_{(m)}}
\newcommand{\rpvalues}{p_{1},\ldots,p_{m}}
\newcommand{\pvec}{\mathbf{p}}
\newcommand{\mcp}{\mathcal{M}}
\newcommand{\Phibar}{\overline{\Phi}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\distsu}{\mathcal{D}}
\newcommand{\distsd}{\widetilde{\mathcal{D}}}
\newcommand{\Sti}[2]{\genfrac{\{}{\}}{0pt}{}{#1}{#2}}
\renewcommand{\l}{\ell}
\newcommand{\SU}{\mbox{SU}}
\newcommand{\SD}{\mbox{SD}}
\newcommand{\cH}{{\mtc{H}}}
\newcommand{\mtc}{\mathcal}
\newcommand{\ol}[1]{\overline{#1}}
\newcommand{\PExI}{\ol{P}^{I}}
\newcommand{\PExN}{\ol{P}^{N}}
\newcommand{\gam}{\Gamma(i, r)}
\newcommand{\gw}{\frac{\alpha}{m} w_i}
\newcommand{\Pow}{\mbox{Pow}}
\newcommand{\cf}{\mtc{F}}
\newcommand{\lla}{\mtc{L}_{\lambda}}


\begin{document}
\title%[A comprehensive error rate]
{A comprehensive error rate for multiple testing}
\author{Djalel Eddine Meskaldji\footnote{Signal Processing Laboratory (LTS5), Ecole Polytechnique F\'{e}d\'{e}rale de Lausanne (EPFL), Lausanne, Switzerland. Email: djalel.meskaldji@epfl.ch.} \footnote{This work was supported in part by the FNS grant N$^0$144467.}
\and Jean-Philippe Thiran* \and Stephan Morgenthaler\footnote{FSB/MATHA, Ecole Polytechnique F\'{e}d\'{e}rale de Lausanne (EPFL), Lausanne, Switzerland.}}
\maketitle
%\newpage
\doublespacing
%\linenumbers
\begin{abstract}
In multiple testing, a variety of control metrics for false positives have been introduced, among them the Family-Wise Error Rate (FWER), the Per-Family Error Rate (PFER), the False Discovery Rate (FDR), etc. In this paper we propose a comprehensive family of error rates together with a corresponding family of multiple testing procedures (MTP). Based on the needs of the problem at hand, the user can choose a particular member among these MTPs. The new error rate limits the number of false positives relative to an arbitrary non-decreasing function $s$ of the number of rejections $R$, that is, $\mathrm{FP}/s(R)$. An example is $\mathrm{FP}/(R\wedge \tau)$, the ratio of the number of false positives divided by the smaller among $\tau$ and all rejections. This particular error rate accepts a fixed percentage of   false rejections among all rejections, but only up to $R=\tau$, then a stricter control takes over and for $R>\tau$, FP is limited to a percentage of $\tau$. We investigate both the control of quantiles and of expectations and describe the corresponding multiple testing procedures. In the above example, the expectation of the criterion leads to a family of multiple testing procedures that bridges the gap between the PFER $(\tau=1)$ and the FDR $(\tau=\infty)$.
\end{abstract}


%\keywords
\noindent Keywords: {Multiple comparisons, Family-Wise Error Rate, False Discovery
  Rate, ordered p-values.}

\section{Introduction}
Consider a set of $m$ hypotheses $H_i, i\in I = \{1, \ldots,m\}$ to be tested. Let $H_i =0$ if the hypothesis is true and $H_i=1$ otherwise. Let $I_0 =\{i\in I: H_{i}=0\}$ with $|I_0|=m_0$ and $I_1=I \backslash I_0$ with $|I_1|=m_1=m-m_0$. The hypotheses are associated to the p-values $\{p_1,\ldots,p_m\}$.

A multiple testing procedure (MTP) is a decision function $\mathcal{D}: [0,1]^m \rightarrow \{0;1\}^m$ that associate to each set of p-values $\{p_1,\ldots,p_m\}$ a vector $\{R_1,\ldots,R_m\}$ where $R_i =1$ if the hypothesis $H_i$ is rejected and $0$ otherwise. A {\em rejection set} ${\cal R}$ is a subset of $I$ that indicates the rejected hypotheses, that is, ${\cal R}=\{i \in I: R_i=1\}.$ The number of erroneously rejected hypotheses is the number of false positives $\mathrm {FP} =|\mathcal {R} \cap I_0|$.
For example, the non-multiplicity correction procedure (NMCP) consists in rejecting $H_i$ if $p_i$ is less than a predefined level $\alpha$, that is, ${\cal R}=\{ i \in I : p_i\leq\alpha\}.$
The outcome of a multiple testing procedure is summarized in Table \ref{general outcome}.
%The probability of committing at least one false rejection is called \emph{the family wise error rate (FWER)}
%\begin{equation}\label{example}
%\mathrm{FWER}= P(\mathrm{FP>0}).
%\end{equation}

%The expected number of false positives is called \emph{the Per Family Error Rate}, $\mathrm{PFER}= E(|\mathcal {R} \cap J^N|) = \mathbb E(\mathrm{FP})$.

\begin{table}
\centering
\begin{tabular}{|c|cc|c|}\hline
&$ i \in I\backslash \cal R$&$i \in \cal R$&\\ \hline
$I_0$ & $\mathrm{TN}$ & $\mathrm{FP}$ & $m_{0}$ \\
$I_1$ & $\mathrm{FN}$ & $\mathrm{TP}$ & $m_{1}$ \\
\hline
Total& $m-R$ & $R$ & $m$ \\\hline
\end{tabular}
\caption{\label{general outcome} General outcome of applying an MTP. FP denotes the number of false positives, $\mathrm{FN}$ denotes the number of false negatives, $\mathrm{TP}$ denotes the number of true positives and $\mathrm{TN}$ denotes the number of true negatives. $R$ is the total number of rejections. $\mathrm{TP/}m_1$ is an estimator of the average power.}

\end{table}
The idea that the control of false positives FP should be considered in conjunction with the number of rejections $R=|\mathcal{R}|=\sum_{i=1}^m R_i$ has been widely accepted by users after the introduction of the False Discovery Proportion $\mathrm{FDP}=\mathrm{FP}/(R\vee 1)$ and the False Discovery Rate $\mathrm{FDR}=E\left[\mathrm{FDP}\right]$. Nevertheless, in situations where false discoveries have expensive consequences, the much stricter control provided by the $\mathrm{FWER}=P[\mathrm{FP}>0]$ remains a viable alternative. Our comprehensive error rate is based on the scaled false discovery proportion $\mathrm{SFDP}=\mathrm{FP}/s(R \vee 1)$, where $s$ is a non-decreasing scaling function which typically grows more slowly than $R$ itself. This simple device covers and generalizes almost all the existent error rates. In Section \ref{Definition of the concept}, we propose multiple testing procedures that control two different stochastic functions of the SFDP under different assumptions. The main results are the control of the expectation using a step-up (SU) procedure, and the control of a tail probability by an adaptive step-down (SD) procedure.
Additionally to the example of scaling functions given in the abstract $s(R)=(R\wedge \tau),$ that bridges the gap between the $\mathrm{PFER}=E[\mathrm{FP}]$ and the FDR, a similar such bridge is obtained when $s(R)=R^\gamma$ with $0\leq\gamma \leq 1$, for $\gamma=0$ and $\gamma=1$ respectively. In Section \ref{Scaling functions section}, we discuss the choice of the parameters $\tau$ and $\gamma$ based on the maximization of the expected gain $\mathcal{G} =E[\mathrm{TP}] - \lambda E[\mathrm{FP}],$ which is based on the idea that a false positive costs a penalty of $\lambda>1$ units, while a true positive corresponds to a gain of $1$ unit.

\section{Historical background and motivation}\label{Motivation}

Traditional MTPs control the FWER, that is, they guarantee $\mathrm{FWER}\leq \alpha$ for an $\alpha$ chosen by the user. A famous such MTP is the \cite{bonferroni1936} procedure which performs each of the $m$ tests at level $\alpha/m$, that is, $\mathcal R =\{i \in I: p_i\leq \alpha/m\}$. The Bonferroni procedure is the simplest and the strongest procedure in terms of control of FP. Besides the FWER, it also controls the $\mathrm{PFER}$ at level $\alpha$. However, when $m$ grows, the power of the Bonferroni procedure at any fixed alternative tends to $0$.

Many other MTPs that control the FWER have been proposed, although they
typically give only a slight improvement over the Bonferroni
procedure. Most of them compare the ordered p-values to thresholds which depend
on the global control level $\alpha$ and the rank of the p-value. Let
$p_{(i)}$ denote the p-value with rank $i$ and let $(t_1,\ldots,t_m)_{1 \leq t_i \leq m}$ be an
non-decreasing sequence of threshold values, that is, $t_1\leq
t_2\leq\cdots\leq t_m$.  A step-down procedure (SD) defines the rejection set by ${\cal R} = \{ i:\ p_{(i)} \leq p_{(U)}\}$ where $U = \max\{ i: p_{(1)} \leq t_1,\ldots,p_{(i)} \leq t_i\}$, that is, $U+1$ is the first rank exceeding the barrier. On the other hand, a step-up procedure (SU) defines the rejection set by ${\cal R} = \{ i:\ p_{(i)} \leq p_{(U)}\}$ where $U = \max\{ i: p_{(i)} \leq t_i\}$. %procedure compares the ordered p-values to the thresholds, starting with the largest or the least significant p-value. It looks for the first down-crossing of the thresholds and rejects all hypotheses with smaller or equal p-values. In other words, it computes the rank $r$ which satisfies $p_{(r)}=\max\{p_{(k)}\leq t_k\mbox{ for }k=1,2,\ldots\}$ and rejects the hypotheses up to and including rank $r$. None of the hypotheses is rejected, if all the p-values are bigger than their thresholds. A step-down (SD) procedure starts with the smallest or the most significant p-value and is based on the first up-crossing, that is, it computes $p_{(r)}=\min\{p_{(k)}> t_k\}$. The hypotheses corresponding to the ranks $1,2,\ldots,r-1$ are rejected. If we find $r=1$, then no hypothesis is rejected. Examples of step-wise procedures include \cite{Holm1979}, \cite{SIMES1986}, \cite{HOCHBERG1988}.
%Since the first down-crossing is the same as the last up-crossing, the SU procedure and the SD procedures with a given threshold $t_r$ can be compared.
A SU procedure rejects at least as many hypotheses as a SD procedure that uses the same set of thresholds \citep{HornDunnett2004}. In the Bonferroni case, the thresholds are constant $t_i\equiv\alpha/m$ and the SU agrees with the SD.

Safeguards against false positives is not the unique purpose of testing. Detecting real effects is also of great importance. \cite{Victor82}, for example,
considered the k-FWER $=P\left( \mathrm{FP}\geq k\right)$, which tolerates more false positives and thus increases the power. This seems appropriate when the
number of hypotheses $m$ is large.  \cite{HommelHoffmann88} and \cite{LehmannRomano2005} derived a single step and a step-down procedures to control the k-FWER.  The single step procedure is identical to the Bonferroni procedure except that the p-values are compared to $t_i\equiv k \alpha /m$ for some $k \geq 1$ instead of $t_i \equiv \alpha /m$. This procedure is evidently more powerful than the Bonferroni procedure. However, the weak control of the FWER at level $\alpha $ is no longer guaranteed. In fact, the expected number of false positives under the complete null hypothesis ($m_{0}=m$) is $k \alpha$.

Also, in the aim of increasing the power of testing, \cite{BenjaminiFDR1995} introduced the FDR. Since $$ \mathrm{FDR}= E[\mathrm{FDP}] = E\left[\frac{\mathrm{FP}}{R}\;\middle\vert\;R>0\right]P[R>0] \leq P [\mathrm{FP}>0],$$ the FDR is
less stringent than the FWER, which leads to more rejections. The FDR has the same behavior as the FWER when all hypotheses are true, that is, when $m=m_0$, since  $\mathrm{FP}=R$. Any MTP that bounds the FDR also provides weak control of the FWER with the same bound. The paper cited above proves that the linear SU procedure with thresholds $(t_i\equiv i \alpha/m)_{1\leq i\leq m}$ guarantees that $\mathrm{FDR}\leq \alpha$. Since the FDP measures the proportion of false rejections among all rejections, it is to be expected that the number of false positives increases with the number of rejections $R$. Furthermore, it is shown that for $m$ not large, the convergence of the FDP to its expectation is slow \citep{Roquain2011exact}. Despite this drawback, the FDR has been widely adopted in many fields of application. % and it is fair to say that the paper of \cite{BenjaminiFDR1995} had a huge impact on the practice of statistics, with more than 10,000 citations in 2012.
On the other hand, \cite{LehmannRomano2005} derived step-wise procedures to control %another alternative error metric that uses the FDP,
the FER, which is defined by $P\left[\mathrm{FDP}> \beta \right]$ with $\beta\in(0,1)$.

Many other concepts of false positives error rates have been proposed in the literature. All these concepts have a certain control of false positives situated in between two extremes, the Per Comparison Error Rate $\mathrm{PCER} = E(\mathrm{FP}/m)$ and the PFER control.
\cite{DudoitLaan2007} and \cite{BenjaminiSI2010} are good sources for additional information.

The false positives metrics can be classified via two obvious criteria. First, one can distinguish between metrics that control the probability of exceeding a constant %such as the FWER, the k-FWER or the FER,
and metrics that control the expected number of a certain quantity. % such as the PFER and the FDR. % and the k-FDR or the positive FDR (pFDR) (\cite{Storey02}).
Second, one distinguishes between metrics that do not consider the number of rejections $R$ %such as the FWER, the k-FWER or the PFER,
and metrics that consider $R$ and tolerate more false positives as more hypotheses are rejected. %, such as the FDR and the FER.
Table \ref{error metrics summary} summarizes this information and is intended to assist the reader in navigating the jungle of abbreviations.
\begin{table}[h]
\begin{tabular}{|l|l|l|}\hline
Type& Tail probability & Expectation \\
\hline
Not dependent on R & $\mathrm{FWER}=P[\mathrm{FP} \geq 0] \leq \alpha$ & $\mathrm{PFER}=E[\mathrm{FP}]\leq \alpha$  \\
&k-FWER$=P[\mathrm{FP} \geq k]\leq \alpha$ & $\mathrm{PCER}=E[\mathrm{FP}/m]\leq \alpha$ \\
\hline
Dependent on R & $\mathrm{FER}=P[\mathrm{FP}/R>\beta]\leq \alpha$ &
$\mathrm{FDR}=E[\mathrm{FP}/R]\leq \alpha$ \\\hline
%&& k-$\mathrm{FDR}=E((V-k)_{+}/R)\leq \gamma$ \\
%&& $\mathrm{pFDR}=E(V/R|R>0)\leq \gamma$ \\
\end{tabular}
\caption{\label{error metrics summary}Definition of some Type I error metrics.}
\end{table}

%The error rates that control the proportion of false positives are especially appealing for large scale testing problems, compared to error rates that do not consider the number of rejections, as they maintain power when the number of tests $m$ increases. See \cite{DudoitLaan2007}.
% xxx: what does remain stable mean, give a page number
%The range of metrics is confusing, especially for non experts. In addition, most of the procedures need additional assumptions in order to become attainable. If one or more of theses assumptions are not satisfied, a control failure results.
According to \cite{BenjaminiSI2010}, none of the metrics is superior in all aspects. In fact users may wish to have several distinct controls achieved by one procedure. For example, one might be willing to derive a powerful procedure that weakly controls the FWER. The linear SU procedure seems ideal, but at the risk of large values of $\mathrm{FP}$ when $R$ becomes large. To avoid this, one might want to add strong control of the k-FWER as the single step of \cite{HommelHoffmann88} does. Our comprehensive error rate offers such compromises. If we choose the scaling function from the introduction, $s_\tau(R)=R$ for $R\leq \tau$ and $s_\tau(R)= \tau$ for $R\geq \tau$, the scaled error rate demands for control of $E[\mathrm{FP}/R]$ up to $\tau$ rejections, but then switches to the control of $E[\mathrm{FP}]/\tau$. We call this procedure the truncated linear SU procedure ($\tau\mathrm{SU}$). For $\tau=1$, this results in control of the FWER, while for $\tau=m$ we obtain the FDR, thus providing an example of a family of scaling functions that bridges the gap between these two extremes.  We can show that for independent or positively dependent
test statistics \citep{BY01}, the $\tau$SU procedure controls the FDR to
be less than $\alpha$ and controls the PFER to be less than $\tau \alpha$
which implies the control of the $\tau$-$\mathrm{FWER}$ at level $\alpha$ since $%
P\left[ \mathrm{FP}\geq \tau\right] \leq E\left[ \mathrm{FP}/\tau\right] $ by Markov's
inequality. The use of $s_{\tau}(R)$ is appropriate in
applications where the number $m$ of hypotheses being tested is very large (of
the order of $10^5$ or even $10^6$). Examples are fMRI as well as some
genomic studies. This procedure has the control of the FDR when the number
of rejections $R$ is small, but puts a firm limit on the expected number of
false positives as $R$ grows.
%\begin{procedure}
%  We test $M$ hypotheses. Let $p_{\left( 1\right) }\leq p_{\left( 2\right)}\leq \cdots \leq p_{\left( M\right) }$ be the ordered
%  p-values, and denote by $H_{(i) }$ the null hypothesis that corresponds to $p_{(i) }$. Let $r$ be the largest rank that
%  satisfies simultaneously $p_{(r)}\leq \frac{r\alpha}{m}$ as well as
%  $p_{(r)}\leq \frac{\tau \alpha}{m}$, that is,
%  $p_{(r)}\leq t_r=\frac{s_\tau(r)\alpha}{m}$; then reject all $ H_{\left(
%      i\right) }$ with $i=1,2,...,r$.
%\end{procedure}

Theorem \ref{PartI: SEV control indep} gives a general result of
this type. In order to control the rate $E[\mathrm{FP}/s(R \vee 1)]$, the sequence of
thresholds to be used is $(t_i=s(i)\alpha/m)_{1\leq i\leq m}$, that is, the threshold values
grow like the scaling function. Other papers have gone beyond the two popular scaling functions $s(i)$, constant or growing linearly. \citet[p. 508 and p. 513]{genovese2002operating} allude to it as a possibility and show that asymptotically the expected false non-discovery rate is minimized by the linear choice. \citet{Finner2009false} derive an alternative $s(i)$ also based on asymptotic arguments, while \citet{Roquain2011exact} investigate the operating characteristics of FDR under a general increasing sequence of
threshold values $(t_i)_{1 \leq i \leq m}$. But none of these papers make the connection to a generalization of the error criterion.


\section{The scaled error rates and control procedures}\label{Definition of the concept}

The SFDP takes into account the number of rejections when controlling false
positives. This concept has a Bayesian background in the sense that $R$
contains information about $m_{1}$, which should be exploited when making
decisions about rejections.% The SFDP is a generalization of the FDP.

\begin{definition}
Let $s: \{1,...,m\} \to (0,\infty)$ be a non-decreasing function. The Scaled False Discovery Proportion $\mathrm{SFDP}$ with scaling function $s$ is defined by
\[
\mathrm{SFDP}=\left\{
\begin{array}{l}\frac{\mathrm{FP}}{s\left( R\right) },\ \mathrm{ if }\ R\geq 1, \\
0,{ \  \text{otherwise}\,.}
\end{array}
\right.
\]
\end{definition}
Based on this quantity we define two types of general error rates using two
different stochastic functions, the expectation and the tail probability of
exceeding a constant. %, which we denote as scaled Tail Probability (STP). The second uses the expected value of the SFDP, which we denote as scaled expected value (SEV).
Related concepts were introduced in
\cite{VDL2004} and described in \citet[p. 238 and ch. 6,
7]{DudoitLaan2007}. These authors consider transformations which involves
both $\mathrm{FP}$ and $R$, while we concentrate on the denominator. The reason for
doing so can be gleaned from
\begin{equation}
\frac{\mathrm{FP}}{s(R)}=\frac{\mathrm{FP}}{R}\cdot\frac{R}{s(R)},
\end{equation}
which shows that control of the $\mathrm{SFDP}$ is equivalent to control of the FDP times a positive multiplier that depends
on $R$ and that could be greater or less than $1$ depending on the level of conservativeness that the researcher desires. Furthermore, when the scaling
function $s$ is a constant, the SFDP depends only on $\mathrm{FP}$ and the dependence on the number of rejections $R$ is suppressed.

\subsection{The Scaled Expected Value (SEV) error rate}
%In this section we present generalizations of some existing procedures to
%control the STP and the SEV. The procedures proposed are not limited to
%particular choices but for any choice of the scaling function. We show that
%some of the existing procedures can be simply modified to give a more
%general control.
%To comprehend some of the false positives error rates which are defined as
%the expected number of a certain random quantity, such as the PFER and the
%FDR, we define the SEV by

The first scaled error rate that we introduce in this paper is the Scaled Expected Value (SEV) defined by
$ \mathrm{SEV}=E\left[ \mathrm{SFDP}\right]$. When $s\left(R\right) = R$, the SEV is identical to the FDR%of \cite{BenjaminiFDR1995}
, while for %$l=k\geq 1$, and $s\left(R\right) =R$,
%it is identical to the $k$-FDR of \cite{Sarkar2007}. In addition, SEV with
$s\left( R\right) =1$, the SEV becomes $\mathrm{PFER}$. %One can also generalize some other concepts in the same way. For example, we
%define the positive SEV by%
%\[
%E\left[ \frac{V}{s\left( R\right) }\;\middle\vert\; R>0\right].
%\]
%%add reference
%
%Note that for both STP and SEV, when the scaling function is a constant, we
%find the error rates with names containing the word "family". Furthermore,
Note that when $m=m_{0}$,
$$
\mathrm{SEV}=E\left[\frac{R}{s(R)} \;\middle\vert\; R>0\right]P[R>0]=E\left[\frac{R}{s(R)} \;\middle\vert\; R>0\right]\times \mathrm{FWER}.
$$
This shows that if $s(R)\leq R$ for any $R$ in ${1,...,m}$, the control of
the SEV at level $\alpha$ implies the weak control of the $\mathrm{FWER}$
at level $\alpha$.

The linear SU procedure can be adapted in a straightforward manner to obtain a procedure that controls the SEV. %We call this procedure the scaled Benjamini and Hochberg procedure (SBH).

\begin{procedure}\label{The generalized BH procedure}%
  Let $p_{\left( 1\right)}\leq p_{\left( 2\right) }\leq \cdots \leq
  p_{\left( m\right) }$ be the ordered p-values of $m$ tests, and denote
  $H_{\left( i\right) }$ the null hypothesis that corresponds to $p_{\left(
      i\right) }$. The SU procedure with scaling function $s$ is determined by the rejection set ${\cal R} = \{ j:\ p_{(i)} \leq p_{(U)}\}$ where
$U = \max\{ i:\ p_{(i)} \leq s(i)\alpha/m\}.$
\end{procedure}

When $s\left( i\right) =i$, the procedure becomes the linear SU, whereas when $s\left( i\right)\equiv 1 $, the procedure becomes the
Bonferroni procedure.
%procedure proposed by \cite{BenjaminiFDR1995} (BH95 from now on) to control the FDR. If
Furthermore, when $s(i)=k$, we find the single step
procedure proposed by \cite{HommelHoffmann88} to control the k-FWER. In the
two later cases, the procedure is a single step procedure and there is no
need to order the p-values.

A simple way to prove general results on the control of the SEV is based on the work of \cite{Blanchard2008TwoConditions}, who provided very simple and general proofs for the control of the FDR under different assumptions. Their proofs cover the FDR control via SU procedures under independence \citep{BenjaminiFDR1995}, positive dependence and distribution free \citep{BY01}, as well as the control via weighted MCPs \citep{BH97weightedMTP, Genovese2006WFDR}. The proofs are based on three lemmas (Lemma~\ref{PartI: lemma_forindep}, \ref{PartI: lemma_alt} and
\ref{PartI: occamshammer}), and on the notion of \emph{Self Consistency}.
In the following, we adapt these proofs to give a more general control of the SEV via weighted SU MCPs.\\

First, we introduce the heuristic definition of the SEV. It is commonly to define a rejection set of a MCP by $\Rj = \{ i \in I : p_i \leq t \}$, where $t$ is a given threshold which could depend on the p-values $\pvalues$. This is the case for example in CTP, SU and SD procedures. The SEV with scaling function $s$ can be defined by
\begin{align}
\sev_s(t)= \E\left(\frac{\V}{s(R)} \ind{R>0}\right) & = \sum_{i \in I_0}  \E\left(\frac{\ind{p_i \leq t}}{s(R)}\ind{R>0}\right). \label{PartI: heuristic definition of the SEV}
\end{align}
where $s: \mathbb{R}^+ \rightarrow \mathbb{R}^+$ is a non-decreasing function (the scaling function).\\

Consider a factorized collection of thresholds $\Gamma: I \times \mathbb{R}^+ \mapsto \mathbb{R}^+$ with $$\gam=\gw \xi(s(r))\, ,$$ where $\{w_i, i \in I\}$ is a set of weights such that $\forall i \in I, \; w_i \in (0,\infty)$ and $\xi: \mathbb{R}^+ \rightarrow \mathbb{R}^+$ is a non-decreasing function, called a shape function.
\begin{definition}[Self-consistency condition]
Given a threshold collection $\Gamma$, a MCP with rejection set $\Rj$, satisfies the self consistency condition for $\Gamma$, denoted by $\mathbf{SC(\Gamma)}$, if $$ \Rj \subset \{i \in I \st p_i \leq \Gamma(i,R)\}\,.$$
\end{definition}

\subsubsection{Control of the SEV under independence}
\begin{theorem}\label{PartI: SEV control indep}
Assume that the $p$-values $\pvalues$ are independent random variables. Let $\mcp$ be an MCP such that the number of rejections $R(\pvec=(\pvalues))$ is a non increasing function in each $p_i$ with $i \in I_0$ and satisfies the $\mathbf{SC(\Gamma)}$ condition with the collection of thresholds $\gam=\frac{\alpha}{m} w_i s(r)$, where $s$ is a scaling function. Then, $$\sev_s (\mcp)\leq \frac{\alpha}{m} \sum_{i \in I_0} w_i\,.$$
\end{theorem}
In the independent case we use the following lemma stated and demonstrated by \cite{Blanchard2008TwoConditions}.
\begin{lemma}\label{PartI: lemma_forindep} Let $g:[0,1]\rightarrow
  (0,\infty)$ be a non-increasing %and left-continuous
function. Let $U$ be a random variable which has a distribution
stochastically lower bounded by a uniform
  distribution,
that is, $ \forall u \in [0,1], \; \Pro(U\leq u) \leq u$\,. Then, for any
constant $c>0$, we have
$$\E\paren{\frac{\ind{U\leq cg(U)}}{g(U)}} \leq c\,.$$
\end{lemma}
By using this lemma and the technique of \cite{Blanchard2008TwoConditions}, the proof of Theorem~\ref{PartI: SEV control indep} is relatively easy.
\begin{proof}
We denote by $\pvec_{-i}$ the set of $p$-values $\{p_j,j\neq i\}$.
From \eqref{PartI: heuristic definition of the SEV}, and using the $\mathbf{SC(\Gamma)}$, we have
\begin{align*}
\sev_s (\mcp) & = \sum_{i \in I_0}  \E\left(\frac{\ind{p_i \leq \frac{\alpha}{m} w_i s(R)}}{s(R)}\right) \\
& \leq\sum_{i\in I_0}  \E \left( \E\left(\frac{\ind{p_i\leq \frac{\alpha}{m} w_i s(R(\pvec))}}{s(R(\pvec))} \middle\vert \pvec_{-i}\right)\right)\\
& \leq  \frac{\alpha}{m} \sum_{i \in I_0} w_i\,,
\end{align*}
where we used the fact that given $\pvec_{-i}$, $p_i$ is stochastically lower bounded by a uniform distribution in the independent case. The Lemma~\ref{PartI: lemma_forindep} is applied with $U=p_i, \; g(U)=s(R(\pvec_{-i},U))$ and $c=\gw$. Note that $g(s(R(\pvec_{-i},U)))$ is non increasing in $U$ since $s$ is non decreasing and $R$ is non increasing in $U$.
\end{proof}
\subsubsection{Control of the SEV under positive dependence}
Now, we extend the last result to the positive dependence case.
\begin{theorem}\label{PartI: SEV control pos. dep.}
Let $\mcp$ be an MCP that satisfies the $\mathbf{SC(\Gamma)}$ condition with the collection of thresholds $\gam=\frac{\alpha}{m} w_i s(r)$, where $s$ is a scaling function, and such that for any $i \in I_0$, the conditional distribution of the number of rejections $R(\pvec)$ given $p_i \leq u$ is stochastically decreasing in $u$, that is,
\begin{equation}\label{PartI: Condition of lemma pos. dep.}
\text{for any }r\geq 0, \text{the function }u \mapsto \Pro (R < r|p_i \leq u) \text{ is non-decreasing.}
\end{equation}
Then, $$\sev_s(\mcp) \leq \frac{\alpha}{m} \sum_{i \in I_0} w_i\,.$$
\end{theorem}
In this case we use the following lemma stated and demonstrated by \cite{Blanchard2008TwoConditions}.
\begin{lemma}\label{PartI: lemma_alt} Let $U,V$ be two non-negative real variables. Assume the following:
\begin{enumerate}
\item The distribution of $U$ is stochastically lower bounded by a
    uniform distribution, that is, $ \forall u \in [0,1], \; \Pro(U\leq u) \leq u$\,.
\item The conditional distribution of $V$ given $U\leq u$ is
    stochastically decreasing in $u$, that is,
\begin{equation}
\forall\,  v\geq 0\,\qquad \forall\, 0\leq u \leq u'\,, \qquad \Pro(V < v\st U \leq u) \leq \Pro(V < v\st U \leq u')\,.
\end{equation}
\end{enumerate}
Then, for any constant $c>0$, we have
\[
\E\paren{\frac{\ind{U\leq cV}}{V}} \leq c\,.
\]
\end{lemma}
To prove Theorem~\ref{PartI: SEV control pos. dep.}, we use the technique of \cite{Blanchard2008TwoConditions}.
\begin{proof}
We use Theorem~\ref{PartI: SEV control indep} and we apply Lemma~\ref{PartI: lemma_alt} with $U=p_i$, $V=s(R)$ and $c=\gw \,$.
\end{proof}
\cite{Blanchard2008TwoConditions} proved that condition (\ref{PartI: Condition of lemma pos. dep.}) is satisfied if the p-values $\pvec=\{\pvalues\}$ are positively regressively dependent on each one from a subset (PRDS) \citep[see][]{BY01}. Before giving the formal definition of the PRDS, remember that a subset $D \subset [0,1]^{I}$ is said to be non-decreasing if for all $\mathbf{z},\;\mathbf{z'} \in [0,1]^{I}$ such that $z_i\leq z'_i, \forall i \in I\, ,$ we have $\mathbf{z}\in D \Rightarrow \mathbf{z'}\in D \,$.
\begin{definition}
For $I'$ a subset of $I$, the p-values $\pvec = (p_i, i \in I)$ are said to be positively regressively dependent on each one from $I'$ (PRDS on I'), if for any non-decreasing set $D \subset [0,1]^{I}$, and for any $i \in I$, the function $$u \in [0,1] \mapsto \pro (\pvec \in D \st p_i =u)$$ is non-decreasing.
\end{definition}
This leads to the following corollary.
\begin{corollary}
Suppose that the p-values $\pvec=(p_i, i \in I)$ are PRDS on $I_0$, and consider a MCP $\mcp$ with number of rejections $R$ such that $R(\pvec)$ is non-increasing in each p-value $p_i, i \in I$. If $\mcp$ satisfies the self-consistency condition $\mathbf{SC(\Gamma)}$ with $\gam= \frac{\alpha}{m}w_i s(r)$, then $$\sev_s (\mcp) \leq \alpha \sum_{i \in I_0} w_i\,.$$
\end{corollary}
\begin{proof}
The proof of consists of checking the condition (\ref{PartI: Condition of lemma pos. dep.}) of Proposition \ref{PartI: SEV control pos. dep.}. The proof is identical to the proof of the Proposition 3.6 in \cite{Blanchard2008TwoConditions}.
\end{proof}

\subsubsection{Control of the SEV under any dependence structure}
\begin{theorem}\label{PartI: SEV control general}
Let $\mcp$ be an MCP that satisfies the $\mathbf{SC(\Gamma)}$ condition with the collection of thresholds $\gam=\frac{\alpha}{m} w_i \xi(s(r))$ and a shape function $\xi$ of the form:
\begin{equation}
\mbox{ for any }r\geq 0,\; \xi(r) = \int_0 ^r u d\nu (u)\, , \label{PartI: equ_beta}
\end{equation}
where $\nu$ is a probability distribution on $(0,\infty)$. Then, $$\sev_s(\mcp) \leq \frac{\alpha}{m} \sum_{i \in I_0} w_i\,.$$
\end{theorem}
To prove this theorem, we use the following lemma stated and demonstrated by \cite{Blanchard2008TwoConditions}.
\begin{lemma}\label{PartI: occamshammer}
Let $U,V$ be two non-negative real variables and $\xi$ be a function of the
form (\ref{PartI: equ_beta}). Assume that the distribution of $U$ is stochastically
lower bounded by a uniform distribution,
that is, $ \forall u \in [0,1], \; \Pro(U\leq u) \leq u$\,. $X$ is
stochastically larger than a uniform variable in $[0,1]$\,that is $ \forall x
\in [0,1], \; \Pro(X\leq x) \leq x$\,. Then, for any constant $c>0$, we have
$$\E\paren{\frac{\ind{U\leq c\,\xi(V)}}{V}} \leq c\,.$$
\end{lemma}
To prove Theorem~\ref{PartI: SEV control general}, we use the technique of \cite{Blanchard2008TwoConditions}.
\begin{proof}
We use Theorem~\ref{PartI: SEV control indep} and we apply Lemma~\ref{PartI: occamshammer} with $U=p_i$, $V=s(R)$ and $c=\gw$.
\end{proof}
An example of shape functions is $\xi(r)=r/(1+1/2+\ldots+1/m)$, which leads to a collection of thresholds $\gam = \alpha s(r)/(m(1+1/2+\ldots+1/m))\,$. The SU procedure that uses this collection of thresholds can be seen as a scaled version of the procedure proposed by \cite{BY01}, which is met when the scaling function $s$ is the identity function. \cite{Blanchard2008TwoConditions} proposed other shape functions depending on the prior belief of the user.\\

When the shape function $\xi$ is the inverse of the scaling function $s$, that is, $\xi=s^{-1}$, the sequence of thresholds becomes $\gam=i/m \alpha.$ This shows that the LSU procedures controls the quantity $$ \E\paren{\frac{FP}{s(R \vee 1)}} =\E\paren{\frac{FP}{\xi^{-1}(R \vee 1)}}$$ at level $\alpha$ under any dependency structure of the p-values. This result is obvious for the shape function $\xi(r)=r/(1+1/2+\ldots+1/m)$, but it is not for non linear shape functions.

%Note that when $s(r)\equiv 1$, for any choice of the shape function $\beta$ with the form (\ref{PartI: equ_beta}), we find the Bonferroni procedure which controls the $\pfer=\E\paren{\V}$ under any dependency assumption on the p-values.\\
%
In their work, \cite{Blanchard2008TwoConditions} proved that any SU MCP satisfies the self consistency condition. Therefore, the results proved in this section are valid with the weighted SU procedures with scaling function $s$. In the weighted case, the p-values $p_i$ have to be replaced by $p_i/w_i$ for all $i \in I\,$.

\subsection{The scaled Tail Probability (STP) error rate}
Many error rates introduced in the literature are tail probabilities of
of control metrics, among them the FWER, the k-FWER and the FER. We
define the $\mathrm{STP_{\beta}}$ as the probability that the
$\mathrm{SFDP}$ exceeds a non-negative constant $\beta \geq 0$, $\mathrm{STP_{\beta}}=P \left[ \mathrm {SFDP}>\beta \right].$ The control metric $\mathrm{STP_{\beta}}$ with $s\left( R\right) =R$ and $\beta \in (0,1)$ is
identical to the FER, while $\mathrm{STP_{\beta}}$ with $s\left( R\right)\cdot\beta=(k-1)$
is identical to the $k$-FWER.
In particular, $\mathrm{STP_{\beta}}$ with %$s\left( R\right) >0$
and $\beta =0$ becomes $P \left[\mathrm{FP}>0\right] =\mathrm{ FWER}$.\\
The control of the $\mathrm{STP}_{\beta}$ implies the control of quantiles
of the SFDP, because $\mathrm{STP}_{\beta} \leq \alpha$ implies that the
$1-\alpha$ quantile of SFDP is smaller than $\beta$. In particular, when
$\alpha=0.5, \mathrm{STP}_{\beta}\leq \alpha$ is equivalent to
$median[\mathrm{SFDP}]\leq \beta$.\\
\cite{LehmannRomano2005} proposed a SD procedure to control the FER. Here,
we give a simply modified version of their procedure to control the STP.
\begin{procedure}\label{The generalized Lehmann and Romano procedure}
  Let $p_{\left( 1\right)}\leq p_{\left( 2\right) }\leq \cdots \leq
  p_{\left( m\right) }$ be the ordered p-values of $m$ tests, and denote by
  $H_{\left( i\right) }$ the null hypothesis that corresponds to $p_{\left(
      i\right) }$. Set
\begin{equation}
t _{i}=\left\{
\begin{array}{ll}
\frac{\left(\left\lfloor \beta s(i)\right\rfloor +1\right) }{m}\alpha, & \hbox{$\mathrm{if }$ $i \leq \left\lfloor \beta s(i)\right\rfloor +1$;} \\
\frac{\left(\left\lfloor \beta s(i)\right\rfloor +1\right) }{m+\left\lfloor \beta s(i)\right\rfloor +1-i}\alpha, & \hbox{$\mathrm{if}$ $i > \left\lfloor \beta s(i)\right\rfloor +1$.}
\end{array}
\right.
\end{equation}
The procedure is determined by the rejection set ${\cal R} = \{ j:\ p_{(i)} \leq p_{(U)}\}$ where
$U = \max\{ i:\ p_{(1)}\leq t_{1},...,p_{(i)} \leq t_i\}.$
\end{procedure}

Note that if $s\left(i\right) =i$ and $0< \beta < 1$, this SD procedure is
equal to the procedure proposed by \cite{LehmannRomano2005} for controlling the FER. Furthermore, if $s(i)$ is a constant and
$\beta s(i) \equiv k-1$, we find the critical values of the SD procedure
of \cite{LehmannRomano2005} to control the k-FWER and of course, if
$\beta=0$, we find the \cite{Holm1979} procedure that controls the FWER.

We already know by \cite{LehmannRomano2005} that for the case where $\beta
s(i)$ is constant, the procedure defined above controls the
$\mathrm{STP}_{\beta}$ at level $\alpha$, under any dependency assumption
of the p-values. The following theorem states the control in the case where
$\beta s(i)$ is not a constant.

\begin{theorem}\label{generalized Lehmann an Romano theorem}
  Denote by $q_{\left( 1\right) }\leq \cdots \leq q_{\left( m_{0}\right) }$
  the ordered p-values corresponding to the $m_{0}$ true null hypotheses.
  Set $M=\min \{ \left\lfloor \beta s(m)\right\rfloor +1,m_{0}\}$.%
  \newline (i) For the SD procedure with $t _{i}$ defined in
  procedure \ref{The generalized Lehmann and Romano procedure}, we have
\begin{equation}\label{general control of STP}
P\left[\mathrm{SFDP}>\beta \right] \leq P\left[ \bigcup_{k=\left\lfloor
\beta s(1)\right\rfloor +1}^{M}\left\{ q_{\left( k\right) }\leq \frac{
k\alpha}{m_{0}}\right\} \right].
\end{equation}
\newline (ii) Therefore, if the joint distribution of the p-values
corresponding to the null hypotheses satisfies the \cite{SIMES1986} inequality, that
is
\begin{equation}\label{simes inequality}
P\left[ \left\{ q_{\left( 1\right) }\leq \frac{\alpha }{m_{0}}%
\right\} \cup \left\{ q_{\left( 2\right) }\leq \frac{2\alpha }{m_{0}}%
\right\} \cup ...\cup \left\{ q_{\left( m_{0}\right) }\leq \frac{%
m_{0}\alpha }{m_{0}}\right\} \right]\leq\alpha,
\end{equation}
then $P\left[ \mathrm{SFDP}>\beta \right] \leq \alpha$.
\end{theorem}
The Simes inequality holds for many joint distributions of positively
dependent variables. \cite{Sarkar1998} for example, showed that the Simes inequality
holds for any \emph{multivariate positive distributions of order 2
  $\mathrm{(MTP_{2})}$}. Obviously, the condition (\ref{general control of
  STP}) is less strict than the Simes inequality condition. For the
particular case where $\lfloor \beta s(i) \rfloor +1=c$ (a constant), the
right side of (\ref{general control of STP}) holds for any dependency
distribution of the p-values. To show this, note that
\begin{equation}
P\left[ \bigcup_{k=\left\lfloor
\beta s(1)\right\rfloor +1}^{M}\left\{ q_{\left( k\right) }\leq \frac{
k\alpha}{m_{0}}\right\} \right]=P\left[ \bigcup_{k=c}^{c}\left\{ q_{\left( k\right) }\leq \frac{
k\alpha}{m_{0}}\right\} \right]=P\left[ q_{\left( c\right) }\leq \frac{
c\alpha}{m_{0}}\right] \leq \alpha.
\end{equation}

In the general case, that is, when $\lfloor \beta s(i) \rfloor +1$ is not
a constant, the following lemma stated by \cite{LehmannRomano2005}, can be
used to give a sharp upper bound for the right side of equation (\ref{general control of STP}).

\begin{lemma}\label{Generalization of LR lemma}
(Lemma 3.1 in Lehmann and Romano (2005)) Let $p_{1},...,p_{n}$ be $n$ p-values that satisfy $%
P\left\{ p_{i}\leq u\right\} \leq u$ for all $i=1,...,n$ and for
any $u\in \left( 0,1\right) $. Let $0 \leq \theta _{1}\leq \theta _{2}\leq ...\leq
\theta _{h}\leq 1$ for some $1\leq h\leq n$. Then
\[
P\left[ \bigcup\limits_{i=1}^{h}\left\{ p_{\left( i\right) }\leq
\theta _{i}\right\} \right] \leq n\sum_{i=1}^{h}\left( \theta
_{i}-\theta _{i-1}\right) \frac{1}{i}.
\]
\end{lemma}
The previous lemma leads to the the following result.
\begin{theorem}\label{Theorem of generalization}
If the critical values $(t _{i})_{1 \leq i \leq m}$ of procedure \ref{The generalized Lehmann and Romano procedure} are replaced by
\begin{equation}
t _{i}^{\prime}=%
\frac{t _{i}}{C_{\left(\left\lfloor \beta s(1)\right\rfloor +1,\left\lfloor \beta s(m)\right\rfloor\right)}}
\end{equation} with $%
C_{l,h}=\sum_{i=l}^{h}\frac{1}{i}$, then $P\left[ \mathrm{SFDP}>\beta \right] \leq
\alpha $ for any dependency of the p-values corresponding to the true null hypotheses.
\end{theorem}

The constant $C_{\left(\left\lfloor \beta s(1)\right\rfloor
    +1,\left\lfloor \beta s(m)\right\rfloor\right)}$ is usually greater
than $1$, which means that the control under any assumption is more strict
than under Simes inequality. This constant may be less than one in some
particular cases depending on the value of $\left\lfloor \beta
  s(1)\right\rfloor +1$ but this could happen only when $\beta s(1)$ is
greater than $1$ which is less frequent. In addition, if the lower index is
1, the constant $C$ is greater than 1. Depending on the scaling function
and the value of $\left\lfloor \beta s(m)\right\rfloor$, the constant $C$
could be greater or smaller than the one proposed in
\cite{LehmannRomano2005}.

%It is also worth to investigate two particular cases.
%First, let $\gamma s(\cdot)=0$ and suppose that $m_{0}\geq 1$. In this case, $C_{\left(\left\lfloor \gamma s(1)\right\rfloor +1,\left\lfloor \gamma s(m)\right\rfloor\right)}=C_{1,1}=1$ which is equivalent to say that the Holm procedure holds under any assumption on the the dependence of the p-values.
%Now, suppose that $\gamma s(\cdot)=k-1$ and suppose that $m_{0}\geq k$. In this case, $C_{\left(\left\lfloor \gamma s(1)\right\rfloor +1,\left\lfloor \gamma s(m)\right\rfloor\right)}=C_{k,k}=1/k$


\section{Simulations}\label{Scaling functions section}

%The MTP based on the last upcrossing of $t_r=r\alpha/m$ can for large $m$
%be thought of as rejecting the null hypotheses whose associated p-value is
%smaller than some bound $p_u$. For independent tests we can think of the
%p-values as drawn randomly from a mixture distribution $G(u)=(m_0/m)u +
%(m_1/m)F(u)$, where $F$ is the distribution of the p-values of the true
%alternatives, which might itself be a mixture distribution. The r-th order
%statistic $p_{(r)}$ is then approximately equal to $G^{-1}(r/m)$ and the
%intersection with the FDR line happens when $G^{-1}(R/m)=\alpha R/m=p_u$,
%where $R$ denotes the number of rejections and $p_u$ is the upper bound we
%are looking for. In order to ensure a unique intersection, conditions have
%to be put on $F$. Substituting for $G$ we find $R/m = (m_0/m)p_u +
%(m_1/m)F(p_u)$ or $F(p_u) = p_u(m/\alpha-m_0)/m_1$
%\citep[p. 508]{genovese2002operating}. Since $\alpha/m \leq p_u\leq
%\alpha$, FDR defines an MTP that lies between the Bonferroni MTP and
%testing without correction for multiplicity.
%
%The methods we propose in this paper further subdivide the range of
%possibilities. With any scaled error rate, the approximate argument above
%leads to $G^{-1}(R/m)=\alpha s(R)/m=p_u^s$, which implies $R=m_0p_u^s +
%m_1F(p_u^s)$ and finally, if $s(R)=mp_u^s/\alpha$ has a unique solution, to
%$F(p_u^s)= (s^{-1}(mp_u^s/\alpha)-m_0p_u^s)/m_1$.

The choices offered by the scaled error rates opens the question of how to
proceed in practice. While this question in its full generality goes beyond
the scope of our paper, we will investigate one aspect of it in this
section. Let $\alpha$ be fixed. We have shown that the family of thresholds
$t_i=\alpha s_\gamma(i)/m=\alpha i^\gamma/m$
provides weak control of the FWER at a common level $\alpha$. This defines
the family of MTPs we will consider. They are indexed by the parameter
$0\leq\gamma\leq 1$ and will be denoted by MTP$_\gamma$. When $\gamma=0$,
the Bonferroni procedure results, while $\gamma=1$ corresponds to the linear SU procedure.

For independent tests we can think of the p-values as a mixture of $m_0$
random draws from the uniform distribution and $m_1$ random draws from the
alternative distribution $F(u)$, which might itself be a mixture
distribution. A simple choice for $F$ is the distribution of the p-value
one obtains from a standardized Gaussian test statistic which under the
alternatives is shifted to the right by a common value $\Delta >0$. The
distribution of the p-values for one-sided tests is then $F(u) =
1-\Phi(z_{1-u}-\Delta)$ where $z_u =\Phi^{-1}(u)$. The three parameters $m_0$, $m_1$ and
$\Delta$ characterize a multiple testing problem of the kind we are going
to simulate.

Any MTP attempts to flag or reject the $m_1$ alternatives and not to flag
or not to reject the $m_0$ true hypothesis. With regard to the notation of
Table 1, the number of desired or true rejections is $\mathrm{TP}=R-\mathrm{FP}$, while the
number of undesired or false rejections is $\mathrm{FP}$. One may think of the
underlying problem in terms of costs. Each true rejection is worth one
unit, while each false rejections leads to a loss of $\lambda \geq 1$. An
MTP$_\gamma$ applied to a data set produces an expected gain of
$$\mbox{E}[\mathrm{TP}_\gamma] -\lambda \mbox{E}[\mathrm{FP}_\gamma] =
\mbox{E}[R_\gamma] - (\lambda+1)\mbox{E}[\mathrm{FP}_\gamma]\,.$$ The cost $\lambda$
of a false discovery is a tuning constant to be set by the user. It acts as
a penalty against false discoveries. If $\lambda=1$, the true and the false
discoveries are of equal value, in which case maximizing the gain $R-2\mathrm{FP}$ is
equivalent to minimizing $m_1-R+2\mathrm{FP}$, the sum of false rejections and false
discoveries. In a multiple testing situation, $\lambda$ is usually larger
than 1 and in our simulations, we consider a range from 1 to 30. In a more
refined theory, the cost $\lambda$ should probably rather be seen as a
marginal price, which increases with the number of false discoveries, but
we will stay with the simpler model of a fixed price per false
rejection. The cost $\lambda$ can also be seen as a shadow price, that is, the
value of a Lagrange multiplier at the optimum. This interpretation appears
if we optimize the number of true rejections under constraints involving
the false discoveries.

To gain further insight, consider a model case, where $m=2$ with
$m_0=m_1=1$ and we observe independent test statistics $X_0\sim{\cal
  N}(0,1)$, a unit Gaussian, and $X_1\sim {\cal N}(\Delta,1)$. We are
testing a zero mean vs. a positive mean and the two tests reject if the
observed value exceeds a critical value $\text{cv}>0$. If we reject based
on $X_0$ we have a false rejection and if we reject based on $X_1$ we have
a true rejection. In this case, $\mathrm{TP}$ and $\mathrm{FP}$ are independent Bernoulli
variables with success probabilities
$p_1=1-\Phi(\text{cv})=\Phi(-\text{cv})$ and
$p_2=1-\Phi(\text{cv}-\Delta)=\Phi(\Delta-\text{cv})$. The criterion thus
has value $\mbox{E}[\mathrm{TP}] -\lambda \mbox{E}[\mathrm{FP}] = p_2-\lambda p_1$. For a fixed
price $\lambda$, the largest value of the criterion, the optimal gain, is
achieved for the critical value that satisfies
$-\varphi(\Delta-\text{cv}_\text{opt}) + \lambda
\varphi(\Delta-\text{cv}_\text{opt})=0$, or
$\text{cv}_\text{opt}=\log(\lambda)/\Delta + \Delta/2$. The optimal gain is
always positive, increases with $\Delta$ and decreases with $\lambda$. In
this simple model, the two tests are determined by the critical value. For
a fixed price $\lambda$, the optimal critical value $\log(\lambda)/\Delta +
\Delta/2$ as a function of the effect $\Delta$ is convex and has a minimum
at $\Delta = \text{cv}_\text{opt}=\sqrt{2\log(\lambda)}$. This is the
optimal test with the minimal level. To link this with the classical
testing theory, consider the Bonferroni procedure for two one-sided tests
with overall FWER of $\alpha=0.05$. It has critical value
$\text{cv}_\text{Bonferroni}=1.96$, the 0.975 quantile of the unit
Gaussian. This corresponds to a price of $\lambda=e^{3.84/2}=6.8$. If
instead we use the 0.95 quantile, which for $m_1=1$ also controls the FWER,
we find $\lambda=3.9$. At the very least, this model suggests that the
price of a false discovery has to be substantially higher than 1. There has
to be a real penalty associated with a false discovery.

\begin{figure}[h]
\begin{center}
\includegraphics[height=10cm]{Fig1000.ps}
\caption{The three curves show the optimal value of $\gamma$ as a function
  of the penalty $\lambda$ for a false positive. The number of tests is
  $m=1000$. The goal of the tests is to determine the
  $m_1$ true alternatives (whose mean is $\Delta=3$ for the dark curves and
  $\Delta=5$ for the grey curves), while at the same time
  making as few mistaken rejections as possible. The curves were computed
  by Monte Carlo simulation with 1000 replications.}
\end{center}
\end{figure}
Now, we turn to a more realistic multiple testing problem with $m=1000$
tests. We consider $m_1=10, 50$ and $m_1=100$. The distribution of the test
statistics is the same as in the above model situation with the alternative
effect equal to $\Delta=3$. The protection level is $\alpha=0.05$. Figure 1
shows the value of $\gamma$ to be used in $t_i=\alpha i^\gamma/m$ in order
to minimize the expected loss. The optimal value of $\gamma$ decreases as
the penalty $\lambda$ for each false discovery increases. The value
$\gamma=1$ which corresponds to the linear SU procedure is only optimal for
relatively small penalties, for larger and more reasonable values it
quickly drops towards $\gamma=0.5$ if there are few true alternatives and
towards $\gamma=0.7$ otherwise. For $m=1000$, the effect $\Delta=3$ is
relatively small and hard to detect. For a larger and more easily
detectable effect, the values of $\gamma$ drop even quicker. The value
$\gamma=0.5$ is a good default choice if little is known about the number of
alternatives and the effect size. Figure 2 shows a quite similar behavior when using the truncated linear SU ($\tau$SU) procedure. Large values of $\tau$ are optimal only for small values of $\lambda$. This highlights the drawback of the FDR and shows that the $\tau$SU procedure could be useful in such situations. The parameter could be chosen directly by the user since the use of the $\tau$SU procedure guarantees the control of $E[\mathrm{FP}/\tau]$ at the same time with the FDR.
\begin{figure}[h]
\begin{center}
\includegraphics[height=10cm]{FigTrunc.ps}
\caption{The three curves show the optimal value of $\tau$ as a function
  of the penalty $\lambda$ for a false positive. The number of tests is
  $m=1000$. The goal of the tests is to determine the
  $m_1$ true alternatives (whose mean is $\Delta=3$ for the dark curves and
  $\Delta=5$ for the grey curves), while at the same time
  making as few mistaken rejections as possible. The curves were computed
  by Monte Carlo simulation with 1000 replications.}
\end{center}
\end{figure}
%
%%%%%%%%%%%%%%%%
%
%The two scaling functions we consider are members of the families mentioned
%before, that is,
%\[
%s_{1}(i)=
%\left\{
%  \begin{array}{ll}
%    i, & \hbox{$\mathrm{if }$ $i < k$ ;} \\
%    k, & \hbox{$\mathrm{if }$ $i \geq k$}
%  \end{array}
%\right.
%\]
%and
%\[
%s_{2}(i)=i^\beta.
%\]
%
%The constants of the scaling functions were chosen as $k=1/\gamma$ for
%$m=1,000$ and $k=2.5/\gamma$ for $m=10,000$. Note that as long as
%$E(V)<k\gamma$, this scaling function gives the same control as the FDR, but
%will not allow $E(V)$ to grow beyond $k\gamma$. For the second function we
%chose $\beta=0.8$.
%
%The general error rate procedures were compared to some existing standards,
%Bonferroni, BH95 and LR05.
%
%Figures 1 through 4 show the simulated average power and expected false
%positives E(V). The power of the Bonferroni procedure with $\alpha=0.05$
%can be computed analytically and is equal to
%
%\begin{equation}\label{Bonferroni power}
%\mathrm{Pow_{Bonf}} = 1-\Phi(\Phi^{-1}(1-\frac{0.05}{m})-\Delta)
%\end{equation}
%
%The construction of the figures is always the same. There are four panels,
%each plotting a performance or safety measure as a function of $\pi$, the
%fraction of false null hypotheses. The top row of the panels contains the
%scaled tail probability procedures and uses the LR05 procedure as a
%standard. These are step-down tests. The bottom row shows the scaled
%expected value procedures and uses the BH95 for comparison. These are
%step-up tests. The left-hand column shows the average power (the
%performance indicator) and the right-hand column depicts E(V) (the safety
%indicator).
%\begin{figure}
%  % Requires \usepackage{graphicx}
%\centering
%\makebox{  \includegraphics[angle=270,width=12.5cm]{STP-1000.ps}}%\\
%  \caption{Average power and $E(V)$ for different procedures. Bonferroni (horizontal line), LR05 (---), STP with $s_{1}$ ($-\cdot-$) and with $s_{2}$ (- - -). $m=10^3,\;\Delta=2$ or $4,\;\alpha=0.05$, $\gamma=0.05$.}\label{STP-1000}
%\end{figure}
%
%\begin{figure}
%  % Requires \usepackage{graphicx}
%\centering
%\makebox{  \includegraphics[angle=270,width=12.5cm]{STP-10000.ps}}%\\
%  \caption{Average power and $E(V)$ for different procedures. Bonferroni (horizontal line), LR05 (---), STP with $s_{1}$ ($-\cdot-$) and with $s_{2}$ (- - -). $m=10^4,\;\Delta=2$ or $4,\;\alpha=0.05$, $\gamma=0.05$.}\label{STP-10000}
%\end{figure}
%
\section{Conclusion}

We introduced a new indicator for the control of false positives, the
scaled false discovery proportion SFDP, and we proposed two control
metrics, the SEV and the STP. We showed that in this framework, one can
embed the classical metrics in a family of multiple testing procedures, for
example, FWER and FER or PFER and FDR. The freedom offered by the scaling
function generalize the existent error rates and offer the user a finer
control between tests whose aim is a screening of the hypotheses and tests
whose aim is the detection of true alternatives.

We also proposed the corresponding multiple testing procedures to control
either the SEV or the STP under varying assumptions. Other classes of
procedures could be generalized in the same way as presented in this paper.
Two families of scaling functions were proposed and in a simulation study
we showed that an intermediate choice is usually preferable. The framework
in which we studied this choice used a new point of view, different from
the classical view of level and power. The classical approach needs to be rethought and adapted to the multiple testing context with large numbers of hypotheses.
\bibliographystyle{apalike}
\bibliography{D:/Thesis/Bib}
\appendix
\section{Proofs}


\subsection{Proof of the Theorem \ref{generalized Lehmann an Romano theorem}}

The proof is based on the method of \cite{LehmannRomano2005}
The event $\left\{ \mathrm{SFDP}>\beta
\right\} $ occurs only if for at least one random index $i$, the quantity $\mathrm{SFDP}$
exceeds $\beta $. Among these indexes, denote the smallest one by $j$. Then $P\left[
\mathrm{SFDP}>\beta \right] \leq P\left[ \mathrm {such}\;j \;\mathrm{ exists}\right] $. The
range of the possible values of $\beta s\left( j\right) $ ($0\leq \beta
<1, $ $1\leq j\leq m$) is divided into $\left\lfloor \beta
s(1)\right\rfloor \leq \beta s\left( j\right) <\left\lfloor \beta
s(1)\right\rfloor +1,~\left\lfloor \beta s(1)\right\rfloor +1\leq \beta
s\left( j\right) <\left\lfloor \beta s(1)\right\rfloor +2,...,~\left\lfloor
\beta s(m)\right\rfloor \leq \beta s\left( j\right) <\left\lfloor \beta
s(m)\right\rfloor +1$. \\
Because of the definition of $j$, we must have $%
p_{\left( j\right) }\leq \alpha _{j}$, $H_{\left( j\right) }$ is true and $%
\left\lfloor \beta s(j)\right\rfloor +1\leq m_{0}$. \\
Therefore, \\$ P\left[\mathrm{SFDP}>\beta \right] \leq P \left[\left\{ \left\lfloor \beta
s(1)\right\rfloor \leq \beta s\left( j\right) <\left\lfloor \beta
s(1)\right\rfloor +1\right\}
\cup \{ \left\lfloor \beta s(1)\right\rfloor
+ 1\leq \beta s\left( j\right) <\left\lfloor \beta
s(1)\right\rfloor +2\}\right.$  $\left.\cup ...\cup \left\{ M-1\leq \beta s\left(
j\right) <M\right\}\right],$ with $M=min\left\{\beta s(m),m_{0}\right\}$.\\
Let $k-1\leq \beta s\left( j\right) <k$ for $k$ in $\left\{ \left\lfloor
\beta s(1)\right\rfloor +1,...,M\right\} $. Then $p_{\left( j\right)
}=q_{\left( k\right) }\leq \alpha _{j}$ because $%=\frac{k\alpha }{m+k-j}
\frac{k-1}{s\left( j\right) }\leq \beta $ and $\frac{k}{s\left( j\right) }%
>\beta $. This implies that $H_{\left( j\right)}$
is the $k$th rejected true hypothesis, and $k\leq j\leq m-(m_{0}-k)$ which
implies that $m_{0}\leq m+k-j$. Therefore, if $k-1\leq \beta s\left(
j\right) <k$, the event $\{\mathrm{SFDP}>\beta \}$ at step $j$ implies that $q_{\left( k\right)
}\leq \frac{k\alpha }{m_{0}}$. So,
\begin{eqnarray*}
P\left[\mathrm{SFDP}>\beta \right] &\leq &\sum_{k=\left\lfloor \beta
s(1)\right\rfloor +1}^{M}P\left[ q_{\left( k\right) }\leq \frac{k\alpha }{%
m_{0}},k-1\leq \beta s\left( j\right) <k\right] \\
&\leq &\sum_{k=\left\lfloor \beta s(1)\right\rfloor +1}^{M} P\left[
\bigcup\limits_{k=\left\lfloor \beta s(1)\right\rfloor +1}^{M}\left\{ q
_{\left( k\right) }\leq \frac{k\alpha }{m_{0}}\right\} ,k-1\leq \beta
s\left( j\right) <k\right] \\
&\leq &P\left[ \bigcup\limits_{k=\left\lfloor \beta s(1)\right\rfloor
+1}^{M}\left\{ q_{\left( k\right) }\leq \frac{k\alpha }{m_{0}}\right\}
\right].
\end{eqnarray*}%
Part (ii) follows trivially.
\subsection{Proof of the Theorem \ref{Theorem of generalization}}
By replacing in lemma \ref{Generalization of LR lemma}, $h$ and $n$ by
  $M$ and $m_{0}$ respectively and by setting $\theta_{i}=0$ for $i=1,...,
  \left\lfloor \beta s(1)\right\rfloor$ and $\theta_{i}=\frac {i
    \alpha}{m_{0}}$ for $i=\left\lfloor \beta s(1)\right\rfloor+1,...,M$,
  we obtain
\begin{eqnarray*}
% \nonumber to remove numbering (before each equation)
   P\left[ \mathrm{SFDP}>\beta \right] &\leq& P\left[ \bigcup\limits_{i=\left\lfloor
\beta s(1)\right\rfloor +1}^{M}\left\{ p_{\left( i\right) }\leq
\frac{i \alpha}{m_{0}}\right\} \right] \\ &\leq& m_{0} \sum_{i=\left\lfloor
\beta s(1)\right\rfloor +1}^{M}\left( \frac{\alpha}{m_{0}}\right) \frac{1}{i} \\
   &=& \alpha \sum_{i=\left\lfloor
\beta s(1)\right\rfloor +1}^{M} \frac{1}{i}.
\end{eqnarray*}
It suffices then to replace $\alpha$ by $\frac{\alpha}{C_{\left(\left\lfloor \beta s(1)\right\rfloor +1,\left\lfloor \beta s(M)\right\rfloor\right)}}$ to have $P\left[ \mathrm{SFDP}>\beta \right]$ bounded by $\alpha$.
\end{document}

Manuscript.bbl
\begin{thebibliography}{}

\bibitem[Benjamini, 2010]{BenjaminiSI2010}
Benjamini, Y. ({2010}).
\newblock {Simultaneous and selective inference: Current successes and future
  challenges}.
\newblock {\em {Biometrical Journal}}, {52}({6, SI}):{708--721}.

\bibitem[Benjamini and Hochberg, 1995]{BenjaminiFDR1995}
Benjamini, Y. and Hochberg, Y. (1995).
\newblock {Controlling the false discovery rate: a practical and powerful
  approach to multiple testing}.
\newblock {\em J. Roy. Statist. Soc. Ser. B}, 57(1):289--300.

\bibitem[Benjamini and Hochberg, 1997]{BH97weightedMTP}
Benjamini, Y. and Hochberg, Y. (1997).
\newblock Multiple hypotheses testing with weights.
\newblock {\em Scandinavian Journal of Statistics}, 24(3):407--418.

\bibitem[Benjamini and Yekutieli, 2001]{BY01}
Benjamini, Y. and Yekutieli, D. (2001).
\newblock {The control of the false discovery rate in multiple testing under
  dependency}.
\newblock {\em Ann. Statist.}, 29(4):1165--1188.

\bibitem[Blanchard and Roquain, 2008]{Blanchard2008TwoConditions}
Blanchard, G. and Roquain, E. (2008).
\newblock Two simple sufficient conditions for {FDR} control.
\newblock {\em Electron. J. Stat.}, 2:963--992.

\bibitem[Bonferroni, 1936]{bonferroni1936}
Bonferroni, C.~E. (1936).
\newblock {Teoria statistica delle classi e calcolo delle probabilit{\`a}}.
\newblock {\em {P}ub. del {R} {I}st. {S}up. di {S}ci. {E}co. e {C}om. di
  {F}ir.}, 8:3--62.
\newblock Bonferroni adjustment for multiple statistical tests using the same
  data.

\bibitem[Dudoit and van~der Laan, 2008]{DudoitLaan2007}
Dudoit, S. and van~der Laan, M.~J. (2008).
\newblock {\em {Multiple testing procedures with applications to genomics}}.
\newblock {Springer Series in Statistics}. Springer, New York.

\bibitem[Finner et~al., 2009]{Finner2009false}
Finner, H., Dickhaus, T., and Roters, M. (2009).
\newblock {On the false discovery rate and an asymptotically optimal rejection
  curve}.
\newblock {\em The Annals of Statistics}, 37:596--618.

\bibitem[Genovese and Wasserman, 2002]{genovese2002operating}
Genovese, C. and Wasserman, L. (2002).
\newblock {Operating characteristics and extensions of the false discovery rate
  procedure}.
\newblock {\em Journal of the Royal Statistical Society: Series B (Statistical
  Methodology)}, 64:499--517.

\bibitem[Genovese et~al., 2006]{Genovese2006WFDR}
Genovese, C.~R., Roeder, K., and Wasserman, L. (2006).
\newblock False discovery control with p-value weighting.
\newblock {\em Biometrika}, 93(3):509--524.

\bibitem[Holm, 1979]{Holm1979}
Holm, S. (1979).
\newblock {A simple sequentially rejective multiple test procedure}.
\newblock {\em Scand. J. Statist.}, 6(2):65--70.

\bibitem[Hommel and Hoffmann, 1987]{HommelHoffmann88}
Hommel, G. and Hoffmann, T. (1987).
\newblock Controlled uncertainty.
\newblock In Bauer, P., Hommel, G., and Sonnemann, E., editors, {\em Multiple
  Hypotheses Testing}, pages 154--161. Springer, Heidelberg.

\bibitem[Horn and Dunnett, 2004]{HornDunnett2004}
Horn, M. and Dunnett, C.~W. (2004).
\newblock {Power and Sample Size Comparisons of Stepwise FWE and FDR
  Controlling Test Procedures in the Normal Many-One Case}.
\newblock {\em Lecture Notes-Monograph Series}, 47:48--64.

\bibitem[Lehmann and Romano, 2005]{LehmannRomano2005}
Lehmann, E.~L. and Romano, J.~P. (2005).
\newblock {Generalizations of the familywise error rate}.
\newblock {\em Ann. Statist.}, 33(3):1138--1154.

\bibitem[Roquain and Villers, 2011]{Roquain2011exact}
Roquain, E. and Villers, F. (2011).
\newblock Exact calculations for false discovery proportion with application to
  least favorable configurations.
\newblock {\em The Annals of Statistics}, 39:584--612.

\bibitem[Sarkar, 1998]{Sarkar1998}
Sarkar, S.~K. (1998).
\newblock {Some Probability Inequalities for Ordered MTP2 Random Variables: A
  Proof of the Simes Conjecture}.
\newblock {\em The Annals of Statistics}, 26(2):pp. 494--504.

\bibitem[Simes, 1986]{SIMES1986}
Simes, R.~J. (1986).
\newblock {An improved {B}onferroni procedure for multiple tests of
  significance}.
\newblock {\em Biometrika}, 73(3):751--754.

\bibitem[van~der Laan et~al., 2004]{VDL2004}
van~der Laan, M.~J., Dudoit, S., and Pollard, K.~S. (2004).
\newblock {Augmentation procedures for control of the generalized family-wise
  error rate and tail probabilities for the proportion of false positives}.
\newblock {\em Stat. Appl. Genet. Mol. Biol.}, 3:Art. 15, 27 pp. (electronic).

\bibitem[Victor, 1982]{Victor82}
Victor, N. (1982).
\newblock {Exploratory data analysis and clinical research}.
\newblock {\em Methods of Information in Medicine}, 21(2):53--54.

\end{thebibliography}
