  In this paper we aim to refine the concept of grand challenges in medical
image analysis, based on statistical principles from quantitative and
qualitative experimental research. We identify two types of challenges based on
their generalization objective: 1) a deployment challenge and 2) an insight
challenge. A deployment challenge's generalization objective is to find
algorithms that solve a medical image analysis problem, which thereby requires
the use of a quantitative experimental design. An insight challenge's
generalization objective is to gain a broad understanding of what class of
algorithms might be effective for a class of medical image analysis problems,
in which case a qualitative experimental design is sufficient. Both challenge
types are valuable, but problems arise when a challenge's design and objective
are inconsistent, as is often the case when a challenge does not carefully
consider these concepts. Therefore, in this paper, we propose a theoretical
framework, based on statistical principles, to guide researchers in challenge
design, documentation, and assessment. Experimental results are given that
explore the factors that effect the practical implementation of this
theoretical framework.
