  In the medical image analysis field, organizing challenges with associated
workshops at international conferences began in 2007 and has grown to include
over 150 challenges. Several of these challenges have had a major impact in the
field. However, whereas well-designed challenges have the potential to unite
and focus the field on creating solutions to important problems, poorly
designed and documented challenges can equally impede a field and lead to
pursuing incremental improvements in metric scores with no theoretic or
clinical significance. This is supported by a critical assessment of challenges
at the international MICCAI conference. In this assessment the main observation
was that small changes to the underlying challenge data can drastically change
the ranking order on the leaderboard. Related to this is the practice of
leaderboard climbing, which is characterized by participants focusing on
incrementally improving metric results rather than advancing science or solving
the driving problem of a challenge. In this abstract we look beyond the
leaderboard of a challenge and instead look at the conclusions that can be
drawn from a challenge with respect to the research problem that it is
addressing. Research study design is well described in other research areas and
can be translated to challenge design when viewing challenges as research
studies on algorithm performance that address a research problem. Based on the
two main types of scientific research study design, we propose two main
challenge types, which we think would benefit other research areas as well: 1)
an insight challenge that is based on a qualitative study design and 2) a
deployment challenge that is based on a quantitative study design. In addition
we briefly touch upon related considerations with respect to statistical
significance versus practical significance, generalizability and data
saturation.
