  Cohen's and Fleiss' kappa are well-known measures for inter-rater
reliability. However, they only allow a rater to select exactly one category
for each subject. This is a severe limitation in some research contexts: for
example, measuring the inter-rater reliability of a group of psychiatrists
diagnosing patients into multiple disorders is impossible with these measures.
This paper proposes a generalisation of the Fleiss' kappa coefficient that
lifts this limitation. Specifically, the proposed $\kappa$ statistic measures
inter-rater reliability between multiple raters classifying subjects into
one-or-more nominal categories. These categories can be weighted according to
their importance, and the measure can take into account the category hierarchy
(e.g., categories consisting of subcategories that are only available when
choosing the main category like a primary psychiatric disorder and
sub-disorders; but much more complex dependencies between categories are
possible as well). The proposed $\kappa$ statistic can handle missing data and
a varying number of raters for subjects or categories. The paper briefly
overviews existing methods allowing raters to classify subjects into multiple
categories. Next, we derive our proposed measure step-by-step and prove that
the proposed measure equals Fleiss' kappa when a fixed number of raters chose
one category for each subject. The measure was developed to investigate the
reliability of a new mathematics assessment method, of which an example is
elaborated. The paper concludes with the worked-out example of psychiatrists
diagnosing patients into multiple disorders.
