Treffer: Measuring agreement among several raters classifying subjects into one or more (hierarchical) categories: A generalization of Fleiss' kappa.

Cohen's and Fleiss' kappa are well-known measures of inter-rater agreement, but they restrict each rater to selecting only one category per subject. This limitation is consequential in contexts where subjects may belong to multiple categories, such as psychiatric diagnoses involving multiple disorders or classifying interview snippets into multiple codes of a codebook. We propose a generalized version of Fleiss' kappa, which accommodates multiple raters assigning subjects to one or more nominal categories. Our proposed INLINEMATH statistic can incorporate category weights based on their importance and account for hierarchical category structures, such as primary disorders with sub-disorders. The new INLINEMATH statistic can also manage missing data and variations in the number of raters per subject or category. We review existing methods that allow for multiple category assignments and detail the derivation of our measure, proving its equivalence to Fleiss' kappa when raters select a single category per subject. The paper discusses the assumptions, premises, and potential paradoxes of the new measure, as well as the range of possible values and guidelines for interpretation. The measure was developed to investigate the reliability of a new mathematics assessment method, of which an example is elaborated. The paper concludes with a worked-out example of psychiatrists diagnosing patients with multiple disorders. All calculations are provided as R script and an Excel sheet to facilitate access to the new INLINEMATH statistic.
(© 2025. The Author(s).)

Declarations. Ethical Approval: Not applicable. Conflicts of Interest: We have no conflicts of interest to disclose. Consent to Participate: Not applicable. Consent for Publication: Not applicable. Open practice statement: All materials and analysis code are available as Supplementary material. We provide an Excel spreadsheet containing all examples discussed in the paper, as well as an R script with the required data files. A detailed proof of Theorem 1 is also included. For the most recent version of the materials, additional code (e.g., for additional statistical programs), and updates, please refer to the OSF project: https://osf.io/q5nft/ . None of the reported studies were preregistered. An earlier version of this manuscript has been published as an arXiv preprint prior to formal peer review and publication: https://doi.org/10.48550/arXiv.2303.12502 .