Volume 14, Issue 3 (Summer- In Press 2026)
Iran J Health Sci 2026, 14(3): 0-0 |
Back to browse issues page
Ethics code: N/A
Clinical trials code: N/A
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Shabankhani B, Bakhshandeh M, Rezvani-Ahangarkolaei M, Babaei Z, Shabankhani K. An Exploratory, Data-Informed Approach to Characterizing Weighting Choices for Cohen’s Kappa in Ordinal Data. Iran J Health Sci 2026; 14 (3)
URL: http://jhs.mazums.ac.ir/article-1-1159-en.html
URL: http://jhs.mazums.ac.ir/article-1-1159-en.html
Bizhan Shabankhani 
, Mahsa Bakhshandeh , Mohammadmehdi Rezvani-Ahangarkolaei , Zeinab Babaei , Keihan Shabankhani *
, Mahsa Bakhshandeh , Mohammadmehdi Rezvani-Ahangarkolaei , Zeinab Babaei , Keihan Shabankhani *
Department of Anesthesiology, School of Medicine, Mazandaran University of Medical Sciences, Sari, Iran. , kshabankhani@gmail.com
Abstract: (2 Views)
Background and Purpose: Selecting an appropriate weighting scheme for Cohen’s kappa in ordinal data depends on the structure and practical meaning of disagreements between categories. Although linear and quadratic weighting schemes are widely used, their selection is often guided by substantive considerations rather than a standardized statistical rule. This methodological short communication described an exploratory data-informed approach for characterizing ordinal association and category-level heterogeneity using Spearman’s rank correlation and one-vs-rest receiver operating characteristic (ROC) analysis. The approach is intended to provide supplementary information for interpreting disagreement patterns rather than to establish a validated rule for selecting a specific kappa weighting scheme.
Materials and Methods: We described a two-component exploratory approach. First, Spearman’s rank correlation coefficient was used as a descriptive measure of ordinal association between two raters and was interpreted separately from agreement statistics. Second, one-vs-rest ROC analysis was used to describe variation in category-level discrimination across ordinal categories. The difference between the maximum and minimum AUC values was summarized using a descriptive heterogeneity index (D = AUCmax − AUCmin). The approach was illustrated using a simulated ordinal dataset. The index was used only to characterize the observed heterogeneity and was not assigned a validated cutoff or used as a definitive decision rule for selecting linear or quadratic weighting.
Results: In the illustrative example, the two raters demonstrated a moderately strong positive ordinal association (Spearman’s ρ = 0.737). One-vs-rest ROC analyses showed variation in AUC values across categories and between raters. The highest AUC was 0.833 and the lowest was 0.528, resulting in a descriptive heterogeneity index of D = 0.305. This variation illustrates non-uniform category-level discrimination across the ordinal scale. The observed pattern may provide supplementary information when considering the characteristics of disagreement and the interpretation of alternative weighting schemes, but it does not establish that quadratic weighting is universally preferable.
Conclusion: This methodological short communication presented an exploratory approach for describing ordinal association and category-level heterogeneity when considering weighting choices for Cohen’s kappa. Spearman’s correlation and ROC/AUC analysis provide complementary descriptive information but do not directly measure inter-rater agreement or constitute a validated rule for selecting linear or quadratic weighting. The proposed D index is intended only as a descriptive summary of AUC heterogeneity. Larger simulation studies and independent datasets are required to determine whether these measures provide useful information for weighting decisions in different ordinal settings.
Materials and Methods: We described a two-component exploratory approach. First, Spearman’s rank correlation coefficient was used as a descriptive measure of ordinal association between two raters and was interpreted separately from agreement statistics. Second, one-vs-rest ROC analysis was used to describe variation in category-level discrimination across ordinal categories. The difference between the maximum and minimum AUC values was summarized using a descriptive heterogeneity index (D = AUCmax − AUCmin). The approach was illustrated using a simulated ordinal dataset. The index was used only to characterize the observed heterogeneity and was not assigned a validated cutoff or used as a definitive decision rule for selecting linear or quadratic weighting.
Results: In the illustrative example, the two raters demonstrated a moderately strong positive ordinal association (Spearman’s ρ = 0.737). One-vs-rest ROC analyses showed variation in AUC values across categories and between raters. The highest AUC was 0.833 and the lowest was 0.528, resulting in a descriptive heterogeneity index of D = 0.305. This variation illustrates non-uniform category-level discrimination across the ordinal scale. The observed pattern may provide supplementary information when considering the characteristics of disagreement and the interpretation of alternative weighting schemes, but it does not establish that quadratic weighting is universally preferable.
Conclusion: This methodological short communication presented an exploratory approach for describing ordinal association and category-level heterogeneity when considering weighting choices for Cohen’s kappa. Spearman’s correlation and ROC/AUC analysis provide complementary descriptive information but do not directly measure inter-rater agreement or constitute a validated rule for selecting linear or quadratic weighting. The proposed D index is intended only as a descriptive summary of AUC heterogeneity. Larger simulation studies and independent datasets are required to determine whether these measures provide useful information for weighting decisions in different ordinal settings.
Keywords: Cohen’s Kappa, Weighted Kappa, Spearman’s Rank Correlation, ROC Curve, Area Under the Curve, Ordinal Data, Inter-Rater Agreement
Type of Study: Original Article |
Subject:
Biostatistics
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
This is an open access article distributed under the terms of the Creative Commons Attribution License (CC-By-NC), which permits use, distribution, and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.
Contact Information
