Editverse Free Calculator
Cohen’s Kappa Calculator (Inter-Rater Agreement)
Measure how well two raters agree on a categorical judgement beyond chance — Cohen’s kappa with observed and expected agreement — and copy the exact R command behind the result. A full interpretation guide follows below. Validated against R 4.3.3.
In Simple Words
Cohen’s kappa measures agreement between two raters classifying items into categories, correcting for the agreement you would expect by chance alone. Raw percent agreement can look high simply because one category is common, so kappa subtracts that chance baseline to give a fairer picture.
Kappa ranges from below 0 (worse than chance) up to 1 (perfect agreement). It is computed from observed agreement (how often the raters actually matched) and expected agreement (how often they would match by chance given their marginal rates). The further observed sits above expected, the higher the kappa.
When to Use Cohen’s Kappa
Use Cohen’s kappa when two raters independently assign items to the same set of categories — for example, two reviewers screening abstracts, or two clinicians coding a diagnosis. It tells you whether their agreement reflects a shared, reliable judgement or merely chance overlap.
For continuous ratings rather than categories, use the intraclass correlation coefficient instead. Report kappa with its interpretation band and be aware of the prevalence effect: when one category dominates, kappa can be low even when raters agree most of the time. Not sure which agreement measure fits your data? Our Test Selector can help.
Expected format: Agreement matrix: rows on new lines, comma-separated
Drop a CSV here or click to choose. Parsed in your browser: nothing is uploaded.
Expected format: Agreement matrix: rows on new lines, comma-separated
library(irr)
tbl <- matrix(c({mat_flat}), nrow = {nrows}, byrow = TRUE)
r1 <- rep(row(tbl), tbl); r2 <- rep(col(tbl), tbl) # one pair of ratings per item
kappa2(cbind(r1, r2))User guide
When to use. Measure categorical inter-rater agreement beyond what is expected by chance.
Assumptions. Two raters; same categories; independent ratings.
Common mistakes. Kappa is sensitive to prevalence: high agreement can yield low kappa.
Worked example
Click Load example data above, then Run Analysis to see a fully worked result with figure, formula and interpretation for this test.
References
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37-46.
Worked Examples
Example 1 — Two reviewers screening abstracts
Two reviewers each label 50 abstracts as include or exclude. They agree on 42, with observed agreement of 0.84. Given how often each said “include,” chance agreement is about 0.55, so kappa = (0.84 − 0.55) / (1 − 0.55) ≈ 0.64 — substantial agreement, suitable for proceeding.
Example 2 — The prevalence paradox
If 95% of items truly belong to one category, two raters might agree 92% of the time yet score a kappa near 0.20, because chance agreement is already very high. Here the low kappa reflects the skewed marginals, not poor raters — report percent agreement alongside kappa to make this clear.
Assumptions to Check
Two raters, same categories. Standard Cohen’s kappa is for exactly two raters using an identical, mutually exclusive set of categories. For more raters use Fleiss’ kappa instead.
Independent ratings. The raters must judge each item without conferring. If they discuss cases, their agreement is inflated and kappa no longer measures independent reliability.
Nominal categories of equal seriousness. Plain kappa treats every disagreement as equally bad. When some disagreements matter more (ordered categories), use weighted kappa.
Common Mistakes
Reading kappa without considering prevalence
A skewed category distribution drives chance agreement up and kappa down. Report percent agreement alongside kappa so a low value is not misread as poor rating.
Using plain kappa for ordered categories
When categories have a natural order (mild/moderate/severe), unweighted kappa ignores how far apart disagreements are. Use weighted kappa instead.
Applying Cohen’s kappa to more than two raters
Cohen’s kappa is defined for two raters. For three or more, Fleiss’ kappa is the correct generalisation.
Treating the interpretation bands as absolute
The slight/fair/moderate/substantial labels are conventions, not laws. What counts as adequate agreement depends on the consequences of misclassification in your setting.
Frequently Asked Questions
What is a good kappa value?
By common convention, 0.21–0.40 is fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and above 0.80 almost perfect — but the adequate level depends on your context.
Why is my kappa low when agreement looks high?
When one category is very common, chance agreement is already high, so kappa subtracts a large baseline and can be low even at high percent agreement.
When should I use weighted kappa?
Use it when categories are ordered and some disagreements are more serious than others, so that near-misses are penalised less than far-off ones.
Does the result match R and SPSS?
Yes. The calculator uses the standard observed-minus-expected formula and shows the R command (irr::kappa2) so you can reproduce the value.
Related Calculators
References
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. PMID 843571.
- McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–282. PMID 23092060.
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37–46.
Need This Analysis Done for Your Manuscript?
Our PhD biostatisticians compute inter-rater reliability — Cohen’s or weighted kappa with confidence intervals — and deliver publication-ready agreement tables with reproducible R output.