Editverse Free Calculator
ICC Calculator (Intraclass Correlation Coefficient)
Quantify the reliability of continuous measurements across raters or repeated measures with the one-way random-effects, single-measure intraclass correlation coefficient, ICC(1,1) in Shrout and Fleiss notation, and an R command that reproduces it. A full interpretation guide follows below. Validated against R 4.3.3.
In Simple Words
The intraclass correlation coefficient (ICC) measures how much of the variation in continuous measurements is due to real differences between subjects rather than to inconsistency between raters or occasions. A high ICC means raters or repeated measures agree closely; a low ICC means much of the variation is measurement noise.
Unlike a simple correlation between two raters, the ICC handles any number of raters and treats them within one model. Crucially there are several forms — differing in whether raters are fixed or random and whether you care about absolute agreement or just consistency — and they are not interchangeable, so the form must be reported.
When to Use the ICC
Use the ICC for continuous data when you want reliability across raters (inter-rater), across repeated measurements (test-retest), or within a single rater (intra-rater). It is the continuous-data counterpart to Cohen’s kappa, which is for categories.
Always state which ICC form you used — the choice between one-way and two-way models, random versus fixed raters, and absolute agreement versus consistency, changes the value and its meaning. Report the ICC with its confidence interval, since reliability estimates from small samples are imprecise. Unsure which reliability measure fits your design? Our Test Selector can help.
Expected format: Raters (one per line): RaterName: v1, v2 …
Drop a CSV here or click to choose. Parsed in your browser: nothing is uploaded.
Expected format: Raters (one per line): RaterName: v1, v2 …
library(irr)
ratings <- {icc_ratings} # rows = subjects, columns = raters
icc(ratings, model = "oneway", unit = "single")User guide
When to use. Quantify reliability of continuous measurements across multiple raters.
Assumptions. Continuous ratings; consistent rater set across subjects.
Common mistakes. Specify the ICC form (1,1 / 2,1 / 3,1): they are not interchangeable.
Worked example
Click Load example data above, then Run Analysis to see a fully worked result with figure, formula and interpretation for this test.
References
- Shrout PE, Fleiss JL. Intraclass correlations. Psychol Bull. 1979;86:420-428.
Worked Examples
Example 1 — Two raters scoring the same subjects
Two raters score five subjects on a continuous scale with very similar values (the built-in example). The calculator returns ICC(1,1) = 0.93, excellent reliability, meaning almost all the variation reflects true subject differences rather than rater disagreement. The two-way absolute-agreement ICC(2,1) on the same data is also 0.93.
Example 2 — Why the form matters
On the same data, a consistency ICC (which ignores systematic differences between raters) may read higher than an absolute-agreement ICC (which penalises a rater who scores uniformly higher). Reporting the value without the form leaves readers unable to know which question you answered.
Assumptions to Check
Continuous measurements. The ICC is for interval or ratio data. For categorical agreement use Cohen’s kappa instead.
A consistent rating structure. The same raters (or the same number of raters) should assess every subject, matching the model you choose — one-way, two-way random, or two-way mixed.
The chosen form matches the question. Decide in advance whether you need absolute agreement or consistency, and whether raters are a random sample or the only raters of interest. The wrong form gives a technically valid but misleading number.
Common Mistakes
Reporting an ICC without its form
There are ten ICC variants. A bare “ICC = 0.85” is uninterpretable; always state the model, type, and definition (for example, two-way random, absolute agreement).
Confusing consistency with absolute agreement
Consistency ignores systematic rater differences; absolute agreement does not. Picking the wrong one can hide a rater who is reliably biased high or low.
Ignoring the confidence interval
With few subjects or raters, the ICC point estimate is unstable. Report the confidence interval so readers see how precise the reliability estimate really is.
Using a single correlation for more than two raters
A Pearson correlation handles only two raters and ignores systematic differences. The ICC is the correct measure for multiple raters or repeated measures.
Frequently Asked Questions
What is a good ICC value?
A common guide rates below 0.50 as poor, 0.50–0.75 moderate, 0.75–0.90 good, and above 0.90 excellent — but the adequate level depends on how the measurement will be used.
How is the ICC different from Pearson’s r?
Pearson’s r measures only two raters and ignores systematic offsets between them; the ICC handles any number of raters and can penalise systematic disagreement.
Which ICC form should I choose?
Decide whether raters are a random sample or fixed, and whether you need absolute agreement or consistency; those two choices determine the form, which you must report.
Does the result match R and SPSS?
Yes, for the one-way model. The value equals irr::icc(ratings, model = "oneway", unit = "single") in R and the one-way random, single-measures ICC in SPSS. Two-way and average-measures forms, and the confidence interval, are not computed here; with several raters and systematic rater differences the two-way forms can differ markedly, so compute them in R if your design needs them.
Related Calculators
References
- Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163. PMID 27330520.
- Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–428.
- McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1(1):30–46.
Need This Analysis Done for Your Manuscript?
Our PhD biostatisticians compute the correct ICC form with confidence intervals and deliver publication-ready reliability tables with reproducible R output.