Published | Last updated
What Is the Reproducibility of Diagnostic Tests?
A diagnostic result is only useful if it does not depend on where the sample was measured. Reproducibility asks whether two laboratories, instruments or operators return the same result for the same specimen. It differs from repeatability, which asks the same question of one laboratory measuring the sample again.6 Studies of this kind are called reliability and agreement studies: they are intended to show the amount of error inherent in any diagnosis, score or measurement, and the authors of the GRRAS guideline note that reporting of them is often incomplete.1
This post does not cover how well a test separates sick from healthy people. That is diagnostic accuracy, reported with STARD.7 Here the question is narrower: would another laboratory get the same answer?
Why Laboratories Disagree on the Same Sample
Variation in laboratory results can arise at any stage of testing, and a review in the Journal of the Nepal Medical Association groups what remains after laboratory errors are removed into three kinds.6
Swipe sideways to see every column.
| Source of variation | What the source says | Source |
|---|---|---|
| Pre-analytical | Patient preparation is the commonest factor: diet, physical activity and timing of sampling can all change results, and some of this is outside the control of the laboratory. | 6 |
| Biological | Physiological fluctuation of an analyte around the person’s own set point, with cyclical and random components. | 6 |
| Analytical | Differences in methods and equipment, plus the inherent random imprecision of every analytical technique, even for one sample on one instrument. | 6 |
That review also gives a way to judge whether two results differ by more than expected: the reference change value, RCV = 21/2 × Z × (CVA2 + CVI2)1/2, where CVA is analytical and CVI within-person biological variation.6 Its own glucose example uses CVA 2.7% and CVI 5.6% with Z = 1.96. Editverse recomputed that example in R and obtained 17.2%, so a change smaller than about 17% is within expected variation.
Two further causes show up in the case studies below. When no recognised international reference standard exists, different platforms can report different values for the same sample, as the free light chain authors point out.5 And when no standardised protocol exists, each laboratory may set its own cut-off, which is what happened in the anti-FGFR3 ELISA study.8
How to Measure Agreement: ICC, Kappa and Bland-Altman
Choose the statistic from the kind of result. Table 1 collects how the sources read each one. Note that the same kappa can earn different labels: 0.77 is substantial on the Cohen bands and moderate in McHugh’s table.3
Swipe sideways to see every column.
| Measure | How the sources read the value | Source |
|---|---|---|
| ICC | Below 0.5 poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good, above 0.90 excellent, judged on the 95% confidence interval of the estimate. | 2 |
| Kappa, Cohen bands | 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, 0.81 to 1.00 almost perfect, as quoted by McHugh. | 3 |
| Kappa, McHugh table | 0.60 to 0.79 moderate, 0.80 to 0.90 strong, above 0.90 almost perfect; any kappa below 0.60 indicates inadequate agreement. | 3 |
| Percent agreement | McHugh notes that many texts recommend 80% agreement as the minimum acceptable interrater agreement. | 3 |
| Bland-Altman | Plots the difference between paired measurements against their mean; limits of agreement are the mean difference plus or minus 1.96 standard deviations of the differences. The plot does not say whether the agreement is good enough: that depends on analytical, biological or clinical goals. | 4 |
Table 1. Interpretation thresholds as stated in the cited papers. They are conventions, not laws, and each source warns about its own limits.
Two cautions run through these papers. Koo and Li note that an ICC comes in several forms, so the form used must be reported.2 McHugh notes that a kappa interval from a small sample can be wide enough to include no agreement.3
Worked Example 1: Eight Laboratories, Fifteen Samples
Natali and colleagues asked eight laboratories to measure the kappa free light chain (KFLC) index in 15 pooled cerebrospinal fluid and serum samples. The K-index divides the CSF/serum KFLC ratio by the CSF/serum albumin ratio, and a value of 5.8 or above was read as positive. Four laboratories used Binding Site instruments and assays, three used Siemens instruments and assays, and one used a Siemens instrument with a Binding Site assay. The samples came from 37 patients and were chosen to include high values, very low values and values near the cut-off.5 The paper is open access under CC BY 4.0 and prints every laboratory’s K-index for every sample, so we redrew and recomputed it in R.

Eleven samples received the same call in all eight laboratories. In the other four, 7 of the 120 laboratory-by-sample calls fell on the minority side of the call made by most laboratories, which Editverse computed as 5.8%. Three samples had readings of 4.6 to 5.5, between 0.3 and 1.2 below the cut-off, and one sample (number 10) was read as 9.8 by the mixed-platform laboratory while the other seven gave 0.4 to 3.2.
The mixed laboratory is not a one-off in this table: it read at or above every other laboratory in 14 of the 15 samples. Figure 2 compares it with the geometric mean of the other seven on a ratio scale; Giavarina describes plotting differences as percentages or ratios, and log-transforming the data when differences are not normally distributed.4

On average the mixed laboratory read 2.27 times higher than the geometric mean of the other seven (95% CI 1.83 to 2.83), with 95% limits of agreement of 1.05 to 4.94. With only 15 samples each limit is itself uncertain: the lower limit has a 95% interval of 0.72 to 1.53 and the upper of 3.38 to 7.23, following the interval method Giavarina describes.4 The authors of the paper note that this laboratory seemed to underestimate serum KFLC, which raises the K-index, and say no conclusion can be drawn yet.5
Paper against recomputation
Table 2 sets the published statistics beside ours. Two reproduce: the 11 of 15 count, and the all-laboratory kappa of 0.77 when it is computed as a Fleiss kappa across the eight laboratories. The Binding Site ICC reproduces exactly when it is computed as an average-measures, two-way, absolute-agreement ICC.
Swipe sideways to see every column.
| Quantity | Paper reports | Editverse result (R) |
|---|---|---|
| Samples with the same positive or negative call in all 8 laboratories | 11 of 15 | 11 of 15 |
| Kappa for a positive K-index (5.8 or above), all laboratories | 0.77 | 0.77 (Fleiss kappa); 0.78 (mean of 28 pairwise Cohen kappas) |
| K-index ICC, 4 Binding Site laboratories | 0.97 (0.94 to 0.99) | 0.97 (0.94 to 0.99) as an average-measures ICC; 0.89 (0.79 to 0.96) single-measure |
| K-index ICC, 3 Siemens laboratories | 0.95 (0.89 to 0.98) | 0.99 (0.98 to 1.00) single-measure |
| K-index ICC, 7 laboratories without the mixed one | 0.91 (0.82 to 0.96) | 0.94 (0.89 to 0.98) single-measure |
| K-index ICC, all 8 laboratories | 0.65 (0.43 to 0.84) | 0.77 (0.61 to 0.90) single-measure |
| Kappa within the Binding Site and within the Siemens groups | 0.89 and 0.71 | 0.75 and 0.79 (Fleiss) |
Table 2. Published values from Natali et al.; Editverse values computed in R with the irr package from the 15-by-8 table. Variants tried: one-way and two-way models, single and average measures, agreement and consistency, log scale, a strict greater-than 5.8 rule, and Fleiss versus mean pairwise Cohen kappa. The table headings print B1 twice; we read the columns left to right as B1 to B4, mixed, S1 to S3.
The other rows did not reproduce under any convention we tried, and we cannot say whether the cause is a column label, an ICC convention or the way the groups were formed. That matters for reading, not for the conclusion: kappa and ICC depend on the model chosen, so a methods section must name the one used.2 Removing the mixed laboratory raises the Fleiss kappa from 0.77 to 0.80.
What this case shows: how a multi-laboratory design is analysed, that disagreement clusters near the cut-off, and that a single platform mismatch can shift a derived index. What it does not show: how the K-index performs in patients, which platform is correct, or how other tests behave. The sample is 15 pooled samples from 37 patients, selected around the cut-off, from one 2022 study, so the 11-of-15 figure depends on that selection.5
Worked Example 2: One ELISA in Two Laboratories
Appeltshauser and colleagues ran an anti-FGFR3 antibody ELISA on 42 sera in one French and one German laboratory with identical protocols: 18 patients with sensory neuronopathy, 18 with small fiber neuropathy and 6 healthy controls. The sera were selected to include all seropositive samples from the German screen and matched seronegative ones, so the set is enriched for borderline cases.8 The paper is open access under CC BY 4.0.

The two laboratories agreed on 34 of 42 sera (13 positive in both, 21 negative in both), which is 81.0% with a Wilson 95% interval of 66.7% to 90.0% computed by Editverse. The paper reports Cohen’s kappa 0.61 and calls it substantial. The text does not say how the 8 discordant sera split between the two directions, so we computed kappa for every possible split: it ranges only from 0.605 to 0.613 when each direction of disagreement has at least one serum, as the paper’s separate medians for sera positive in laboratory 1 only and in laboratory 2 only imply, so the reported 0.61 sits inside that range. On McHugh’s stricter table that same value is moderate.8,3
The cut-offs differ more than the calls do. Each laboratory set its own cut-off from healthy-control sera, 65 for laboratory 1 and 50 for laboratory 2: 0.371 in laboratory 1 and 0.122 in laboratory 2, a ratio of 3.04 computed by Editverse, with a median optical density of 0.293 against 0.076. The authors conclude that cut-off values have to be determined separately for every laboratory. Seven of the 8 discordant sera were only weakly positive.8
A second finding concerns protocol detail. Laboratory 2 first built the assay from published information alone, then repeated it with the detailed protocol from laboratory 1.8
Swipe sideways to see every column.
| Laboratory 2 protocol | Cut-off (OD440) | Positive: sensory neuronopathy | Positive: small fiber neuropathy | Positive: healthy controls |
|---|---|---|---|---|
| Published information only | 0.102 | 1 of 17 | 1 of 60 | 1 of 50 |
| Detailed protocol from laboratory 1 | 0.122 | 1 of 17 | 5 of 60 | 2 of 50 |
Table 3. Counts and cut-offs as reported by Appeltshauser et al. for laboratory 2. The authors conclude that small protocol modifications can change cut-off values and results.
What this case shows: a research ELISA can reach substantial kappa on a borderline-rich panel while its cut-off differs about three-fold between sites, and a protocol detail can move who is called positive. What it does not show: which laboratory is right. The authors state that no independent gold standard exists, so it is unclear whether discordant sera are false positives or false negatives, and a positive result should be read in clinical context.8
How to Design and Report a Multi-Laboratory Agreement Study
The two examples and the cited guidance point to the same checklist.
- 1Send identical samples and say how they were chosen. The free light chain study pooled stored samples to include high, very low and near-cut-off values; the FGFR3 study selected borderline-rich sera. Say so, because it shapes any agreement figure.5,8
- 2Write the protocol so another laboratory can repeat it. STARD 2015 asks for the index test in sufficient detail to allow replication (item 10a), and the FGFR3 authors published a detailed ELISA protocol for that reason.7,8
- 3State each laboratory cut-off and whether it was pre-specified. STARD 2015 item 12a asks for the definition of and rationale for test positivity cut-offs, distinguishing pre-specified from exploratory. Report them per laboratory, as the FGFR3 paper does.7,8
- 4Match the statistic to the data and name its form. ICC for numeric results with the form stated, kappa for categorical calls, Bland-Altman limits of agreement for two methods.2,3,4
- 5Give intervals, not only point estimates. Koo and Li judge reliability on the confidence interval of the ICC, McHugh explains that kappa is an estimate for which a confidence interval is of interest, and Giavarina gives intervals for the limits of agreement.2,3,4
- 6Look at samples near the cut-off separately. Both worked examples found discordance concentrated there, so show those samples, as Figure 1 does.5,8
- 7Report to GRRAS. The GRRAS guideline proposes 15 issues to address when reporting reliability and agreement, matched to the usual headings of a paper. We read its abstract and the EQUATOR Network record; the publisher page was not accessible to us, so take the item wording from the paper itself.1
Verification and Proficiency Testing
Agreement studies sit beside routine quality rules. In the United States, CLIA regulations set what a laboratory must do before it reports patient results with a new test system, and require enrolment in proficiency testing. Table 4 summarises the text of the Electronic Code of Federal Regulations dated 30 October 2026; other countries have their own rules.
Swipe sideways to see every column.
| Situation (United States) | Required before reporting patient results | Source |
|---|---|---|
| Unmodified, FDA-cleared or approved test system | Demonstrate performance comparable to the manufacturer for accuracy, precision and reportable range, and verify that the manufacturer reference intervals suit the patient population. | 9 |
| Modified FDA-cleared test, in-house method, or no manufacturer specifications | Establish accuracy, precision, analytical sensitivity, analytical specificity (including interfering substances), reportable range and reference intervals, plus any other characteristic required. | 9 |
| Proficiency testing | Enrol in an HHS-approved proficiency testing program for each specialty and subspecialty, and test the samples in the same manner as patient specimens. | 10 |
Table 4. United States rules only, from 42 CFR 493.1253 and 493.801 as published on the eCFR with a date of 30 October 2026. Regulations change, so check the current text.
A review of method verification lists the characteristics a laboratory assesses when it introduces a new method: precision, trueness, analytical sensitivity, detection limits, analytical specificity, interference, measuring range, linearity and measurement uncertainty.11 The same review says trueness can be assessed against certified reference materials or proficiency testing material, with the consensus mean and standard deviation of proficiency material used only when obtained from at least ten peer participants, and that comparability between two methods or instruments can be assessed with 40 to 100 samples run on both, naming Bland-Altman analysis among the possible methods.11
Need the Analysis Run or Checked? Consult Editverse
Editverse subject-matter experts work to COPE, ICMJE and GPP 2022 guidance, with institutional and country-specific ethics followed.
Editverse
SPSS, R and Stata Analysis
Agreement data from several laboratories and the analysis not run or checked?
You send: your data, research questions and preferred software (SPSS, R or STATA; SAS on request).
You get: annotated code or syntax, complete output files, an output interpretation guide and publication-ready tables.
When: You have the laboratory data and need the analysis run or checked.
SPSS/R/STATA Statistical Analysis · $400
Editverse
Manuscript Writing
Agreement study analysed and the paper still unwritten?
You send: your data, results and target journal.
You get: a manuscript written by PhD-qualified subject matter experts, with editing, statistical review, journal selection and submission support.
When: The analysis is done and the paper is not.
Manuscript Writing Services · $1,500
Editverse
Journal Selection
Not sure where a laboratory method study belongs?
You send: your manuscript or abstract and your publication goals.
You get: a ranked list of 5-10 recommended journals with the rationale, based on scope fit, impact factor and acceptance rates.
When: You have results and need the right place to send them.
Scientific Journal Selection Services · $115
Free Research Tools from Editverse
- Bland-Altman Plot Calculator: mean bias and 95% limits of agreement for two measurement methods, with the R command behind the result.
- ICC Calculator (Intraclass Correlation Coefficient): the one-way random-effects, single-measure ICC(1,1) with its formula and R code.
- Cohen’s Kappa Calculator (Inter-Rater Agreement): observed and expected agreement and kappa for two raters.
Frequently Asked Questions
What is the difference between repeatability and reproducibility in diagnostic tests?
Repeatability describes variation when the same laboratory measures the same sample again. Reproducibility describes variation between laboratories, instruments or operators measuring the same sample. Pradhan and colleagues use these meanings, with reproducibility for results released by two laboratories and repeatability for results released by one.
Which statistic should I use to compare laboratories?
Match the statistic to the data. For numeric results across raters or laboratories, Koo and Li describe the intraclass correlation coefficient and ask authors to state the ICC form used. For two methods on the same numeric scale, Giavarina describes Bland-Altman limits of agreement. For positive or negative calls, McHugh describes the kappa statistic and discusses its confidence interval.
What ICC value counts as good reliability?
Koo and Li give values below 0.5 as poor, 0.5 to 0.75 as moderate, 0.75 to 0.9 as good and above 0.90 as excellent, judged on the 95% confidence interval and not on the point estimate. In the eight-laboratory example the serum ICC of 0.81 had an interval of 0.53 to 0.93, which spans moderate to excellent.
Is a kappa of 0.61 good agreement?
It depends on the scale. Cohen bands, as quoted by McHugh, call 0.61 to 0.80 substantial. McHugh proposes a stricter table that calls 0.60 to 0.79 moderate and says any kappa below 0.60 shows inadequate agreement. In the two-laboratory FGFR3 example the paper reports 0.61, and Editverse found it computes to between 0.605 and 0.613 whatever the split of the 8 discordant sera, since the paper reports sera positive in only one laboratory in each direction.
Why do laboratories disagree most near the cut-off?
A result close to the threshold can fall on either side with small differences between laboratories. In the kappa free light chain example the readings behind three discordant samples were 0.3 to 1.2 below the 5.8 cut-off, and in the FGFR3 example 7 of the 8 discordant sera were only weakly positive. The FGFR3 authors list probe handling, laboratory temperature, reagent preparation, hardware and pipetting as possible reasons.
Do laboratories need to verify an FDA-cleared test before using it?
Under the United States CLIA rule at 42 CFR 493.1253, a laboratory introducing an unmodified FDA-cleared or approved test system must, before reporting patient results, demonstrate comparable accuracy, precision and reportable range and verify that the manufacturer reference intervals suit its patients. A modified or in-house test needs the laboratory to establish a longer list of performance specifications. This describes the eCFR text dated 30 October 2026 and applies to the United States only.
Can two laboratories use different cut-offs for the same assay?
In the FGFR3 ELISA study the cut-off was 0.371 in one laboratory and 0.122 in the other, and the authors concluded that cut-off values have to be determined separately for every laboratory because mean optical densities vary between settings. That is the conclusion of one research assay, not a rule for every test.
How many samples does an agreement study need?
Neither worked example sets a rule. The kappa free light chain authors name the low number of samples as a limit, and McHugh notes that with very small samples the kappa interval can be wide enough to include no agreement. With 15 samples, the Editverse Bland-Altman limits of agreement were themselves uncertain, with a 95% interval for the lower limit of 0.72 to 1.53.
References
- Kottner J, Audigé L, Brorson S, et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. doi:10.1016/j.jclinepi.2010.03.002. Abstract and the EQUATOR Network record read; the publisher full text was not accessible to us. pubmed.ncbi.nlm.nih.gov/21130355/
- Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012. pmc.ncbi.nlm.nih.gov/articles/PMC4913118/
- McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282. pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
- Giavarina D. Understanding Bland Altman analysis. Biochem Med (Zagreb). 2015;25(2):141-151. doi:10.11613/BM.2015.015. pmc.ncbi.nlm.nih.gov/articles/PMC4470095/
- Natali P, Bedin R, Bernardi G, et al. Inter-laboratory concordance of cerebrospinal fluid and serum kappa free light chain measurements. Biomolecules. 2022;12(5):677. doi:10.3390/biom12050677. Open access, CC BY 4.0; K-index values taken from the authors’ Table 2, all statistics recomputed and figures drawn by Editverse in R. pmc.ncbi.nlm.nih.gov/articles/PMC9138559/
- Pradhan S, Gautam K, Pant V. Variation in laboratory reports: causes other than laboratory error. JNMA J Nepal Med Assoc. 2022;60(246):222-224. doi:10.31729/jnma.6022. pmc.ncbi.nlm.nih.gov/articles/PMC9200014/
- Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. doi:10.1136/bmj.h5527. pmc.ncbi.nlm.nih.gov/articles/PMC4623764/
- Appeltshauser L, Moritz CP, Reinhardt L, et al. Inter-laboratory comparison of a serum fibroblast growth factor receptor 3 (FGFR3) antibody test in sensory neuropathies. Front Immunol. 2025;16:1604456. doi:10.3389/fimmu.2025.1604456. Open access, CC BY 4.0; counts and cut-offs taken from the text, derived values computed by Editverse in R. pmc.ncbi.nlm.nih.gov/articles/PMC12549695/
- 42 CFR 493.1253, Standard: establishment and verification of performance specifications. Electronic Code of Federal Regulations, text dated 30 October 2026 (United States). www.ecfr.gov/current/title-42/section-493.1253
- 42 CFR 493.801, Condition: enrollment and testing of samples (proficiency testing). Electronic Code of Federal Regulations, text dated 30 October 2026 (United States). www.ecfr.gov/current/title-42/section-493.801
- Abdel Ghafar MT, El-Masry MI. Verification of quantitative analytical methods in medical laboratories. J Med Biochem. 2021;40(3):225-236. doi:10.5937/jomb0-24764. pmc.ncbi.nlm.nih.gov/articles/PMC8199534/
Related Reading
Editverse Research Team | Updated November 2026
