Newcastle-Ottawa Scale (NOS): Cohort and Case-Control Scoring, Reporting and Limitations

Published October 29, 2026

The Newcastle-Ottawa Scale (NOS) is a star-based rating tool built to judge the methodological quality of non-randomised studies in systematic reviews and meta-analyses, with one version for cohort studies and one for case-control studies. A study earns stars in three domains, and the maximum is nine. This guide walks through both versions item by item, scores a hypothetical cohort study star by star, shows how to report the scale in a methods section, and sets out what the published literature says about its reliability and limits.

What the Newcastle-Ottawa Scale Measures and Who Built It

The Ottawa Hospital Research Institute describes the Newcastle-Ottawa Scale as a tool for assessing the quality of nonrandomised studies, such as case-control and cohort studies, for use in systematic reviews and meta-analyses. It grew out of a collaboration between the Universities of Newcastle (Australia) and Ottawa (Canada). The institute lists the source as Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M and Tugwell P, “The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses”, and hosts the coding manual and the rating sheets for download.

The scale uses a star system. Each study is judged from three perspectives, namely the selection of the study groups, the comparability of the groups, and the ascertainment of either the exposure or the outcome of interest. The same page states that content validity and inter-rater reliability have been established, while criterion validity, intra-rater reliability and construct validity were still under evaluation. That last sentence matters for how much weight a star total can carry, and later sections return to it.

Two points shape everything that follows. First, the NOS appraises a whole study, not a single result. Second, it was designed for observational designs where participants were not randomised. If your review includes randomised trials, you need a different tool, and our guide to quality assessment tools in systematic reviews compares the options by study design.

Newcastle-Ottawa Scale Structure: Three Domains, Nine Stars

Both versions share one architecture. According to the rating sheets, a study can be awarded a maximum of one star for each numbered item within the Selection category and within the Outcome category (cohort version) or Exposure category (case-control version). A maximum of two stars can be given for Comparability. Four selection items, one comparability item worth up to two stars, and three outcome or exposure items give nine stars in total for either version.

The difference between the versions sits in the third domain. A cohort study is judged on how the outcome was assessed and followed up, because the exposure is known at the start. A case-control study is judged on how the exposure was ascertained, because the outcome is known at the start. If you are unsure which design a paper really used, our explainer on cohort and case-control designs and their effect measures separates them.

The Cohort Version Item by Item

The table below restates the cohort rating sheet. The “star” column lists the responses the sheet marks with a star, and the “no star” column lists the rest.

The Case-Control Version Item by Item

The case-control sheet keeps the same selection and comparability skeleton but replaces the outcome block with three exposure items. It also asks whether cases and controls were handled the same way, which has no cohort equivalent.

Coding Manual Rules That Change Your Score

The star sheets are short, and most scoring disagreements come from the coding manual, which carries the rules the sheets leave out. Several of them deserve a place in your review protocol before anyone opens a paper.

• Comparability needs design or analysis, not a p-value. The manual states that either the groups must be matched in the design or confounders must be adjusted for in the analysis, and that statements of no differences between groups, or of differences that were not statistically significant, are not sufficient. If the relative risk (or odds ratio in case-control studies) is adjusted for the listed confounders, the groups count as comparable on each variable used in the adjustment.
• You choose the “most important factor”. The sheet leaves a blank for it, and the second comparability star can be tied to a specific second factor. Fix both in the protocol so every rater applies the same list.
• Follow-up length is set before assessment. The manual says an acceptable length of time should be decided before quality assessment begins, and gives five years for exposure to breast implants as its example. The same logic applies to the retention percentage in the adequacy-of-follow-up item.
• Outcome evidence depends on the outcome. For a hip fracture, reference to the medical record is enough to confirm the event, but vertebral fracture needs reference to x-rays.
• Mortality studies still need “no prior disease”. For death outcomes, the outcome of interest is still the presence of the disease or incident, so a statement of no history of the disease earns the star.
• Control definition depends on the case definition. If cases are a first occurrence, the paper must state that controls have no history of the outcome. If cases are new but not necessarily first occurrences, controls with previous occurrences should not be excluded.
• Case definition asks for independent validation. The manual gives examples such as more than one person, record, time or process extracting the information, or reference to a primary record source such as x-rays or hospital records.

Scoring a Cohort Study With the Newcastle-Ottawa Scale, Star by Star

The example below is hypothetical, built by Editverse to show the mechanics. Imagine a published cohort comparing hospital employees who work regular night shifts with colleagues on day shifts, followed for incident type 2 diabetes. Our review protocol, written before screening, fixed three settings. Age is the most important comparability factor, body mass index is the additional factor, and a follow-up of at least five years with at least 90% retention counts as adequate.

The total of 7 hides two separate gaps. The paper never says whether participants were free of diabetes at baseline, and it does not describe the 18% who left the study. Item-level reporting shows a reader exactly where the evidence is thin, which a bare “7/9” does not.

Now see how fragile a category can be. The table applies the Agency for Healthcare Research and Quality conversion described in the next section to three plausible single-item disagreements between raters.

One star moved the same study from Good to Fair or to Poor. That is arithmetic on a published convention, not an empirical finding, but it explains why two independent raters and a written disagreement rule are worth the effort. Hartling and colleagues measured how often raters actually diverge, and their figures appear in the limitations section below.

Turning Stars Into Good, Fair or Poor Is a Convention

The Ottawa manual and rating sheets contain no pass mark. They tell you how to award stars and stop there. The labels you see in published reviews come from conversions that review authors add themselves, and one conversion attached to the NOS in published reviews is described as a conversion to the standards of the Agency for Healthcare Research and Quality (AHRQ). Esmailian et al. (2026, PMID 42006405) label it that way, and Heera and Babu (2025, PMID 40756036) apply the same thresholds.

Notice the structure of the AHRQ rule. It looks at each domain, not at the grand total, so a study with 7 stars can still be Poor if one domain collapses. The NOS-xs bands, by contrast, work on the total, and they belong to a new adapted tool rather than to the original NOS. Because none of these cut-offs is part of the scale, state the one you use in the protocol and cite who defined it.

A Real Example of Newcastle-Ottawa Scale Scoring on a Published Cohort Study

The hypothetical study above shows the mechanics. A published paper shows what the same eight decisions look like when the evidence sits in real Methods text and real tables. This section scores one open-access cohort study with the cohort rating sheet, one item at a time, and points to the part of the paper where each answer is found. It is illustrative Editverse scoring for teaching and not a judgement on the authors or on the quality of their work.

The review question and protocol settings assumed

NOS comparability depends on the review question, so the question has to be fixed before any star is counted. Assume a review that asks whether, in adults of working age, an incident hospital-diagnosed VTE is associated with later permanent loss of work capacity, measured as receipt of a disability pension. The protocol for that review sets age as the most important comparability factor, comorbidity and socioeconomic status as additional factors, and a median follow-up of at least 3 years as adequate. All three settings are written before screening, as the coding manual rules require.

Star-by-star scoring against the cohort rating sheet

The option letters in the reasoning column follow the cohort rating sheet of the Ottawa Hospital Research Institute, and the reasoning applies the coding manual. Section and table pointers refer to the open-access article.

Three rows carry a judgement call badge because the star depends on a setting the review team chooses and not on the paper alone. Comparability depends on which factors the protocol names. The follow-up row depends on the minimum length, and the adequacy row depends on how a rater treats a registry that tracks every resident. The other four rows follow directly from the sheet wording, although Selection 3 rests on treating a validated hospital registry as a secure record.

The Good, Fair or Poor label is a separate convention

The Ottawa documents give no pass mark, so the label below is a second step that a review team chooses to add. The AHRQ conversion described earlier, as applied by Esmailian et al. (2026) and Heera and Babu (2025), checks each domain and not the grand total. Selection 4, Comparability 2 and Outcome 3 meet the “Good” rule of 3 or 4 stars, 1 or 2 stars and 2 or 3 stars. The second row of the table shows how a stricter rater would score the same paper.

Two stars disappeared in the second scenario and the category did not change. In the hypothetical night-shift example earlier on this page, a single star moved a study between categories. Which pattern appears depends on where the stars are lost, and that is a reason to report stars by item and not only the total.

What a 9 of 9 does and does not tell you

A total of 9 means the sheet found no visible design weakness in the items it asks about. It does not mean the estimate is free of bias. The authors write that “residual confounding due to unmeasured factors cannot be completely ruled out” (Discussion), and they name individual health behaviour and workplace factors as information the registers lack. The sheet has no item that takes such a statement into account, so a reviewer who wants that information in the synthesis has to record it separately, for example in a sensitivity analysis by domain.

The paper reports its design under the STROBE guideline, which is one reason each row above could be answered from the text without writing to the authors. Our STROBE guidelines tutorial for observational studies lists what a cohort paper needs to report. In a supplementary appraisal table this study would take one line, with Selection 4, Comparability 2, Outcome 3 and a total of 9, next to the three protocol settings that produced those stars.

How to Report the Newcastle-Ottawa Scale in a Systematic Review

A reader should be able to repeat your appraisal. A reporting block that does the job has five parts.

• Version and source. Name the cohort or case-control version, cite Wells et al. with the Ottawa Hospital Research Institute page, and say whether you used the sheet unmodified.
• Protocol settings. Report the “most important” and “additional” comparability factors, the minimum follow-up length, and the retention threshold, each fixed before screening.
• Raters and disagreement handling. Give the number of raters, whether they worked independently, and how conflicts were resolved. Published reliability studies used two reviewers (Hartling) and six raters (Kalaycioglu), so two independent raters with a stated tie-break rule is a defensible minimum.
• Item-level table. Show stars per item for every included study, in the main text or a supplement, rather than only totals.
• Use of the scores. Say whether you describe, subgroup or run sensitivity analyses by quality, and state any category threshold with its source.

Here is a methods sentence you can adapt. “Two reviewers independently appraised each cohort study with the Newcastle-Ottawa Scale (cohort version; Wells et al., Ottawa Hospital Research Institute), awarding up to nine stars across selection, comparability and outcome. Age was predefined as the most important comparability factor and body mass index as the additional factor; follow-up of at least five years and retention of at least 90% were predefined as adequate. Disagreements were resolved by discussion with a third reviewer. Studies were classified as good, fair or poor using the AHRQ conversion thresholds.”

When a paper is silent on an item, you can write to the authors. Lo and colleagues compared reviewer scores with scores supplied by the authors themselves and concluded that contacting authors is needed for information that studies do not publish. Where you do not hear back, record the item as not reported, not as a failure of the study. Reporting the same appraisal in a manuscript is easier when the underlying paper also follows reporting guidance, which is the subject of our STROBE guidelines tutorial for observational studies.

Known Limitations of the Newcastle-Ottawa Scale in the Literature

The scale is widely applied, and it has also been examined critically. The points below are what the published studies report, with their numbers.

Arbitrary item definitions and a misquoted critique

Stang (2010, European Journal of Epidemiology, volume 25, pages 603-605) published a critical evaluation of the NOS. A later analysis by Stang, Jonas and Poole (2018) describes that commentary as criticising the scale for arbitrary definitions of quality items. They examined a random sample of 100 papers citing it, of which 96 were systematic reviews. None quoted the commentary directly, and all but 2 of the 96 indirect quotations (98%) portrayed it as supporting the use of the NOS when the opposite was the case. The commentary had been cited 1,250 times through December 2016. The practical lesson is to read the critique before citing it, and to cite it accurately.

Reliability between raters

Hartling and colleagues had two reviewers apply the NOS to 131 cohort studies from eight meta-analyses. Agreement ranged from substantial for length of follow-up (kappa 0.68) to poor for selection of the non-exposed cohort (kappa -0.03) and for demonstration that the outcome was absent at the start (kappa -0.06). Overall-score reliability was fair (kappa 0.29, 95% confidence interval 0.10 to 0.47). The reviewers found the tool difficult to use and the decision rules vague, and the study found no association between individual items or the overall score and effect estimates. Notice that the two poorest items are items 2 and 4 of the cohort selection block used in the worked example above.

Dependence on what authors chose to report

Lo, Mertz and Loeb compared NOS scores from the reviewers of a published systematic review with scores completed by study authors for the same cohort studies. Authors of 65 of 182 studies (36%) responded. The median overall score was 6 (interquartile range 6 to 6) for reviewers and 5 (interquartile range 4 to 6) for authors, a significant difference (p below 0.001), and agreement on the overall score was poor (kappa -0.004). Item-level agreement ranged from slight (kappa 0.15) to poor (kappa -0.06).

Results are not uniformly poor

Kalaycioglu and colleagues had six raters assess 30 exposure studies with each of three tools, rating them as low, intermediate or high risk of bias. All tools, including the NOS, reached substantial inter-rater reliability (intraclass correlation 0.61 to 0.80), while the SIGN50 checklist reached almost perfect agreement. The authors note that tools with fewer and simpler criteria may yield more reliable assessments. Taken with the studies above, the fair reading is that reliability depends on the rating design, the training of raters, and the way scores are collapsed into categories.

Scope

The NOS was written for cohort and case-control studies. Carra and colleagues proposed an adaptation for cross-sectional studies (NOS-xs, with a four-star variant for prevalence studies), which shows both the demand for a wider tool and the fact that the original does not cover that design. For non-randomised studies of interventions, chapter 25 of the Cochrane Handbook (version 6.5, 2024) describes ROBINS-I as the tool for assessing risk of bias, and frames it as a framework for assessing a single result.

NOS vs ROBINS-I vs JBI: A Decision Table

Choosing a tool is mostly a question of what your review asks and which design the included studies used. The descriptive rows below come from the sources named in the last column. The “pick it when” row is Editverse’s own decision rule, offered as practical guidance and not as a published standard.

If the target journal or guideline panel names a tool, use that one and justify any addition. Our ROBINS-I tool guide covers the domain-based alternative in detail.

Frequently Asked Questions

How many stars can a study earn on the Newcastle-Ottawa Scale?

Up to nine in either version, made of 4 Selection stars, 2 Comparability stars, and 3 Outcome stars (cohort) or Exposure stars (case-control).

Is there an official cut-off for a good NOS score?

No. The Ottawa manual and rating sheets give none. Reviews apply their own conversions, such as the AHRQ good, fair and poor thresholds, and should cite the source.

Can I use the NOS for randomised trials or cross-sectional studies?

The scale was written for non-randomised cohort and case-control studies. Randomised trials use other tools, and a cross-sectional adaptation (NOS-xs) has been proposed by Carra and colleagues.

Should the NOS total be used to weight studies in a meta-analysis?

Hartling and colleagues tested validity directly and found no association between individual items or the overall score and effect estimates. Reporting item-level results and checking sensitivity by domain is easier to defend than weighting by stars.

Professional Support for Newcastle-Ottawa Scale Appraisal in Your Review

Choosing the tool, fixing the protocol settings, and training raters to apply them the same way are the steps that decide whether a Newcastle-Ottawa Scale table survives peer review. Editverse methodologists set these decisions up with you, work in alignment with COPE, ICMJE and GPP 2022, and follow institutional and country-specific requirements. The packages below match the stage your review is at.

If your protocol is already written and your included studies are extracted, the standalone analysis package is the fastest route to a sensitivity analysis by quality domain. If you want one team to carry the review from protocol to manuscript, the Complete SR + Meta-Analysis package covers quality assessment and risk of bias figures along the way.

Conclusion

The Newcastle-Ottawa Scale turns a study’s design features into at most nine stars, with a clear item list for cohort and case-control studies and a short coding manual that settles most edge cases. Its strengths are speed and familiarity. Its weaknesses, as the literature shows, are item definitions that critics call arbitrary, uneven agreement between raters, and a dependence on what authors report. Use it with protocol-fixed settings, two independent raters, item-level tables, and a stated source for any good, fair or poor label, and reach for ROBINS-I or a JBI checklist when your question or design calls for it.

References

1. Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute. ohri.ca/programs/clinical_epidemiology/oxford.asp
2. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25(9):603-605. PMID 20652370. doi:10.1007/s10654-010-9491-z
3. Stang A, Jonas S, Poole C. Case study in major quotation errors: a critical commentary on the Newcastle-Ottawa scale. Eur J Epidemiol. 2018;33(11):1025-1031. PMID 30259221. doi:10.1007/s10654-018-0443-3
4. Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A, Dryden DM. Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013;66(9):982-993. PMID 23683848. doi:10.1016/j.jclinepi.2013.03.003
5. Lo CK, Mertz D, Loeb M. Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments. BMC Med Res Methodol. 2014;14:45. PMID 24690082. doi:10.1186/1471-2288-14-45
6. Kalaycioglu I, Rioux B, Briard JN, Nehme A, Touma L, Dansereau B, Veilleux-Carpentier A, Keezer MR. Inter-rater reliability of risk of bias tools for non-randomized studies. Syst Rev. 2023;12:227. PMID 38057883. doi:10.1186/s13643-023-02389-w
7. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. PMID 27733354. doi:10.1136/bmj.i4919
8. Sterne JAC, Hernan MA, McAleenan A, Reeves BC, Higgins JPT. Chapter 25: Assessing risk of bias in a non-randomized study. In Higgins JPT, Thomas J, Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-25
9. Higgins JPT, Morgan RL, Rooney AA, et al. A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E). Environ Int. 2024. PMID 38555664. doi:10.1016/j.envint.2024.108602
10. Carra MC, Romandini P, Romandini M. Risk of bias evaluation of cross-sectional studies: adaptation of the Newcastle-Ottawa Scale. J Periodontal Res. 2026. PMID 40293188. doi:10.1111/jre.13405
11. Esmailian A, Raki C, Han HC, Alasti M. The role of exercise in reducing vasovagal syncope recurrence: a systematic review and meta-analysis. Am J Prev Cardiol. 2026. PMID 42006405. doi:10.1016/j.ajpc.2026.101504
12. Heera T, Babu R. Systematic review and meta-analysis comparing outcomes of reduction versus nonreduction pyeloplasty in children. J Indian Assoc Pediatr Surg. 2025. PMID 40756036. doi:10.4103/jiaps.jiaps_1_25
13. JBI critical appraisal tools (checklists by study design). JBI. jbi.global/critical-appraisal-tools
14. Jørgensen H, Horváth-Puhó E, Laugesen K, Brækkan S, Hansen JB, Sørensen HT. Risk of a permanent work-related disability pension after incident venous thromboembolism in Denmark: a population-based cohort study. PLoS Med. 2021;18(8):e1003770. PMID 34464405. doi:10.1371/journal.pmed.1003770
15. Wells GA, Shea B, O’Connell D, et al. Newcastle-Ottawa Quality Assessment Scale, cohort studies rating sheet, and coding manual for cohort studies. Ottawa Hospital Research Institute. rating sheet (PDF), coding manual (PDF)