Published | Last updated
AMSTAR 2 is a 16-item critical appraisal tool that tells you how much confidence to place in the results of a systematic review of healthcare interventions. Beverley Shea and colleagues published it in the BMJ in 2017 (BMJ 2017;358:j4008) as an update of the original AMSTAR, which was written for reviews of randomised trials only. A reader who applies the tool well ends up with one of four verdicts (high, moderate, low or critically low) and a written reason for each item behind it.
In Simple Words
AMSTAR 2 is a checklist for judging someone else’s systematic review. Sixteen questions look at how the review was planned, searched, appraised and written up. Seven of those questions are critical, because a miss there can undermine the whole review. Count the critical misses first. No critical miss and at most one smaller miss earns a high rating, and two or more critical misses make the review critically low. Nobody adds up a score.
Most appraisers go wrong on the rating logic, so this guide walks through the instrument in the order you will use it. Coverage includes the 16 items, the 7 critical domains, the four confidence levels, a full worked example that takes an invented review from item ratings to a final verdict, and a decision table for choosing between AMSTAR 2 and neighbouring tools. Every item definition below comes from the published paper and its supplementary appendices, not from secondary summaries.
What AMSTAR 2 Is and Why It Replaced the Original AMSTAR
AMSTAR stands for A MeaSurement Tool to Assess systematic Reviews, and the first version appeared in 2007, when Shea and colleagues started from a 37-item pool, applied it to 99 paper-based and 52 electronic systematic reviews, ran an exploratory factor analysis that found 11 components, and kept one item from each component through a nominal group process. Out came an 11-item instrument built for reviews of randomised trials.
By 2017 that design no longer matched the evidence base. Reviews increasingly pooled non-randomised studies of interventions (NRSI), and users had reported gaps in the original wording. An expert group met for a day in Ottawa, reviewed survey results from AMSTAR users and the recorded experience of participants in AMSTAR workshops at Cochrane Colloquiums in 2015 and 2016, before settling on the revised tool. Key changes in AMSTAR 2 include the following.
- AMSTAR 2 has 16 items, and item 12 (the impact of risk of bias on the meta-analysis) is new.
- Items on risk of bias, meta-analysis and study-design selection now cover both randomised trials and NRSI.
- Options for “not applicable” and “cannot answer” were removed, and every question is worded so that Yes marks a positive result.
- Partial Yes was added for items where partial adherence to the standard is worth recording.
- If a review does not report the information, the appraiser rates the item No. Review authors “should not be given the benefit of doubt”.
Shea and colleagues tested the revised tool with six raters on 54 systematic reviews, of which 20 included only randomised trials, 18 only NRSI and 16 a mixture. Of 50 kappa scores, 46 showed moderate or better agreement and 39 showed good or better agreement. Those numbers describe rater consistency, and the authors report that agreement varied substantially across items.
What AMSTAR 2 is not designed for
Shea and colleagues state that the revisions were not intended for diagnostic test reviews, individual patient data meta-analyses, network meta-analyses, scoping reviews or realist reviews. Nor does the instrument teach review methodology, and the paper points readers who want that to the Cochrane Handbook. If you are weighing a scoping review instead, our step-by-step scoping review guide covers the JBI and PRISMA-ScR route.
The 16 AMSTAR 2 Items at a Glance
Each row below gives the item as the supplementary AMSTAR 2 checklist words it, with the response options an appraiser can choose. Critical items are marked.
| Item | Question the appraiser answers | Critical | Response options |
|---|---|---|---|
| 1 | Did the research questions and inclusion criteria include the components of PICO? | No | Yes, No |
| 2 | Did the report state that review methods were established before the review, and justify significant deviations from the protocol? | Yes | Yes, Partial Yes, No |
| 3 | Did the authors explain their selection of study designs? | No | Yes, No |
| 4 | Did the authors use a comprehensive literature search strategy? | Yes | Yes, Partial Yes, No |
| 5 | Was study selection performed in duplicate? | No | Yes, No |
| 6 | Was data extraction performed in duplicate? | No | Yes, No |
| 7 | Did the authors provide a list of excluded studies and justify the exclusions? | Yes | Yes, Partial Yes, No |
| 8 | Were the included studies described in adequate detail? | No | Yes, Partial Yes, No |
| 9 | Was a satisfactory technique used to assess risk of bias (RoB) in individual studies? | Yes | Yes, Partial Yes, No (rated separately for RCTs and NRSI) |
| 10 | Did the authors report the funding sources of the included studies? | No | Yes, No |
| 11 | If a meta-analysis was done, were appropriate methods used to combine results? | Yes | Yes, No, No meta-analysis conducted |
| 12 | If a meta-analysis was done, did the authors assess the impact of RoB on its results? | No | Yes, No, No meta-analysis conducted |
| 13 | Did the authors account for RoB when interpreting and discussing the results? | Yes | Yes, No |
| 14 | Did the authors explain and discuss any heterogeneity? | No | Yes, No |
| 15 | If quantitative synthesis was done, was publication bias investigated and its likely impact discussed? | Yes | Yes, No, No meta-analysis conducted |
| 16 | Did the authors report potential conflicts of interest, including funding for the review? | No | Yes, No |
What earns a Partial Yes or a Yes
Five items carry a Partial Yes option, and the gap between Partial Yes and Yes is where appraisers disagree most. Criteria below come from the supplementary checklist published with the BMJ paper.
| Item | Partial Yes requires | Yes also requires |
|---|---|---|
| 2 Protocol | A stated written protocol or guide covering the review questions, search strategy, inclusion and exclusion criteria, and risk of bias assessment | A registered protocol that also specifies a synthesis plan if appropriate, a plan for investigating heterogeneity, and justification for any deviations |
| 4 Search | At least 2 relevant databases, keywords or a search strategy, and justified publication restrictions such as language | Reference lists of included studies searched, trial or study registries searched, content experts consulted, grey literature searched where relevant, and a search within 24 months of review completion |
| 7 Excluded studies | A list of all potentially relevant studies read in full text and then excluded | A justification for the exclusion of each study |
| 8 Included studies | Populations, interventions, comparators, outcomes and research designs described | Population, intervention and comparator described in detail (including doses where relevant), plus study setting and follow-up timeframe |
| 9 Risk of bias, RCTs | Assessment of unconcealed allocation and of lack of blinding of patients and assessors (unnecessary for objective outcomes such as all-cause mortality) | Assessment of a non-random allocation sequence and of selection of the reported result among multiple measurements or analyses |
| 9 Risk of bias, NRSI | Assessment of confounding and of selection bias | Assessment of methods used to ascertain exposures and outcomes, and of selective reporting of results |
The 7 AMSTAR 2 Critical Domains
Box 1 of the paper lists seven domains that “can critically affect the validity of a review and its conclusions”. Supplementary guidance explains the reasoning behind each one, which makes the list easy to remember.
- Item 2, protocol registered before the review began. Systematic reviews are a form of observational research, and adherence to a well-developed protocol reduces the risk of bias in the review. Registration (for example at PROSPERO), publication in an open journal or a dated submission to a research office all count as independent verification.
- Item 4, adequacy of the literature search. A thin search can miss the studies that change the answer, and the checklist asks for databases, registries, reference lists and a recent search date.
- Item 7, justification for excluding individual studies. Unjustified exclusion may bias findings, and the guidance says exclusion should never be based on risk of bias, which is handled separately.
- Item 9, risk of bias from the included studies. Guidance calls this a key part of appraising any systematic review, particularly one that includes NRSI.
- Item 11, appropriateness of the meta-analytical methods. Pooling NRSI can produce a precise but biased estimate, so the item checks for weighted methods, heterogeneity handling and separate estimates for trials and NRSI.
- Item 13, consideration of risk of bias when interpreting results. Authors who make recommendations that could affect care or policy should show explicit consideration of risk of bias.
- Item 15, assessment of publication bias. Per the paper, a minimum of 10 studies is required to show funnel plot asymmetry, which matters when a review pools only a handful of trials. Our guide to the minimum number of studies for a meta-analysis covers that threshold in practice.
Treat the designation as a default, not a law. Two examples from the authors show where a different item becomes critical, namely failing to include non-randomised studies (item 3) in a review of adverse outcomes, and being unable to explain large variations in treatment effects across studies (item 14). If your team changes the critical list, record the change before rating begins so that the final verdict can be reproduced.
AMSTAR 2 Confidence Ratings: High, Moderate, Low and Critically Low
Box 2 of the paper turns item ratings into an overall verdict by counting two things separately. A critical flaw is a No on a critical item, and a non-critical weakness is a No on any other item. Four levels result.
| Overall confidence | What the appraiser found | What the paper says it means |
|---|---|---|
| High | No or one non-critical weakness | The review provides an accurate and comprehensive summary of the results of the available studies that address the question |
| Moderate | More than one non-critical weakness, no critical flaws | The review has more than one weakness but no critical flaws, and may provide an accurate summary of the included studies |
| Low | One critical flaw, with or without non-critical weaknesses | The review may not provide an accurate and comprehensive summary of the available studies |
| Critically low | More than one critical flaw, with or without non-critical weaknesses | The review should not be relied on to provide an accurate and comprehensive summary of the available studies |
Two rules sit beside the table. Multiple non-critical weaknesses may diminish confidence, so it can be appropriate to move an appraisal down from moderate to low. And the authors strongly recommend that individual item ratings are never combined into an overall score, because a high tally can hide a single critical failure such as an inadequate search or missing risk of bias assessment.
How to Use AMSTAR 2 Step by Step
Short as the instrument is, consistent results depend on a fixed routine. Below, the routine blends the authors’ guidance with a working method Editverse recommends.
- Confirm the review fits. Aim it at a systematic review of healthcare interventions that includes randomised trials, NRSI or both. Diagnostic accuracy reviews, network meta-analyses, individual patient data meta-analyses and scoping reviews fall outside the design brief.
- Gather every document. Pull the full report, supplements, the registered protocol and any list of excluded studies. Several items can only be answered from supplementary files.
- Rate with two independent appraisers. Each appraiser works alone, then both resolve disagreements by consensus. Agreement varied across items in the authors’ reliability study, so two raters and a consensus step are good practice.
- Rate each item strictly. Use the response options on the checklist. Missing information scores No, with no benefit of the doubt.
- Compare the report with the protocol. Guidance tells appraisers to check the published report against the registered protocol when one exists, and obvious unexplained discrepancies should lead to downgrading.
- Tally flaws and weaknesses separately. Count Nos on critical items and Nos on other items as two different numbers.
- Assign the confidence level and document variations. Write down any departure from the default critical list or any convention you adopted, as the authors request.
One practical order saves time. Start with items 2, 4 and 7, because these critical items are visible in the methods section and registry entry within minutes. A review with two failures there is already critically low, and the remaining thirteen items then document the detail rather than decide the outcome.
Worked Example: Rating a Hypothetical Review From Item Ratings to a Final Verdict
Our case here is invented for teaching and does not describe any published review. Imagine a systematic review of supervised exercise therapy for chronic low back pain that pooled 14 randomised trials. An appraiser reads the report, the registry entry and the supplement, then rates each item.
| Item | Rating | Evidence found in the invented report |
|---|---|---|
| 1 PICO | Yes | Adults with chronic low back pain, supervised exercise, usual care comparator, pain and disability outcomes, follow-up timeframe stated |
| 2 Protocol (critical) | Yes | Registered at PROSPERO before screening began, with one added subgroup reported and justified |
| 3 Study designs | Yes | Restricted to randomised trials, with a stated reason |
| 4 Search (critical) | Partial Yes | Three databases, keywords reported, English-only restriction justified, but no registry search and no content experts consulted |
| 5 Selection in duplicate | Yes | Two reviewers screened independently and reached consensus |
| 6 Extraction in duplicate | Yes | Two reviewers extracted data and reached consensus |
| 7 Excluded studies (critical) | No | The flow diagram gives counts of full-text exclusions, but no list of the excluded studies with reasons |
| 8 Included studies | Partial Yes | Populations, interventions, comparators, outcomes and designs described, with no dose or setting detail |
| 9 Risk of bias (critical) | Yes | Tool covered allocation concealment, sequence generation, blinding and selective reporting |
| 10 Funding of studies | No | Trial funding not reported, and no statement that it was sought |
| 11 Meta-analysis (critical) | Yes | Pooling justified, weighted random-effects model, heterogeneity explored |
| 12 RoB impact | Yes | Sensitivity analysis excluding high-risk trials |
| 13 RoB in discussion (critical) | Yes | Discussion states how high-risk trials limit the conclusions |
| 14 Heterogeneity | Yes | Sources investigated and discussed |
| 15 Publication bias (critical) | Yes | Funnel plot and a statistical test with 14 trials, impact discussed |
| 16 Conflicts of interest | Yes | Competing interests and review funding reported |
Tallying the example
Only one critical item scored No, which is item 7, so the review carries one critical flaw. Only one non-critical item scored No, which is item 10, so it carries one non-critical weakness. One critical flaw lands in the Low row of Box 2, regardless of how many items scored Yes.
Nothing in the paper says how a Partial Yes is tallied. In this example a Partial Yes is not counted as a flaw, but the appraiser flags it in the report, as with the incomplete search at item 4. Teams can choose a stricter convention, provided it is written down before rating starts and applied to every review in the batch.
How one change moves the verdict
A what-if table shows how sensitive the final rating is to a few edits, which is also how an author should read an appraiser’s report.
| Scenario | Critical flaws | Non-critical weaknesses | Overall confidence |
|---|---|---|---|
| Baseline as rated above | 1 (item 7) | 1 (item 10) | Low |
| Authors add an excluded-studies list with reasons | 0 | 1 (item 10) | High |
| Search found to cover only one database, so item 4 becomes No | 2 (items 4 and 7) | 1 | Critically low |
| Excluded-studies list added, and items 10 and 14 both rated No | 0 | 2 (items 10 and 14) | Moderate |
Here lies the real lesson. A single missing supplementary file cost this review two full confidence levels, and a missing list is cheap to fix. Authors who run this check on their own manuscript before submission can see which repairs are cheapest.
AMSTAR 2 Real Example: Appraising a Published Open-Access Review Item by Item
The invented review in the worked example shows the mechanics. A published review shows what the same sixteen decisions look like when the evidence sits in real Methods text, real tables and a real supplementary appendix. This section appraises one open-access review with AMSTAR 2, one item at a time, and points to the place in the paper where each answer can be checked. It is an illustrative Editverse appraisal for teaching, not a judgement on the authors or on whether the intervention works, and the rows marked as judgement calls are where another appraiser could reasonably differ.
The Review Appraised in This Example
Citation. Galante J, Friedrich C, Dawson AF, Modrego-Alarcón M, Gebbing P, Delgado-Suárez I, et al. Mindfulness-based programmes for mental health promotion in adults in nonclinical settings: a systematic review and meta-analysis of randomised controlled trials. PLoS Med. 2021;18(1):e1003481. doi:10.1371/journal.pmed.1003481. PMID 33428616. PMCID PMC7799763.
Licence. Creative Commons Attribution 4.0 (CC BY 4.0), confirmed in the Crossref record of the version of record, in the Europe PMC record and in the PubMed Central open-access dataset. Short quotations and one figure are reused below under that licence, with attribution.
Why this review. The authors pooled 136 randomised trials with 11,605 participants, registered the protocol on PROSPERO before the work began, searched 13 databases with no language limit, and used two independent reviewers for selection, extraction and Cochrane Risk of Bias 2 appraisal. The text, the supplementary appendix and the deposited data are all open, so a reader can re-derive every row. It is also a useful teaching case because a review with this much good practice still leaves one critical item that a strict appraiser has to score as No.
Label. Illustrative Editverse appraisal for teaching. AMSTAR 2 rates how the review was conducted and reported. It does not rate the trials, which the authors graded separately, and it does not judge the authors.
The Study Selection Flow Behind Items 4, 5 and 7
Figure 3 is the review’s own flow diagram. It is worth reading before the table, because items 4, 5 and 7 all turn on what the report says about records that were screened, read in full and left out.
Item-by-Item Appraisal
Each row gives the item, what the paper reports with a pointer to where it sits, the rating, and the reasoning against the Yes and Partial Yes criteria in the Shea et al. supplementary checklist. Quotations are short and verbatim, with the journal’s non-breaking hyphens written as ordinary hyphens. S1 Appendix page numbers are the printed numbers in that file. Shaded rows are the seven critical items. As in the worked example above, a Partial Yes is flagged but not counted as a flaw.
| Item | What the paper reports (with pointer) | Rating | Reasoning against the AMSTAR 2 criteria |
|---|---|---|---|
| 1 PICO components | Methods, Study search and selection, and Outcomes. Adults living in the community with no selected clinical condition, group-based mindfulness programmes of at least four one-hour sessions, comparison with no intervention or another intervention, and four primary outcome domains. The paper states trials “included adult (18+ years old) participants living in the community” and outcomes “measured in a primary time range of between 1 and 6 months following programme completion”. | Yes | Population, intervention, comparator and outcome are all stated in the eligibility criteria, and the optional follow-up timeframe is given as well. Nothing is missing. |
| 2 Protocol before the review Critical | Abstract and Methods, first paragraph. A “detailed preregistered protocol” is cited as PROSPERO CRD42018105213, and the review procedures were “published in detail in a prospective protocol”. The paper labels departures, for example dispositional mindfulness as an outcome “although it is not in the review protocol”, and names several analyses as post hoc or ad hoc. | Yes | A registered protocol meets the Yes tier, which asks for registration plus a synthesis plan, a plan for investigating heterogeneity and justification of deviations. The report describes the moderator and sensitivity analyses as prespecified and flags each departure. The PROSPERO page needs JavaScript and could not be read for this appraisal, so the rating rests on the report’s own statements. A careful appraiser opens the registry entry to confirm those plans appear there. |
| 3 Choice of study designs | Introduction, last two paragraphs. Earlier reviews often pooled trials with weaker designs, since “both uncontrolled and nonrandomised trial findings are many times included”. This review “focused only on RCT data”, citing a critical mass of good-quality trials and calls to raise research quality. | Yes | The checklist accepts an explanation for including only randomised trials, and the Introduction gives one. |
| 4 Literature search Critical | Methods, Study search and selection; S1 Appendix Table 1, from p. 8. “Thirteen databases were electronically searched for eligible studies from inception to 1 August 2020”, including the WHO trial registry platform. No language or date restrictions were applied, reference lists were inspected, and unpublished reports such as theses were included. The paper appeared on 11 January 2021. | Partial Yes | Every Partial Yes element is met (two or more databases, search strategies given, no restriction to justify). The Yes tier adds reference lists, trial registries, grey literature and a search within 24 months of completion, all present. It also asks that content experts were included or consulted, and the report does not say so. Authors were contacted only for clarification or data. Judgement call. Many appraisers treat contacting trial authors as consulting experts and would score Yes. A Partial Yes is not counted as a flaw, so the final rating is the same either way. |
| 5 Selection in duplicate | Methods, Study search and selection. “2 reviewers independently assessed the titles and abstracts of retrieved records against inclusion criteria”, and full texts were again assessed independently. | Yes | Two reviewers selected independently at both stages, and full texts were obtained only for records not judged irrelevant by both, so inclusion rested on agreement. Resolution by discussion is stated explicitly for extraction, so consensus at selection is implied rather than written out. The rating stays Yes. |
| 6 Extraction in duplicate | Methods, Study search and selection; extraction forms in S1 Appendix Table 2, p. 11. “Two researchers independently extracted the information from the included full-text papers using prepiloted forms”. | Yes | Two independent extractors worked from prepiloted forms and settled disagreements within the team. |
| 7 List of excluded studies Critical | Figure 3 (Fig 1 of the paper); S1 Appendix Table 5, p. 18. Of 1,420 full texts assessed, 1,208 were excluded, with counts for study design (495), intervention (391), intervention duration (114), population (78), outcomes (75), route of administration (45), conference abstracts (6) and comparator (4). Table 5 lists the conference abstracts that held too little information to judge. No list names the other excluded full texts in the article, the appendix or the deposited data files. | No | Partial Yes needs a list of all potentially relevant studies that were read in full text and then excluded. Counts by reason are not a list, and Table 5 covers one of the eight reasons. Without even the Partial Yes list, the Yes tier (a justification for each study) cannot be reached. This is the review’s one critical flaw. Judgement call. The strict reading of the checklist wording is used here, and this single item decides the final rating (see the what-if table below). |
| 8 Included studies described | Results, Study characteristics; Table 1 of the paper. For each of 136 trials the table gives country, sample size, participant group, mean age, share of women, outcomes, intervention and control (each with a category) and outcome time points. The text adds that 129 trials were participant-level and 7 were cluster-randomised. | Partial Yes | Populations, interventions, comparators, outcomes and designs are described, which meets Partial Yes. The Yes tier wants intervention and comparator in detail, including dose where relevant, plus setting. Table 1 names each programme and control but does not give sessions or contact hours per trial, and setting appears only through the participant group. The extraction form (S1 Appendix Table 2) did capture those details and contact hours were used in the meta-regression. Judgement call. No effect on the final rating. |
| 9 Risk of bias technique Critical | Methods, Outcomes subsection (organisation, assessment and transformation); Results, Risk of bias and confidence in the evidence; Fig 3 of the paper; S1 Appendix Table 25, p. 38. Two researchers independently assessed every trial “using the Cochrane revised tool for assessing risk of bias in randomised trials”. Its five domains are the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome and selection of the reported result. | Yes | Only randomised trials were included, so the RCT column applies. Yes asks for allocation concealment, blinding of participants and assessors, the allocation sequence and selective reporting. The randomisation domain covers sequence and concealment, the deviations and measurement domains cover blinding, and the last domain covers selective reporting. |
| 10 Funding of included studies | Table 1 of the paper; S1 Appendix Table 2, p. 11. The extraction form had a tab for “Sponsorship sources, conflicts of interest”. Table 1, the Results and the appendix give no funding source for any included trial. | No | Yes needs funding sources reported for individual included studies. The checklist also accepts a statement that the reviewers looked and the study authors did not report it. The report states neither. This is a non-critical weakness. Judgement call. The extraction form shows the reviewers looked, and a lenient appraiser could score Yes. The final rating stays Low either way. |
| 11 Methods for combining results Critical | Methods, Data synthesis. The authors computed “pairwise random-effects meta-analyses within comparator categories”, using multivariate models fitted by restricted maximum likelihood, prediction intervals, and meta-regression on region, participant type, duration, added components and control type. | Yes | All three RCT elements are present. Combining is justified by keeping passive, active non-specific and active specific controls apart, the weighting is a random-effects model that accounts for heterogeneity, and causes of heterogeneity were investigated by moderator analyses. The NRSI elements do not apply to a trials-only review. |
| 12 Impact of risk of bias on the pooled result | Results, Sensitivity analyses; S1 Appendix Tables 28 to 31. Because every trial had a high-risk judgement, the authors report that “we removed trials deemed to be at high risk of bias from 3 or more sources” in a sensitivity analysis. Effects on anxiety, depression and well-being against passive controls lost significance, and the effect on distress remained. | Yes | The review did not restrict itself to low-risk trials, so it needs an analysis of the effect of risk of bias on the summary estimates, and the sensitivity analysis is exactly that. |
| 13 Risk of bias in interpretation Critical | Discussion, Summary of findings and Strengths and limitations; S1 Appendix Table 27, p. 43. “All of the trials included in the review were deemed at high risk of bias”, and “the low quality of most of the primary studies significantly affects confidence and therefore utility of the review results”. The risk of bias domain is rated serious in all 12 outcome rows of the GRADE table. | Yes | Trials at high risk of bias were included, so Yes requires a discussion of the likely impact on the results. The Discussion states the finding survives the risk of bias sensitivity analysis only for distress against passive controls and words confidence as very low to moderate. |
| 14 Heterogeneity explained and discussed | Methods, Data synthesis; Results, Moderator analyses; Discussion, Interpretation and comparison with previous research. “prediction intervals were estimated reflecting the variation in intervention effects over the different trial settings”. Inconsistency is rated serious in 10 of 12 rows of S1 Appendix Table 27. | Yes | Heterogeneity was present, so Yes needs an investigation of its sources and a discussion of its effect. Moderator analyses found smaller effects in trials from the USA and larger effects in higher-risk groups, and the Discussion concludes the results do not support generalisation to every setting. |
| 15 Publication bias Critical | Methods, Outcomes subsection (organisation, assessment and transformation); Results, Risk of bias and confidence in the evidence; S1 Appendix Table 26 (p. 42), Table 27 (p. 43) and funnel plots (pp. 70 to 72). Small-study effects were assessed by “visual inspection of funnel plots in meta-analyses of primary outcome domains with at least 10 studies”. The funnel plots showed “evidence of small-study or nonreporting biases in the outcome domain of depression for MBPs compared with passive control groups” and none for distress. Six registry records with no results were listed. | Yes | Yes needs graphical or statistical tests plus a discussion of the likelihood and magnitude of impact. Funnel plots were drawn for the comparisons with at least 10 studies, the findings were reported, and the issue fed into GRADE, where the non-reporting bias domain is rated serious in 5 of 12 rows. Judgement call. Item 15 has no Partial Yes. Table 27 rates non-reporting bias as not serious for depression against passive controls although the text reports small-study evidence there, and an appraiser who reads that as an unresolved discussion could score No, which would add a second critical flaw (see the what-if table). |
| 16 Conflicts of interest and review funding | Article front matter, Competing interests and Funding Statement. “The authors have declared that no competing interests exist.” The funding statement names the funders of the review and says they had no role in study design, data analysis or publication. | Yes | Either route to Yes is enough, and the paper offers both a no competing interests statement and a described funding source. |
From Item Ratings to the Final Confidence Rating
The final rating follows Box 2 without any score. The ratings above contain twelve Yes, two Partial Yes and two No, and those counts are for reading only. The rating comes from the two flaw counts below.
| Step | What the appraiser does | Result for this review |
|---|---|---|
| 1 | Read the seven critical items (2, 4, 7, 9, 11, 13, 15) and count each No | Items 2, 9, 11, 13 and 15 are Yes, item 4 is Partial Yes, item 7 is No. One critical flaw (item 7). |
| 2 | Read the nine non-critical items (1, 3, 5, 6, 8, 10, 12, 14, 16) and count each No | Items 1, 3, 5, 6, 12, 14 and 16 are Yes, item 8 is Partial Yes, item 10 is No. One non-critical weakness (item 10). |
| 3 | Apply the convention that a Partial Yes is flagged but not counted | Items 4 and 8 are noted in the report and do not add to either count. |
| 4 | Match the counts to Box 2. One critical flaw, with or without non-critical weaknesses, falls in the Low row | Low. In Box 2 terms, the review may not give an accurate summary of what the available studies show. |
| 5 | Check whether several non-critical weaknesses justify moving down a level | Only one weakness exists and the rating is already Low, so nothing changes. |
How Much One Item Moves the Verdict
Because the verdict rests on a single critical item, a what-if table is the honest way to show how firm it is. Each scenario changes one thing and recounts.
| Scenario | Critical flaws | Non-critical weaknesses | Overall confidence |
|---|---|---|---|
| As rated above | 1 (item 7) | 1 (item 10) | Low |
| The authors add a supplementary list of the 1,208 excluded full texts with a reason for each, so item 7 becomes Yes | 0 | 1 (item 10) | High |
| A lenient appraiser scores item 10 as Yes because the extraction form shows funding was sought | 1 (item 7) | 0 | Low |
| A stricter appraiser scores item 15 as No because Table 27 and the text disagree on depression | 2 (items 7 and 15) | 1 (item 10) | Critically low |
What This Real Example Shows
Three points come out of the appraisal. The review follows almost every practice AMSTAR 2 rewards, yet one missing list places it in the Low row, and a file listing the exclusions is a cheap repair at the manuscript stage, as the section on fixable critical flaws explains. The two items that failed here, 7 and 10, are the same two that failed in the invented review above, so the pattern in that table is not artificial. And the review’s own GRADE ratings (moderate to very low confidence in the findings) answer a different question from AMSTAR 2, which asks how reliably the review itself was done.
A reader can repeat the exercise on any open-access review. Open the Methods, the flow diagram and the supplementary appendix side by side, mark each critical item first, and write the pointer next to every rating so a second appraiser can check it.
Decision Table: AMSTAR 2, ROBIS, PRISMA 2020 and When to Use Each
Reviewers often confuse appraisal tools with reporting checklists. AMSTAR 2 rates confidence in a finished review, PRISMA 2020 tells authors what to report (its checklist has 27 items), and ROBIS assesses risk of bias in a review through three phases, namely an optional relevance check, identification of concerns with the review process, and a final judgement of risk of bias.
A direct comparison by Perry and colleagues, in which three reviewers independently assessed 31 systematic reviews with both tools, notes that strictly speaking AMSTAR-2 assesses methodological quality and ROBIS assesses risk of bias, although the signalling questions overlap considerably.
| Your situation | Best choice | Why |
|---|---|---|
| Appraising systematic reviews of healthcare interventions (trials, NRSI or both) for an overview, umbrella review or guideline | AMSTAR 2 | Built for exactly this design and yields a four-level confidence verdict |
| Appraising reviews of diagnosis, prognosis or aetiology | ROBIS | ROBIS targets interventions, diagnosis, prognosis and aetiology, while AMSTAR 2 was not designed for diagnostic test reviews |
| Writing your own systematic review | PRISMA 2020 for reporting, plus an AMSTAR 2 self-check | PRISMA 2020 governs the write-up, and AMSTAR 2 shows which methodological gaps a reader will find |
| Network meta-analysis, individual patient data meta-analysis or scoping review | Not AMSTAR 2 | The authors state these were outside the intended scope of the revision |
| Need both a conduct verdict and a bias verdict | AMSTAR 2 and ROBIS together | The overlap is large, but the two tools frame the question differently |
For ranking evidence strength across study designs, pair this with our guide to levels of evidence in clinical research, and for a broader comparison of appraisal instruments see quality assessment tools in systematic reviews. When the next step is certainty grading of outcomes, the free GRADE evidence table calculator builds the table.
Which Critical Flaws You Can Fix Before Submission
Authors who appraise their own manuscript can use AMSTAR 2 as a pre-submission audit. Critical items differ sharply in how repairable they are, and that difference should drive the order of work.
| Critical item | Repairable now? | What to do |
|---|---|---|
| 2 Protocol | Rarely | A Yes needs a registered protocol that predates the review, so the repair is to register before the next review begins. If a written protocol exists, state it explicitly to reach Partial Yes. |
| 4 Search | Yes | Add trial registries, reference-list checks, expert consultation and grey literature where relevant, and update the search so it falls within 24 months of completion. |
| 7 Excluded studies | Yes | Publish a supplementary table of full-text exclusions with one reason per study. |
| 9 Risk of bias | Yes | Re-assess with a tool that covers the domains the item names for randomised trials or NRSI. |
| 11 Meta-analysis methods | Usually | Justify pooling, use a weighted method, investigate heterogeneity, and report trials and NRSI separately. |
| 13 RoB in interpretation | Yes | Rewrite the discussion so that conclusions are tied to the risk of bias in the underlying studies. |
| 15 Publication bias | Depends | Add a funnel plot or statistical test where at least 10 studies exist, and discuss the likelihood and magnitude of its impact. |
Teams planning a new review should read our notes on common mistakes in meta-analysis alongside this table, because items 11, 12 and 15 overlap with the errors listed there. Researchers building an umbrella review on top of AMSTAR 2 ratings will also find the umbrella review walkthrough useful.
Common Mistakes When Appraising Reviews With AMSTAR 2
- Adding up a score. Shea and colleagues strongly recommend against it, and a review with 14 Yes ratings can still be critically low.
- Giving the benefit of the doubt. Unreported means No.
- Treating a stated protocol as a registered one. A statement in the report earns Partial Yes at best, and Yes needs independent verification.
- Pooling randomised and non-randomised estimates in item 11. Separate summary estimates are required when both designs are included.
- Using the tool on out-of-scope designs. Network meta-analyses and scoping reviews need other instruments.
- Skipping the protocol comparison. Unexplained deviations between registry entry and report justify a downgrade.
Free Tools for Your Review Workflow
Free Research Tools from Editverse
- GraphPro: build publication-ready charts, 25 types, with built-in t-test, ANOVA, correlation, and chi-square p-values. Free, no signup.
- StatPro: free statistical calculators and a test-selector tool covering group comparisons, categorical outcomes, time-to-event analysis, and diagnostic accuracy.
- Flowchart Pro: generate PRISMA 2020, PRISMA-ScR, CONSORT, STROBE, STARD, MOOSE, and ARRIVE flow diagrams directly in your browser.
AMSTAR 2 Frequently Asked Questions
How many items and critical domains does AMSTAR 2 have?
AMSTAR 2 has 16 items. Seven of them (items 2, 4, 7, 9, 11, 13 and 15) are designated critical domains.
Does AMSTAR 2 produce a numerical score?
No. Item ratings are never combined into a score under the paper’s recommendation, and one of four confidence levels is assigned from the count of critical flaws and non-critical weaknesses.
Can AMSTAR 2 appraise reviews of observational studies?
Yes. The 2017 update covers systematic reviews that include randomised studies, non-randomised studies of interventions or both, and items 9 and 11 have separate criteria for each design.
What is the difference between AMSTAR 2 and ROBIS?
AMSTAR 2 assesses methodological quality of reviews of healthcare interventions through 16 items, while ROBIS assesses risk of bias in reviews through a three-phase process. Overlap between the two is considerable, and some teams use both.
Professional Support for Systematic Reviews That Hold Up Under AMSTAR 2
Most critical domains are set during protocol, search and analysis, so repair after submission is costly. Editverse builds systematic reviews with the critical items in mind, including a PROSPERO-ready protocol for item 2, documented search strategies for item 4, risk of bias figures for item 9 and funnel plots with statistical tests for item 15. Review support follows PRISMA 2020, and Editverse aligns its work with COPE, ICMJE and GPP 2022.
Small
$1,800
Complete systematic review plus meta-analysis for up to 15 eligible studies. PROSPERO-ready protocol, PRISMA flow diagram, risk of bias figures, forest and funnel plots. 10 to 14 weeks from protocol approval.
Analysis only
$1,000
Standalone meta-analysis from your extracted data. Heterogeneity, funnel plot with bias tests, sensitivity analysis and a manuscript-ready results section. 2 to 3 weeks from data receipt.
For protocol design, search strategy and full manuscript support beyond these two packages, see the Editverse systematic review writing service.
Conclusion
AMSTAR 2 rewards a disciplined routine. Rate each of the 16 items against the published criteria, count Nos on the seven critical domains and on the other nine items as separate tallies, and read the verdict from Box 2 without adding anything up. Our invented back-pain review shows how a single missing list pulled a strong review down to Low and how quickly one supplementary table would restore it.
Whether you are appraising other people’s reviews or auditing your own, document your conventions, use two raters and keep the protocol next to the report. Those habits turn a checklist into a defensible judgement.
References
- Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. doi:10.1136/bmj.j4008. PMID 28935701. Full text and supplementary appendices at PMC5833365.
- Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. doi:10.1186/1471-2288-7-10. PMID 17302989.
- Perry R, Whitmarsh A, Leach V, Davies P. A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2. Syst Rev. 2021;10:273. doi:10.1186/s13643-021-01819-x. PMID 34696810.
- Whiting P, Savovic J, Higgins JPT, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-234. doi:10.1016/j.jclinepi.2015.06.005. PMID 26092286.
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71. PMID 33782057.
- Galante J, Friedrich C, Dawson AF, Modrego-Alarcón M, Gebbing P, Delgado-Suárez I, et al. Mindfulness-based programmes for mental health promotion in adults in nonclinical settings: a systematic review and meta-analysis of randomised controlled trials. PLoS Med. 2021;18(1):e1003481. doi:10.1371/journal.pmed.1003481. PMID 33428616.