Published
The PROBAST tool (Prediction model Risk Of Bias ASsessment Tool) is the standard instrument for judging whether a published prediction model can be trusted and whether it fits the patients you care about. Wolff and colleagues published it in Annals of Internal Medicine in 2019 [1], and a companion explanation and elaboration paper sets out the reasoning behind each question [2]. This post explains how the PROBAST tool is structured, then applies all 20 signalling questions, one row at a time, to a real open-access model, the QUIDS preterm birth model published in PLOS Medicine [3]. Every row points to the section of the paper that supports it, and every row that rests on judgement is labelled.
In Simple Words
A prediction model turns a few facts about a patient into a risk. The PROBAST tool asks 20 targeted questions about how the model was built and tested, grouped into four domains (participants, predictors, outcome and analysis), to decide whether the reported accuracy is likely to be inflated. It then asks a separate question, whether the model suits your own patients. A model can be carefully built and still be the wrong model for your clinic.
What the PROBAST tool assesses
The PROBAST tool was written for systematic reviews of prediction models, and it works equally well for a clinician or editor appraising a single paper. It covers diagnostic models (is the condition present now) and prognostic models (will an event occur later), and it applies to studies that develop a model, validate one, or update one [1,2]. It produces two separate verdicts. The first is risk of bias, defined as shortcomings in design, conduct or analysis that distort the estimates of how well the model predicts. The second is concern about applicability, which asks whether the population, predictors and outcome of the study match your question.
Bias matters here because of optimism. A model that is flexible enough to fit its development data too closely looks excellent in that data and performs worse in new patients. De Jong and colleagues describe this as a systematic gap between the performance claimed at development and the performance realised at validation [4]. Their meta-review of 50 systematic reviews that used the PROBAST tool covered 1,510 studies and 2,104 models, and all domains showed an unclear or high risk of bias [4]. Appraisal is therefore a routine step, not an exception.
Each signalling question takes one of five answers, namely yes, probably yes, probably no, no, or no information. The questions feed a domain rating of low, high or unclear. In the PROBAST guidance a domain is low when every question is answered yes or probably yes, it is potentially high when at least one question is answered no or probably no, and it is unclear when information is missing and nothing is answered no [1,2]. The ratings are not summed into a score, and the final call stays with the reviewer.
Where the PROBAST tool sits among related tools
The PROBAST tool judges methods, not reporting. Reporting completeness belongs to TRIPOD, which our guide to prediction model reporting with TRIPOD+AI covers. Data extraction for reviews of prediction models follows the CHARMS checklist [5], and sample size for model development follows the criteria of Riley and colleagues [6], which replace the older fixed events-per-variable rule. For other study designs the matching tools are RoB 2 for trials (see our RoB 2 walkthrough), ROBINS-I for non-randomised interventions (see the ROBINS-I guide) and QUADAS-2 for diagnostic accuracy studies (see QUADAS-2 in practice). Our risk of bias assessment guide covers the trial tool in depth.
PROBAST+AI, the 2025 update
In 2025 Moons and colleagues published PROBAST+AI in the BMJ [7]. It has two parts. The model development part uses 16 signalling questions to assess quality and applicability, and the model evaluation part uses 18 signalling questions to assess risk of bias and applicability. Both parts keep four domains, now named participants and data sources, predictors, outcome and analysis, and applicability is still rated for the first three. The authors state that PROBAST+AI may replace the original tool [7]. This post applies the original 2019 tool, because its 20 questions are fixed and are the ones used in the existing review literature [4]. If your protocol, supervisor or target journal names PROBAST+AI, use the 2025 question set and keep the same evidence-pointer habit shown below.
The PROBAST tool at a glance, 4 domains and 20 signalling questions
The table paraphrases the 20 questions in plain words and says where in a paper the evidence usually sits. Some analysis questions, for example 4.5 and 4.9, apply to model development only, and several others are adapted to the part you are rating.
| Question | What it asks, in plain words | Where to look |
|---|---|---|
| 1.1 | Is the data source suitable, such as a cohort or trial, rather than a design that distorts absolute risk? | Abstract, study design paragraph |
| 1.2 | Are the inclusion and exclusion criteria appropriate for the intended population? | Eligibility text, participant flow diagram |
| 2.1 | Are predictors defined and measured in the same way for every participant? | Predictor definitions, measurement protocols |
| 2.2 | Were predictors measured without knowledge of the outcome? | Study design, blinding statements |
| 2.3 | Will every predictor be available when the model is meant to be used? | Predictor list against the intended use |
| 3.1 | Was the outcome determined by a suitable method? | Outcome definition and data source |
| 3.2 | Was a prespecified or standard outcome definition used? | Protocol, registration, methods |
| 3.3 | Is the outcome defined without using any predictor? | Outcome definition against predictor list |
| 3.4 | Was the outcome defined and determined in the same way for all participants? | Multicentre procedures, supplement |
| 3.5 | Was the outcome determined without knowledge of predictor values? | Blinding, assessor description |
| 3.6 | Is the time between predictor measurement and outcome appropriate? | Follow-up window, intended use |
| 4.1 | Are there enough outcome events for the task? | Events and candidate predictors, sample size section |
| 4.2 | Were continuous and categorical predictors handled appropriately? | Statistical methods, transformations |
| 4.3 | Were all enrolled participants included in the analysis? | Flow diagram, analysis population |
| 4.4 | Was missing data handled appropriately? | Missing data paragraph, Table 1 |
| 4.5 | Was predictor selection free of univariable screening? (development only) | Model building paragraph |
| 4.6 | Were censoring, competing risks and sampling of controls handled where relevant? | Model type, follow-up description |
| 4.7 | Were suitable performance measures reported, including calibration and discrimination? | Performance section, calibration plot |
| 4.8 | Were overfitting, underfitting and optimism handled? | Internal validation, shrinkage |
| 4.9 | Do the final predictors and weights match the reported multivariable analysis? (development only) | Model table, equation |
Applicability uses three further judgements rather than signalling questions. Participants and setting, predictors, and outcome are each rated as low, high or unclear concern against your review question. Applicability is the reason the same paper can be a good fit for one review and a poor fit for another.
How to run a PROBAST tool appraisal in five steps
- Write the review question. State the population, predictors, outcome, timing, setting and intended use of the model before you open any paper. Applicability cannot be rated without it.
- Classify the paper. Mark each paper as development, validation or both, and rate the parts separately. A paper that does both can earn different ratings for each part.
- Answer the signalling questions with an evidence pointer. Record the section, table or figure that supports each answer, and quote sparingly.
- Rate each domain. Apply the rule that any no or probably no makes a domain potentially high, then decide whether the reason is a real source of distortion.
- Give the overall judgement. The paper is low risk of bias only when all four domains are low, and the same logic applies to applicability.
Use two reviewers who work independently and then reconcile. A 2023 meta-review of PROBAST use reported stable trends in bias over time and low inter-rater agreement [8], which is the practical reason to agree the review question, the events threshold and the handling of no information answers before screening begins.
The open-access paper we appraise
The paper is Stock and colleagues, Development and validation of a risk prediction model of preterm birth for women with preterm labour symptoms (the QUIDS study), published in PLOS Medicine in 2021 [3]. It is a strong teaching case for three reasons. It contains both a development part and an external validation part, it reports its methods in enough detail to answer most signalling questions from the main text, and its authors state the weak points of their own design, which makes the judgement calls visible. The licence is CC BY 4.0, confirmed from the Crossref version-of-record licence field, the Europe PMC licence field and the JATS permissions statement in the full text. The paper is indexed in PubMed as PMID 34228732 (PMCID PMC8259998).
The model estimates the risk of spontaneous preterm birth within 7 days in women with signs and symptoms of preterm labour. It was developed in an individual participant data meta-analysis of five prospective cohort studies (1,783 women, 139 events) and validated in a new prospective cohort in 26 UK units (2,924 women, 85 events). The final model combines quantitative fetal fibronectin (qfFN) with smoking, ethnicity, nulliparity and multiple pregnancy.
| Review question element | Wording used for this appraisal |
|---|---|
| Population and setting | Pregnant women at 22+0 to 34+6 weeks with signs and symptoms of preterm labour and intact membranes, assessed in hospital obstetric units that have a quantitative fFN analyser |
| Predictors | Quantitative fFN plus routine clinical factors available at the first assessment |
| Outcome and timing | Spontaneous preterm birth within 7 days of the test |
| Intended use | Support decisions about admission and treatment in symptomatic women |
These are our choices for the exercise, not statements from the paper. A reviewer with a different question, for example a review of models for use where no analyser exists, would rate applicability differently. The appraisal reads the main article only, including its tables and figures. The supplementary S1 Text tables were not reviewed, so rows that depend on them say so.
Applying the PROBAST tool to QUIDS, domains 1 to 3
Domain 1, participants
| Q | Answer | Evidence and pointer | Label |
|---|---|---|---|
| 1.1 | Yes | Development used “an individual participant data (IPD) meta-analysis of 5 European prospective cohort studies” (Stock et al., Abstract). Validation used a prospective cohort in 26 UK consultant-led units (Methods, External validation). Prospective cohorts preserve absolute risk. | |
| 1.2 | Probably yes | Eligibility is defined by gestation, symptoms, intact membranes and no contraindication to fFN testing (Methods, Model outcome, target population, and candidate predictors). Four early datasets “used ELISA to determine the concentration of fFN and were excluded” (Methods, Model development), and 27 of 2,968 women were ineligible at speculum examination (Results, External validation; Fig 1). Symptoms were defined by each investigator. | Judgement call |
Domain 1 rating, low risk of bias. Applicability, low concern, because the stated population matches the review question.
Domain 2, predictors
| Q | Answer | Evidence and pointer | Label |
|---|---|---|---|
| 2.1 | Probably yes | Development studies used “quantitative fFN results determined by 10Q rapid fFN analyser system” (Methods, Model development). Validation used an adapted analyser at every site with quality control (Methods, External validation; quality control detail sits in S1 Text, not reviewed). How smoking and ethnicity were harmonised across the five cohorts is not described in the main text. | Judgement call |
| 2.2 | Yes | The cohorts were prospective, so the outcome was unknown at measurement. In validation, “The quantitative fFN result was masked from clinicians and stored as a 3-letter code.” (Methods, External validation). | |
| 2.3 | Yes | “All included candidate predictors are available around the time of initial assessment for signs and symptoms of preterm birth.” (Methods, Model outcome, target population, and candidate predictors). |
Domain 2 rating, low risk of bias. Applicability, low concern for units with an analyser. The Introduction notes that quantitative fFN lacks US FDA approval and that NICE found the evidence insufficient to recommend it, so a review aimed at settings without an analyser should raise this to high concern.
Domain 3, outcome
| Q | Answer | Evidence and pointer | Label |
|---|---|---|---|
| 3.1 | Probably yes | Birth date and whether the birth was spontaneous were taken from records. “All other data were collected from the participant records and recorded on the study database.” (Methods, External validation). Timing of birth is objective, but the main text does not describe independent adjudication of spontaneous against medically indicated birth. | Judgement call |
| 3.2 | Yes | “The primary outcome was spontaneous preterm birth within 7 days of quantitative fFN test” (Methods, Model outcome, target population, and candidate predictors), set in published protocols (Methods, Study oversight). Analyses with all preterm births were flagged as not prespecified (Methods, Additional models and subgroup analyses). | |
| 3.3 | Yes | The outcome is a birth event inside a time window. None of the five final predictors forms part of that definition (Results, Model development and internal validation). | |
| 3.4 | Probably yes | One definition was applied to all cohorts, and validation sites were described with different levels of neonatal care (Methods, External validation). The outcome does not depend on local neonatal practice. | |
| 3.5 | Probably yes | The qfFN value was masked, so it could not steer outcome ascertainment. Clinicians did see the qualitative fFN result, and tocolysis could delay birth. The authors state “Tocolysis was included in sensitivity analysis to explore any potential treatment effect on delaying birth.” (Methods, Model outcome, target population, and candidate predictors). | Judgement call |
| 3.6 | Yes | “The choice of this outcome was determined by our previous qualitative study of women and clinicians” (Methods, same subsection). The Introduction ties the benefit of antenatal corticosteroids to administration within 7 days of birth. |
Domain 3 rating, low risk of bias. Applicability, low concern, because the 7-day outcome matches the review question.
Applying the PROBAST tool to QUIDS, domain 4 analysis
Domain 4 is where most prediction papers lose their rating, and it is the only domain where the development and validation parts of QUIDS diverge. The two parts are marked Dev and Val. Two small calculations from the paper’s own numbers sit behind rows 4.1 and 4.9.
- Events per candidate predictor. Development had 139 events and 10 candidate predictors, so 139 divided by 10 is 13.9, which the authors round to about 14 (Methods, Sample size calculations). That sits inside the older 10 to 20 range [4] but below the 20 events per candidate predictor often applied when using the PROBAST tool. We accept probably yes because the authors state the data met modern minimum sample size criteria, so a stricter reviewer could answer probably no and rate the development analysis domain unclear or high.
- Validation events. There were 85 events in 2,924 women (2.9 percent), below the roughly 100 events the authors themselves cite as the minimum for validation (Results, External validation; Methods, Sample size calculations).
- Shrinkage check for 4.9. Dividing each optimism-adjusted coefficient in Table 2 by its original gives 0.943 to 0.946 across all five predictors, consistent with the uniform shrinkage factor of 0.94 that the paper reports.
| Q | Answer | Evidence and pointer | Label |
|---|---|---|---|
| 4.1 | Dev probably yes, Val probably no | Dev ratio 13.9 events per candidate predictor, see above. Val “A potential limitation is the number of events in our external validation (85).” (Discussion). The authors also write that “a minimum of around 100 events and 100 nonevents were needed for risk prediction model validation” (Methods, Sample size calculations). | Judgement call (both parts) |
| 4.2 | Yes | “Continuous predictors were analysed on their continuous scale.” Nonlinearity was handled with fractional polynomials, giving a square root form for qfFN (Methods, Model development; Results). Ethnicity was collapsed to White and not White because of sparse groups (Discussion). | |
| 4.3 | Probably yes | In validation 27 women were ineligible at speculum examination and 17 tests were invalid, so 2,924 of 2,968 were analysed, a loss of 44 women or 1.5 percent (Results, External validation; Fig 1). The exclusions follow test eligibility, not the outcome. | Judgement call |
| 4.4 | Yes | “Multiple imputation of missing values by chained equations was performed within each original study separately” with 60 imputations in development and 8 in validation (Methods, Model development; External validation). A complete-case sensitivity analysis gave similar performance (Results, External validation). BMI was missing for 29.7 percent in development (Table 1) but is not in the final model. | |
| 4.5 | Yes (Dev) | Ten candidate predictors were prespecified, then reduced by backward selection inside a multivariable logistic model using Akaike’s information criterion (Methods, Model development). That is not univariable screening. | |
| 4.6 | Probably yes | Clustering by study was handled with a one-stage IPD model and a separate intercept per study, with a two-stage random-effects check of predictor heterogeneity (Methods, Model development). The validation used a pooled intercept from a random-effects meta-analysis of the five study intercepts (Methods, External validation). The 7-day binary outcome avoids censoring problems if follow-up was complete. | Judgement call |
| 4.7 | Yes | The paper reports AUC, calibration slope, calibration-in-the-large, expected to observed ratio, a calibration plot and Nagelkerke R2 (Methods, Predictive performance measures; Table 3; Fig 2), plus net benefit (Fig 3). Validation AUC was 0.89 (95% CI 0.84 to 0.94), expected to observed ratio 0.80 before recalibration, calibration slope 1.24. | |
| 4.8 | Dev yes, Val probably yes | Bootstrap internal validation used 100 samples “replicating all steps of our model development” (Methods, Internal validation), giving a uniform shrinkage factor of 0.94, after which study-specific intercepts were re-estimated (Results, Model development and internal validation). In validation only the intercept was updated (Table 3, column B), and performance before recalibration is also reported (column A). | |
| 4.9 | Yes (Dev) | The equation in Results, QUIDS risk prediction model, uses 1.915 for the square root of qfFN plus 0.370, 0.834, minus 0.687 and minus 0.139 for the clinical factors, with intercept minus 5.002. These match the optimism-adjusted column of Table 2 and the recalibrated intercept in Table 3. |
Domain 4 rating. The development part is low risk of bias. The validation part is high risk of bias on a strict reading, driven by one item, row 4.1. Everything else in domain 4 is answered yes or probably yes.
Overall ratings and the calibration plot
Read the figure as a map of where to look next, not as a verdict on the study. The development part is low across all domains. The validation part has one red cell, and it comes from an event count of 85 against a threshold of about 100, a gap the authors disclose themselves. A reviewer could argue for an unclear rating instead, but the review protocol should record the reason either way, because the rating changes how the model is weighted in any synthesis.
The calibration plot shows why the event count matters in practice. Almost all grouped observations sit below a predicted risk of 0.03, where events are rare, and one group near a predicted risk of 0.17 shows an observed risk of about 0.22 with a wide interval. The authors report some underprediction at higher risks and a calibration slope of 1.24 after intercept recalibration (Results, External validation). With so few events in the upper groups, a single group moves the picture, which is exactly the instability that item 4.1 is designed to flag.
Row 4.7 asks for calibration and discrimination, and this figure is the calibration evidence. The paper’s net benefit analysis (Fig 3) belongs to a different question, whether the model helps decisions. If you want to run a net benefit analysis on your own model, our free decision curve analysis calculator does it from your predicted risks.
Judgement calls and where appraisers disagree
Seven rows in this appraisal rest on judgement. Naming them is more useful than pretending the tool is mechanical, and it is the practice that raises agreement between two reviewers.
- 1.2 turns on whether excluding four early ELISA datasets is a selection problem. It is method driven, so we rated it probably yes.
- 2.1 depends on harmonisation details that sit in the supplement.
- 3.1 and 3.5 concern how the outcome was classified and whether treatment could have changed it.
- 4.1 is the decisive row for validation and a judgement call for development. A reviewer who accepts 85 events because the intervals are narrow would still need to say so in the protocol.
- 4.3 and 4.6 are probably yes answers where a second reviewer might ask for more detail on loss to follow-up.
A 2023 meta-review linked PROBAST use to low inter-rater agreement [8], so a short calibration exercise on two or three papers before full appraisal pays for itself. Agree the events threshold, agree how to treat no information, and agree what counts as independent outcome adjudication.
Common mistakes with the PROBAST tool
- Using it as a reporting checklist. A study that reports poorly earns no information answers, which is different from a method that is poor. Pair the PROBAST tool with TRIPOD for reporting.
- Rating applicability without a written review question. Applicability is relative, so the question must exist first.
- Summing signalling questions into a score. The tool is built for structured judgement, and a count of yes answers hides which domain failed.
- Merging development and validation. Rate each part. QUIDS shows how a single paper can hold a low and a high rating.
- Confusing backward selection with univariable screening. Question 4.5 targets screening predictors one at a time before modelling. Backward selection inside a multivariable model is a separate issue that question 4.8 checks through optimism correction [4].
- Equating high risk of bias with a bad model. The rating says the performance estimate may be unreliable, not that the model is wrong.
- Taking author claims as evidence. A sentence that says the study followed TRIPOD is not an answer to any signalling question. Point to the table or paragraph that shows it.
Reporting PROBAST results in a systematic review
Report the review question, the version of the tool (original 2019 or PROBAST+AI), the number of reviewers and the rule for resolving disagreements. Show a traffic-light figure like Figure 1 for each included model, with the domain ratings and the reason for every high rating in a supporting table. Then use the ratings in the synthesis, for example through a sensitivity analysis that removes models with high risk of bias in the analysis domain. The flow of records into your review belongs in a PRISMA diagram, and the free Flowchart Pro below builds one in your browser.
If your review also includes trials or cohort studies of treatments, pair the PROBAST tool with the matching instruments in the RoB 2, ROBINS-I and QUADAS-2 guides above, so that every included design is appraised with the tool built for it.
Free tools for prediction model research
Free Research Tools from Editverse
- GraphPro: build publication-ready charts, 25 types, with built-in t-test, ANOVA, correlation, and chi-square p-values. Free, no signup.
- StatPro: free statistical calculators and a test-selector tool covering group comparisons, categorical outcomes, time-to-event analysis, and diagnostic accuracy.
- Flowchart Pro: generate PRISMA 2020, PRISMA-ScR, CONSORT, STROBE, STARD, MOOSE, and ARRIVE flow diagrams directly in your browser.
Professional support for risk of bias appraisal
Appraising 20 or 200 studies with two independent reviewers is slow work, and the ratings only help a review if they are consistent and documented. Editverse supports the appraisal and extraction stages of systematic reviews with fixed deliverables. Send your included studies and their designs, and ask which appraisal instrument can be used for each design before you order.
Risk of bias assessment
$350
Need every included study appraised in a form a reviewer will accept?
You send your included studies and their designs.
You get completed assessment forms, risk of bias tables, traffic light visualisations and a methods section narrative.
Turnaround standard 4 to 6 business days.
Systematic review data extraction
$500
Is the extraction sheet the bottleneck before appraisal can start?
You send your included studies and your review question.
You get a custom extraction form, dual extraction with reconciliation, and quality assessment data captured alongside study characteristics.
Turnaround standard 5 to 7 business days, depending on the number of studies.
For a full review from protocol to manuscript, see our systematic review writing service, which covers the stages the two services above feed into. Our work follows COPE, ICMJE and GPP 2022 recommendations, and institutional and country-specific ethics requirements are followed.
Conclusion
The PROBAST tool turns the vague question of whether a prediction model is any good into 20 questions with evidence attached. Applied to QUIDS, it gives a low risk of bias for model development, a high risk of bias on a strict reading for the external validation because of 85 events, and low concern about applicability for the review question stated here. The work lies in the method, not the colours. Write the review question first, rate development and validation separately, point to the section behind every answer, and name the judgement calls. If your protocol points to the 2025 PROBAST+AI instead, the habits carry over unchanged.
References
- Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51-8. doi:10.7326/M18-1376. PMID 30596875.
- Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies, explanation and elaboration. Ann Intern Med. 2019;170(1):W1-33. doi:10.7326/M18-1377. PMID 30596876.
- Stock SJ, Horne M, Bruijn M, White H, Boyd KA, Heggie R, et al. Development and validation of a risk prediction model of preterm birth for women with preterm labour symptoms (the QUIDS study): a prospective cohort study and individual participant data meta-analysis. PLoS Med. 2021;18(7):e1003686. doi:10.1371/journal.pmed.1003686. PMID 34228732. PMCID PMC8259998. CC BY 4.0.
- de Jong Y, Ramspek CL, Zoccali C, Jager KJ, Dekker FW, van Diepen M. Appraising prediction research, a guide and meta-review on bias and applicability assessment using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Nephrology (Carlton). 2021;26(12):939-47. doi:10.1111/nep.13913. PMID 34138495.
- Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies, the CHARMS checklist. PLoS Med. 2014;11(10):e1001744. doi:10.1371/journal.pmed.1001744. PMID 25314315.
- Riley RD, Ensor J, Snell KIE, Harrell FE Jr, Martin GP, Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441. doi:10.1136/bmj.m441. PMID 32188600.
- Moons KGM, Damen JAA, Kaul T, Hooft L, Andaur Navarro C, Dhiman P, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. doi:10.1136/bmj-2024-082505. PMID 40127903.
- Langenhuijsen LFS, Janse RJ, Venema E, Kent DM, van Diepen M, Dekker FW, et al. Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement. J Clin Epidemiol. 2023;159:159-73. doi:10.1016/j.jclinepi.2023.04.012. PMID 37142166.