Global Spine J 2016; 06(08): 842
DOI: 10.1055/s-0036-1597140
Evidence Assessment
Georg Thieme Verlag KG Stuttgart · New York

EBSJ Evidence Assessment

Further Information

Publication History

Publication Date:
15 November 2016 (online)

 

EBSJ Evidence Assessment: Definition of Risk of Bias (RoB)

Articles on treatment

Studies of therapy

Risk of bias description

Study design

Criteria

Low risk

Study adheres to commonly held tenets of high-quality design, execution, and avoidance of bias

Good quality RCT

• Random sequence generation

• Allocation concealment

• Intent-to-treat analysis

• Blind or independent assessment for author's primary important outcomes[a]

• Co-interventions applied equally

• F/U rate of 80%+ and <10% difference in follow-up between groups

• Controlling for possible confounding[b]

Moderately low risk

Study has potential for some bias; study does not meet all criteria for a good quality RCT, but deficiencies not likely to invalidate results or introduce significant bias

Moderate or poor quality RCT

• Violation of one or two criteria for good quality RCT

Good quality cohort

• Blind or independent assessment in a prospective study, or use of reliable data[c] in a retrospective study

• Co-interventions applied equally

• F/U rate of 80%+ and <10% difference in follow-up between groups

• Controlling for possible confounding[b]

Moderately high risk

Study has significant flaws in design and/or execution that increase potential for bias that may invalidate study results

Poor quality RCT

• Violation of three or more of the criteria for a good quality RCT

Moderate or poor quality cohort

• Violation of any of the criteria for good quality cohort

Case-control

• Any case-control design

High risk

Study has significant potential for bias; lack of comparison group precludes direct assessment of important outcomes

Case series

• Any case series design

a Outcome assessment is independent of health care personnel, investigator, or patient judgment.


b Authors must provide a description of robust baseline characteristics, and control for those that are unequally distributed between treatment groups. RCTs get credit if there is a similar distribution of baseline characteristics between groups but must also control for confounding if distribution is not similar.


c Reliable data are data such as mortality or reoperation.


Articles on prognosis or risk

Studies of prognosis

Risk of bias description

Study design

Criteria

Low risk

Study adheres to commonly held tenets of high quality design, execution, and avoidance of bias

Good quality cohort[a]

• Prospective design

• Patients at similar point in the course of their disease or treatment

• F/U rate of ≥ 80%[b]

• Patients followed long enough for outcomes to occur

• Accounting for other prognostic factors[c]

• Objective and unbiased outcome measure used[d]

Moderately low risk

Study has potential for some bias; does not meet all criteria for good quality cohort, but deficiencies not likely to invalidate results or introduce significant bias

Moderate quality cohort

• Prospective design, with violation of one of the other criteria for good quality cohort study

• Retrospective design, meeting all the rest of the criteria in good quality cohort

Moderately high risk

Study has flaws in design and/or execution that increase potential for bias that may invalidate study results

Poor quality cohort

Good quality case-control or cross-sectional study

• Prospective design with violation of two or more criteria for good quality cohort, or

• Retrospective design with violation of one or more criteria for good quality cohort

• A good case-control study[e]

• A good cross-sectional study[f]

High risk

Study has significant potential for bias; does not include design features geared toward minimizing bias and/or does not have a comparison group

Poor quality case-control or cross-sectional

Case series[g]

• Other than a good case-control study

• Other than a good cross-sectional study

• Any case series[g] design

a Cohort studies follow individuals with the exposure of interest over time and monitor for occurrence of the outcome of interest.


b Applies to cohort studies only.


c Authors must consider other factors that might influence patient outcomes and should control for them if appropriate.


d Examples of objective outcomes: weight loss, change in blood pressure, speed of walking, reoperation, death, etc.; examples of subjective outcomes are patient reported outcomes and self-reported pain.


e A good case-control study must have the all of the following: all incident cases from the defined population over a specified time period, controls that represent the population from which the cases come, exposure that precedes an outcome of interest, and accounting for other prognostic factors.


f A good cross-sectional study must have all of the following: a representative sample of the population of interest, an exposure that precedes an outcome of interest (e.g., sex, genetic factor), an accounting for other prognostic factors, and for surveys, at least an 80% return rate.


g A case-series design for prognosis is one where all the patients in the study have the exposure of interest. Since all the patients have the exposure, risks of an outcome can be calculated only for those with the exposure, but cannot be compared with those who do not have the exposure. For example, a case-series evaluating the effect of smoking on spine fusion that only recruits patients who smoke can simply provide the risk of patients who smoke that result in pseudarthrosis but cannot compare this risk to those that do not smoke.



#

Determination of Overall Strength (Quality) of Evidence (SoE)

After individual article evaluation, the overall quality of the body of evidence with respect to each outcome is determined based on precepts outlined by the Grades of Recommendation Assessment, Development and Evaluation (GRADE) Working Group and recommendations made by the Agency for Healthcare Research and Quality (AHRQ). Qualitative analysis is performed considering the AHRQ required and additional domains. The table below provides an outline of the methods used to determine the final SoE.

The following four possible levels and their definition will be reported:

  • High: High confidence that the evidence reflects the true effect. Further research is very unlikely to change our confidence in the estimate of effect.

  • Moderate: Moderate confidence that the evidence reflects the true effect. Further research may change our confidence in the estimate of effect and may change the estimate.

  • Low: Low confidence that the evidence reflects the true effect. Further research is likely to change the confidence in the estimate of effect and likely to change the estimate.

  • Insufficient: Evidence either is unavailable or does not permit a conclusion.

All AHRQ “required” and “additional” domains[a] are assessed. Only those that influence the baseline grade are listed in table.

Baseline strength: Risk of bias (including control of confounding) is accounted for in the individual article evaluations. High = majority of articles RCTs; L = majority of articles cohort (observational) studies

Downgrade: Inconsistency[b] of results (1 or 2); Indirectness of evidence (1 or 2); Imprecision of effect estimates (1 or 2); Sub-group analyses not stated a priori and no test for interaction (2)

Upgrade (observational studies): Large magnitude of effect (1 or 2); Dose response gradient (1) if no downgrade for domains listed above

Outcome

Strength of evidence

Conclusions and comments

Baseline

Downgrade

Upgrade

Outcome

High

Summary of findings

High

RCTs studies

No

Consistent, direct, and precise estimates

No

Outcome

Moderate

Summary of findings

Low

Cohort studies

No

Consistent, direct, and precise estimates

Yes

Large effect

Outcome

Low

Summary of findings

High

RCTs studies

Yes (2)

Inconsistent

Indirect

No

a Required domains: risk of bias, consistency, directness, precision. Plausible confounding that would decrease observed effect is accounted for in our baseline risk of bias assessment through individual article evaluation. Additional domains: dose-response, strength of association, publication bias.


b Single study = “consistency unknown.”


Strength of Evidence for Existing Systematic Reviews

Level of evidence ratings for Cochrane reviews and other systematic reviews are assigned a baseline score of High if RCTs were used, Low if observational studies were used. The rating can be upgraded or downgraded based on adherence to the core criteria for methods, qualitative, and quantitative analyses for systematic reviews (there is a reference/evaluation table for this).


#
#

Definitions of the Different Levels of Evidence for Reliability Studies

Level

Study type

Criteria

1

Good quality study

• Broad spectrum of persons with the expected condition

• Adequate description of methods for replication

• Blinded performance of tests, measurements, or interpretation

• Second test/interpretation performed independently of the first

2

Moderate quality

• Violation of any one of the criteria for a good quality study

3

Poor quality study

• Violation of any two of the criteria

4

Very poor quality study

• Violation of all three of the criteria


#
#