Download a PDF copy of the Weight of Evidence

Weight of Evidence

ONS PEP (Putting Evidence into Practice) WEIGHT OF EVIDENCE
CLASSIFICATION SCHEMA
Decision Rules for Summative Evaluation of a Body of Evidence

S.A. Mitchell, MScN, CRNP, AOCN^® and C.R. Friese, PhD, MS, RN, AOCN^®
on behalf of the ONS Oncology Nursing Interventions for Patient Outcomes Project Team

Background

The classification schema outlined below was developed to assist in evaluating a collective body of evidence about a health intervention for the purpose of informing decisions on implementation. Based on the work of Ciliska, Cullum and Marks (2001), Hadorn, Baker, Hodges and Hicks (1996), Rutledge, DePalma, & Cunningham (2004), and Ropka and Spencer-Cisak (2001), the schema was intended for application to bodies of existing research-based knowledge on health interventions for patients with cancer. The schema itself does not seek to guide the decision process in relation to an intervention for an individual patient. Such decisions should be made within the interdisciplinary team, and based on individual patient characteristics, values, and preferences, a consideration of potential harms as well as benefits, and an assessment of the feasibility of implementing the intervention within the specific care setting.

A schema developed for appraising evaluative research should not be used to remove interventions from further consideration because of inadequate evidence about intervention effectiveness. Criterion-based evaluation of evidence is valid only where a significant body of high quality evidence is available. It is critical to avoid interpreting insufficient evidence on the one hand, or poor-quality evidence on the other, as meaning that an intervention is unimportant or ineffective. Insufficient evidence or a lack of evidence simply means that evaluative research of an intervention has not been done at the level necessary to make conclusions with confidence that an intervention produces a specific outcome/patient benefit. The lack of evidence on an intervention, or the availability only of poor-quality evidence, may indicate a gap in knowledge and a need for additional research. The schema can therefore also be used to highlight research gaps, and to identify the types of research that could address those gaps.

Panels of advanced practice nurses, staff nurses, and doctorally-prepared nurse researchers reviewed the literature base in the identified outcome areas. Professional health services librarians assisted in the conduct of the literature searches. Based on their analysis, the panels then formulated a judgment about the body of evidence related to the intervention under consideration. Three major components were considered by the panels in classifying the collective evidence into one of six Weight of Evidence categories:

•  Quality of the data, with more weight assigned to levels of evidence higher in the PRISM categorization (such as randomized trials, and meta-analyses)

•  Magnitude of the outcome (eg. effect size or minimal clinically important difference)

•  Concurrence among the evidence (based on the premise that an investigator has less confidence in findings in which the lines of evidence contradict one another)

Recommended for Practice

Interventions for which effectiveness has been demonstrated by strong evidence from rigorously-designed studies, meta-analyses, or systematic reviews, and for which expectation of harms is small compared with the benefits.

•  Supportive evidence from at least two well-conducted randomized controlled trials that were performed at more than one institutional site, and that included a sample size of at least 100 participants

•  Evidence from a meta-analysis or systematic review of research studies that incorporated quality ratings in the analysis, and included a total of 100 patients or more in its estimate of effect size and confidence intervals

•  Recommendations from a panel of experts, that derive from an explicit literature search strategy, and include thorough analysis, quality rating, and synthesis of the evidence

Likely to Be Effective

Interventions for which the evidence is less well established than for those listed under ‘recommended for practice'.

•  Supportive evidence from a single well conducted randomized controlled trial that included fewer than 100 patients or was conducted at one or more institutions

•  Evidence from a meta-analysis or systematic review that incorporated quality ratings in the analysis and included fewer than 100 patients, or had no estimates of effect size and confidence intervals

•  Evidence from a synthetic review of randomized trials that incorporated quality ratings in the analysis

•  Guidelines developed largely by consensus/expert opinion rather than primarily based on the evidence and published by a panel of experts, that are not supported by synthesis and quality rating of the evidence

Benefits Balanced with Harms

Interventions for which clinicians and patients should weigh up the beneficial and harmful effects according to individual circumstances and priorities.

•  Supportive evidence from one or more randomized trials, meta-analyses or systematic reviews, but where the intervention may be associated, in certain patient populations, with adverse effects that produce or potentially produce mortality, significant morbidity, functional disability, hospitalization or excess length of stay

Effectiveness Not Established

Interventions for which there are currently insufficient data or data of inadequate quality.

•  Supportive evidence from a well conducted case control study

•  Supportive evidence from a poorly controlled or uncontrolled study

•  Evidence from randomized clinical trials with one or more major or three or more minor methodological flaws that could invalidate the results

•  Evidence from non-experimental studies with high potential for bias (such as case series with comparison to historical controls)-Evidence from case series or case reports

•  Conflicting evidence, but where the preponderance of the evidence is in support of the recommendation or meta-analysis showing a trend that did not reach statistical significance

Effectiveness Unlikely

Interventions for which lack of effectiveness is less well established than for those listed under ‘not recommended for practice'.

•  Evidence from a single well conducted randomized trial with at least 100 participants or conducted at more than one site and which showed no benefit for the intervention

•  Evidence from a well conducted case control study, a poorly controlled or uncontrolled study, a randomized trial with major methodologic flaws, or an observational study (eg. case series with historical controls) that showed no benefit and a prominent and unacceptable pattern of adverse events and serious toxicities (CTCAE Grade III/IV)

Not Recommended for Practice

Interventions for which ineffectiveness or harmfulness has been demonstrated by clear evidence, or the cost or burden necessary for the intervention exceeds anticipated benefit.

•  Evidence from two or more well conducted randomized trials with at least 100 participants or conducted at more than one site and which showed no benefit for the intervention, and excessive costs or burden expected

•  Evidence from a single well conducted trial that showed a prominent and unacceptable pattern of adverse events and serious toxicities (CTCAE Grade III/IV)

•  Evidence from a meta-analysis or systematic review of research studies that incorporated quality ratings in the analysis, included a total of 100 patients or more in its estimate of effect size and confidence intervals with demonstrated lack of benefit or prominent and unacceptable toxicities

•  Intervention discouraged from use by a panel of experts in the related subject, after conducting a systematic examination, quality rating and synthesis of the available evidence

References

Ciliska, D., Cullum, N. & Marks, N. (2001). Evaluation of systematic reviews of treatment or prevention interventions. Evidence Based Nursing , 4, 100-104.

Hadorn, D.C., Baker, J.S., Hodges, J.S. & Hicks, N. (1996). Rating the quality of evidence for clinical practice guidelines. Journal of Clinical Epidemiology, 49 , 749-754.

Ropka, M.E. & Spencer-Cisek, P. (2001). PRISM: Priority Symptom Management Project. Phase I—Assessment. Oncology Nursing Forum, 28 , 1585-1594.

Rutledge, D., DePalma, J., & Cunningham, M. (2004). A process model for evidence-based literature syntheses. Oncology Nursing Forum, 31 (3), 543-550.