Decision Rules for Summative Evaluation of a Body of Evidence
S.A. Mitchell, MScN, CRNP, AOCN®, and C.R. Friese, PhD, MS, RN, AOCN®, on behalf of the ONS Oncology Nursing Interventions for Patient Outcomes Project Team
The classification schema outlined below was developed to assist in evaluating a collective body of evidence about a health intervention for the purpose of informing decisions on implementation. Based on the work of Ciliska, Cullum, and Marks (2001), Hadorn, Baker, Hodges and Hicks (1996), Rutledge, DePalma, and Cunningham (2004), and Ropka and Spencer-Cisak (2001), the schema was intended for application to bodies of existing research-based knowledge on health interventions for patients with cancer. The schema itself does not seek to guide the decision process in relation to an intervention for an individual patient. Such decisions should be made within the interdisciplinary team, and based on individual patient characteristics, values, and preferences, a consideration of potential harms as well as benefits, and an assessment of the feasibility of implementing the intervention within the specific care setting.
A schema developed for appraising evaluative research should not be used to remove interventions from further consideration because of inadequate evidence about intervention effectiveness. Criterion-based evaluation of evidence is valid only where a significant body of high quality evidence is available. It is critical to avoid interpreting insufficient evidence on the one hand, or poor-quality evidence on the other, as meaning that an intervention is unimportant or ineffective. Insufficient evidence or a lack of evidence simply means that evaluative research of an intervention has not been done at the level necessary to make conclusions with confidence that an intervention produces a specific outcome/patient benefit. The lack of evidence on an intervention, or the availability only of poor-quality evidence, may indicate a gap in knowledge and a need for additional research. The schema can therefore also be used to highlight research gaps, and to identify the types of research that could address those gaps.
Panels of advanced practice nurses, staff nurses, and doctorally-prepared nurse researchers reviewed the literature base in the identified outcome areas. Professional health services librarians assisted in the conduct of the literature searches. Based on their analysis, the panels then formulated a judgment about the body of evidence related to the intervention under consideration. Three major components were considered by the panels in classifying the collective evidence into one of six weight of evidence categories:
- Quality of the data, with more weight assigned to levels of evidence higher in the PRISM categorization (e.g., randomized trials, meta-analyses)
- Magnitude of the outcome (e.g., effect size, minimal clinically important difference)
- Concurrence among the evidence (based on the premise that an investigator has less confidence in findings in which the lines of evidence contradict one another)