Article

The Distress Thermometer: Cutoff Points and Clinical Use

Alexandra Cutillo

Erin O’Hea

Sharina D. Person

Darleen Lessard

Tina L. Harralson

Edwin Boudreaux

Distress Thermometer, oncology, cutoff point, psychological distress
ONF 2017, 44(3), 329-336. DOI: 10.1188/17.ONF.329-336

Purpose/Objectives: To establish an optimal cutoff point for the National Comprehensive Cancer Network’s Distress Thermometer (DT) as a screening measure to identify and address psychological distress in individuals with cancer, and to examine whether distress as measured by the DT significantly changes across the treatment trajectory.

Design: Secondary analyses of baseline data from a longitudinal parent study examining a computerized psychosocial assessment.

Setting: Three diverse comprehensive cancer centers across the United States.

Sample: 836 patients with a current or past diagnosis of cancer.

Methods: Study participants were selected from a randomized clinical trial. Patients during any stage of the cancer treatment trajectory were recruited during a chemotherapy infusion or routine oncology appointment.

Main Research Variables: The Behavioral Health Status Index and the DT were administered and compared using receiver operating characteristic analyses.

Findings: Results support a cutoff score of 3 on the DT to indicate patients with clinically elevated levels of distress. In addition, patients who received a diagnosis within the 1–4 weeks prior to the assessment indicated the highest levels of distress.

Conclusions: Providers may wish to use a cutoff point of 3 to most efficiently identify distress in a large, diverse population of patients with cancer. In addition, results indicate that patients may experience a heightened state of distress within 1–4 weeks postdiagnosis compared to other stages of coping with cancer.

Implications for Nursing: Using a brief measure of distress can help streamline the process of screening for psychosocial distress.

Jump to a section

    Psychological distress as a consequence of cancer care is related to diagnoses of anxiety, depression, adjustment disorders, and decreased quality of life (Mitchell et al., 2011). Despite estimates that 24%–50% of patients with cancer exhibit symptoms of distress, and can experience the aforementioned effects, psychological symptoms are not consistently addressed by all care teams (Carlson et al., 2004; Holland & Bultz, 2007; Jacobson & Ransom, 2007; Mitchell, Vahabzadeh, & Magruder, 2011; van Scheppingen et al., 2011). Even in patients exhibiting high levels of distress, rates of referral and access to psychosocial services tend to be low (Carlson, Waller, & Mitchell, 2012; Ellis et al., 2009; Verdonck-de Leeuw et al., 2009; Zebrack et al., 2015). Whether from of a lack of education regarding the use of psychosocial support or stigma regarding mental health care, highly distressed patients may not even express interest in, use, or follow up with a variety of psychosocial services (Roth et al., 1998; Tuinman, Gazendam-Donofrio, & Hoekstra-Weebers, 2008; Waller, Williams, Groff, Bultz, & Carlson, 2011). This discrepancy between high distress and low engagement in therapeutic interventions is problematic and warrants further investigation.

    Brief screening instruments have been developed to serve as economical and efficient ways to identify potential psychosocial difficulties. Early diagnosis and treatment of distress has the potential to reduce emotional suffering, the severity of physical symptoms, and excessive use of health services (Carlson & Bultz, 2004; Carlson, Groff, Maciejewski, & Bultz, 2010; Faller et al., 2013; Holland & Bultz, 2007; Jacobson & Ransom, 2007; Mehnert & Koch, 2007). Brief screening measures can also improve patient–provider communication and can promote psychosocial referrals (Carlson et al., 2012). In the past decade, efforts have been made to increase the use of psychosocial assessment and subsequent intervention systematically and routinely across hospitals (Jacobsen & Ransom, 2007). Improvements have been made, but a range of patients eligible for psychosocial services often do not receive them (Zebrack et al., 2015). Because of time constraints or treatment demands, oncology providers may avoid using screening or clinical interview measures and, instead, rely on clinical judgment (Mitchell, Hussain, Grainger, & Symonds, 2011; Werner, Stenner, & Schüz, 2012).

    The National Comprehensive Cancer Network (NCCN) created distress management guidelines. Included in the guidelines was a global screener of distress (the Distress Thermometer [DT]), an accompanying Problem List, and treatment recommendations for psychosocial issues (Psychosocial Distress Practice Guidelines Panel, 1999). The DT is a one-item, 11-point Likert-type scale represented on a visual graphic of a thermometer that ranges from 0 (no distress) to 10 (extreme distress). Patients use the DT to indicate their level of distress. Patients who report high levels of distress can be administered the accompanying 40-item Problem List, which details common problems related to the cancer experience. The Problem List helps providers identify whether the patient is experiencing practical, family, emotional, spiritual/religious, or physical problems. The DT has proven to be feasible, accessible, and informative (Jacobson & Ransom, 2007; Mitchell, 2010). Studies have tested the validity of the DT, most of which have compared it to the widely accepted Hospital Anxiety and Depression Scale and the Brief Symptom Inventory (Chambers, Zajdlewicz, Youlden, Holland, & Dunn, 2014; Grassi et al., 2013; Holland & Bultz, 2007; Jacobsen et al., 2005; Mitchell, 2010; Thalén-Lindström, Larsson, Hellbom, Glimelius, & Johansson, 2013). Findings from Akizuki et al. (1997) support the DT as an effective method of identifying psychological distress in cancer populations. Based on the results of diagnostic interviews with psychiatrists, Akizuki et al. (1997) found that patients with clinical diagnoses scored significantly higher on the DT than those without a diagnosis.

    Originally, a cutoff score of 5 was used to identify those patients experiencing significant distress because of its numerical placement at the midpoint of the 0–10 scale (Psychosocial Distress Practice Guidelines Panel, 1999; Roth et al., 1998). However, a more recent version of the NCCN practice guidelines for distress recommends that a DT score of 4 or higher indicates moderate to severe distress (NCCN, 2016). In research involving mixed samples, cutoff scores indicating distress vary by culture, language, setting, and demographics, but most studies support a DT cutoff score of 4 or 5 (Donovan, Grassi, McGinty, & Jacobsen, 2014; Grassi et al., 2013; Jacobsen et al., 2005; Mansourabadi, Moogooei, & Nozari, 2014; Martinez, Galdón, Andreu, & Ibáñez, 2013). When researchers have attempted to establish cutoffs for disease-specific groups, the findings have demonstrated a range of results. For example, using receiver operating characteristic (ROC) analyses, research has validated the use of the DT in many cancer populations. Cutoff points have been established for patients with intracranial tumors (6), breast cancer (5), and brain cancer (4) (Dabrowski et al., 2007; Goebel & Mehdorn, 2011; Iskandarsyah et al., 2013; Keir, Calhoun-Eagan, Swartz, Saleh, & Friedman 2008). Optimal DT cutoff points may vary over time. In a sample of patients with prostate cancer, the DT cutoff score to identify significant distress decreased from 4 soon after diagnosis to 3 at longer-term assessments (Chambers et al., 2014). Although these disease-specific cutoff points may be the best option for providers working in one specialized area of oncology, many community cancer centers treat diverse groups of patients. Providers in those settings need a tool and validated cutoff points to use with heterogeneous groups of patients (National Cancer Institute, 2007).

    The purpose of the current study was to establish a comprehensive DT cutoff point based on a large heterogeneous sample of patients with cancer and to determine whether this cutoff was affected by the time since diagnosis. DT scores were compared to Behavioral Health Status (BHS) Index scores, which were used to indicate low, moderate, and high levels of distress based on patient reports of psychological symptoms (anxiety and depression), functional disability, and subjective well-being (Grissom, Lyons, & Lutz, 2002).

    Methods

    Participants

    Study participants were selected from a previously conducted randomized clinical trial on the efficacy of a computerized psychosocial assessment, the Mental Health Assessment and Dynamic Referral for Oncology (MHADRO) (Boudreaux et al., 2011). A total of 836 patients were included from three comprehensive cancer centers: the University of Massachusetts Medical School Cancer Center (n = 581), the Cancer Institute of New Jersey at Cooper Hospital (n = 126), and the University of Texas MD Anderson Cancer Center (n = 129). Eligibility criteria in that study included having a current or past diagnosis of cancer, being older than age 18 years, and not having a significant cognitive deficit that may affect the ability to consent to the study (Boudreaux et al., 2011). Participant characteristics can be found in Table 1.

    Procedure

    Participants in the Boudreaux et al. (2011) study completed a baseline assessment on a tablet computer during a chemotherapy or routine oncology appointment. Patients were recruited at all points during the cancer trajectory (i.e., diagnosis, treatment, and survivorship). Patients were approached for enrollment during a chemotherapy infusion or an ambulatory care appointment with an oncologist. Eligible participants were recruited, enrolled into the study, and randomized to the intervention or control group before completing the assessment. The consent form signed by all participants was explicit in that patients would be enrolled and then randomized to either the intervention or control condition. Randomization into the intervention or control group was completed by an internal random number generator. The University of Massachusetts Medical School Institutional Review Board approved all procedures. Participants in the intervention group received three printed reports that included details of their psychological adjustment; one was provided to the patients, one was shared with their oncologist, and one was placed in the electronic health record. Participants in the intervention group who scored high in distress automatically received the contact information for two appropriate mental health providers based on their geographic area and insurance carrier. In addition, these participants were given the option to choose to automatically send a dynamic referral for an appointment with one of these providers at the completion of the MHADRO assessment. Participants in the control group completed the same MHADRO assessment and received standard care for psychosocial issues. The current article uses baseline data from Boudreaux et al. (2011). Follow-up assessments were completed at 2, 6, and 12 months from baseline. The full methodology appears in O’Hea et al. (2013).

    Measures

    Behavioral Health Status Index: The BHS Index is a global measure of mental health based on phase and dose-response models of psychotherapy outcome (Howard, Kopta, Krause, & Orlinsky, 1986; Howard, Lueger, Maling, & Martinovich, 1993). The BHS Index is a 45-item composite of three subscales: subjective well-being, psychological symptoms, and functioning. These reflect the three dimensions of the phase model. The psychological symptoms subscale is further broken down into symptoms of anxiety and depression. The BHS Index was validated using a large sample (N = 600) of outpatient mental health patients. Each of the subscales was validated against one or more established scales and has good internal consistency reliability (Grissom et al., 2002). The internal consistency (coefficient alpha) for the BHS Index subscales were all high enough to treat each scale as a single construct (subjective well-being, alpha = 0.82; psychological symptoms, alpha = 0.9; functioning, alpha = 0.84). The composite BHS Index has good reliability (0.88) and concurrent validity (r = 0.87, p < 0.001 versus the Outcome Questionnaire–45, a measure of behavioral health), and good sensitivity to change (effect size = 0.6, p < 0.001). The BHS Index was chosen as the foundation of the MHADRO assessment (Boudreaux et al., 2011) because the goal was to identify patients who would find most benefit from mental health services in conjunction with cancer treatment. The BHS Index has been used clinically with college students, substance abuse outpatients, and patients with cardiovascular disease. One goal of the parent MHADRO study was to validate the BHS Index in oncology populations.

    Using the BHS Index, the MHADRO assessment evaluated patients for psychological symptoms (depression and anxiety), physical functioning, and subjective well-being (see Figure 1). Patients were appraised based on their ability to function emotionally and physically, given the distress they experienced on a daily basis. The BHS Index placed participants into one of three categories: low, moderate, or high distress. The cutoff points for low, moderate, and high distress were based on percentile scores comparing each patient to a normative database that consisted of other MHADRO patients. This normative database began with phase 1 data and was updated as more data were collected (Boudreaux et al., 2011). Because these normative data were gathered from a population of patients with cancer, it may represent the current article’s population of interest more closely than would normative data gathered from a general population. Higher scores signify more severe symptoms and worse functioning. Patients with scores in the 70th percentile or higher were considered clinically elevated in distress, patients who fell below the 30th percentile were considered low in distress, and patients in the 30th to 70th percentiles were considered within normal limits.

    National Comprehensive Cancer Network Distress Thermometer: Participants rated their distress level using the DT. As described previously, the DT has been validated using ROC analyses in numerous oncology populations and has held up against other validated and lengthier measures.

    Statistical Analysis

    Basic descriptive statistics were used to characterize the sample. To characterize the association between DT scores and categorical characteristics of interest, the authors used analysis of variance. When significance was achieved, the authors further examined specific group differences by calculating least squares means. The least squares means approach is preferred because it allows for adjustment for multiple comparison testing and unbalanced designs. Specifically, a Tukey-Kramer adjustment (Kramer, 1956) that accommodates unequal sample sizes in groups was used.

    The authors assessed the association between DT scores and the BHS Index through a Spearman correlation. The Spearman correlation was calculated rather than a Pearson correlation because the former only requires the data to be measured at least at the ordinal scale (rather than the interval scale requirement of Pearson correlation) and makes no assumptions about the underlying distribution.

    The authors further investigated the relationship between the DT and the BHS Index using analysis of variance. Based on the categorization of low, moderate, and high distress on the BHS Index, the authors determined if a difference existed in the DT score based on group membership (low, moderate, or high distress). When significance was achieved, additional examination was conducted on specific group differences by calculating least squares means.

    The authors used ROC curves and the associated c-index to investigate the relationship between psychological distress as measured by the DT and BHS Index scores. Specifically, the authors were interested in determining if a cutoff point with acceptable sensitivity and specificity could be defined on the DT that corresponded to higher distress on the more thoroughly examined and commonly used BHS Index score. As a reference, a BHS Index score in the highest tertile (percentile score greater than or equal to 70%) was considered highly distressed. Several logistic regression models predicting the probability of high distress (as measured by the BHS Index score) with varying cutoff points on the DT score were developed, and the corresponding sensitivity, specificity, positive predictive, and negative predictive values were calculated. All statistical analyses were performed using SAS, version 9.3.

    Results

    Descriptive Statistics

    Eleven mental health diagnoses were indicated in patient demographic information, with the most common being depression (24%, n = 200) and anxiety (14%, n = 117). Four percent of patients (n = 35) had been hospitalized because of an emotional problem at least once in their lifetime. Thirty-one percent (n = 259) of the total sample had been prescribed psychotropic medication, 20% (n = 167) of which were taking medication at the time of enrollment. Thirty-seven percent (n = 309) of patients had seen a counselor or therapist to help manage emotions or stress, and 8% (n = 66) were in therapy at the time of assessment. When asked whether they would benefit from counseling at the time of enrollment, 17% (n = 70) of participants in the intervention group (n = 415) said “yes” and 18% (n = 74) indicated that they were “not sure.”

    Relationship Between Behavioral Health Status Index and the Distress Thermometer

    To examine the relationship between DT and BHS Index scores, a Spearman correlation coefficient and associated p value were calculated and the two variables were found to be significantly related (rho = 0.58, p < 0.0001). As distress measured by the BHS Index increased, the mean DT scores increased. In addition, least squares means analyses revealed that each category of BHS score was related to a significantly different DT score. For all three categories (low distress, within normal limits, clinically elevated distress), mean DT scores (mean = 0.79, 2.58, and 4.78, respectively) differed at the p < 0.0001 level.

    Cutoff Point Analyses

    Cutoff points were determined using ROC analysis. ROC curve and the associated c-index were used to investigate the relationship between psychological distress as measured by the DT and BHS Index scores. Specifically, the authors were interested in determining if a cutoff point with acceptable sensitivity and specificity could be defined on the DT that corresponded to higher distress on the BHS Index. As a reference, a BHS Index score in the highest tertile was considered highly distressed. Several logistic regression models predicting the probability of high distress (as measured by the BHS Index score) with varying cutoff points on the DT score were developed, and the corresponding sensitivity, specificity, positive predictive, and negative predictive values were calculated.

    The predictive accuracy of the DT score in predicting high distress as measured by the BHS Index was 0.784 (c-statistic) (see Figure 2). Based on the cutoff point for distress using the DT score, sensitivity ranged from 0.31–0.78 and specificity from 0.69–0.93. The current recommended cutoff point on the DT scale, 4, had a sensitivity of 0.68 and a specificity of 0.79. The corresponding positive and negative predictive values were 0.51 and 0.89, respectively. A cutoff point of 3 on the DT was found to have higher sensitivity (0.78) and a higher negative predictive value (0.91) but lower specificity (0.69) and a lower positive predictive value (0.44).

    Distress Thermometer Scores and Time Since Diagnosis

    A statistically significant difference in distress was found among patients, depending on the amount of time that had lapsed since the diagnosis of cancer. Figure 3 demonstrates mean levels of distress, measured by the DT, at each of the four times since diagnosis. The following categories are significantly different from each other at the p < 0.05 level: 1–4 weeks postdiagnosis versus 1–6 months, 1–4 weeks versus more than 6 months, and 1–6 months versus more than 6 months.

    Discussion

    Oncology providers and governing bodies in oncology care agree that the psychosocial needs of individuals with cancer should be identified and addressed. Efficient and accurate screening tools are helpful to providers who are attempting to accomplish these goals. The current study examined the DT as a brief screening tool for distress and was compared to a psychometrically validated assessment of distress. In accordance with previous research, scores on the DT were significantly associated with scores on the lengthier BHS Index (Chambers et al., 2014; Donovan et al., 2014; Goebel & Mehdorn, 2011). In addition, time since diagnosis was related to scores on the DT. Specifically, patients who were 1–4 weeks postdiagnosis indicated more distress than patients at any other time in the cancer trajectory, even those patients who were in the first week postdiagnosis. This poses an immediate need to ensure that patients receive the option for psychosocial services when they are diagnosed with cancer, because this may help them navigate the difficult journey they are about to embark on. Fluctuations of distress levels during the course of the cancer trajectory were consistent with previous research (Chambers et al., 2014; Lam, Shing, Bonanno, Mancini, & Fielding, 2012; Nosarti, Roberts, Crayford, McKenzie, & David, 2002; Wang, Tu, Liu, Yeh, & Hsu, 2013; Ziegler et al., 2011). During the first week, patients may be feeling numb or in a state of disbelief before they have fully processed the diagnosis (National Cancer Institute, 2013). Once the diagnosis has sunk in, but before treatment plans have been determined, patients may experience heightened distress, which could explain the high levels of distress during the 1–4 week postdiagnosis interval. This time from diagnosis to beginning of treatment could be described as the “in limbo” time, when patients are anxious because they have not begun actively fighting the cancer through treatment. Providers may need to screen for distress symptoms in patients during this phase, and it may be helpful to offer mental health services. These proactive measures may serve to help patients transition through the different phases of coping with cancer.

    This study may have implications for the clinical use of brief screeners of psychosocial distress in cancer populations. The authors’ analyses indicated that using a cutoff point of 3 to indicate high levels of distress may maximize sensitivity and be a more useful option in some heterogeneous clinical settings. Because this score is lower than the one recommended by the NCCN (2016) guidelines, it raises the issue of overdiagnosing distress in a clinical setting and using valuable provider time and resources. Another option would be to use multiple cutoff points, depending on the situation. For example, providers may consider erring on the side of caution and using a lower cutoff point (e.g., DT score of 3) for those individuals who are 1–4 weeks postdiagnosis, but returning to the recommended cutoff score (DT score of 4) for those further from diagnosis, when distress may not be as prevalent. Future research should examine different DT cutoff points at different stages in the treatment trajectory in a heterogeneous sample.

    Limitations

    The study had limitations that could be addressed through future research. First, the use of the BHS Index does not necessarily predict clinical diagnoses. Optimally, each participant in the study would have undergone clinical psychological evaluation, in addition to administration of the DT, to more accurately predict clinical outcomes. Another major limitation was the demographic characteristics of the sample. Although enrollment did not exclude any cancer type or stage of disease, almost half of the sample consisted of female patients with breast cancer, and cancer survivors comprised more than half of the sample. This may have occurred because those who enroll in a study examining the effectiveness of a psychosocial intervention would likely be those most interested in using such intervention. Women with cancer tend to report greater levels of psychological distress compared to men with cancer (Akin, Can, Avdiner, Ozdilli, & Durna, 2010; Thomas, NandaMohan, Nair, & Pandey, 2010). Although distress levels did not vary among cancer types, previous research suggests that certain types of cancer, specifically brain cancer, might result in higher distress levels (Keir et al., 2008). In addition, race and ethnicity of the sample were largely Caucasian and non-Hispanic, causing generalizability of results to be questionable. Future researchers are encouraged to replicate the current study with more diverse samples regarding disease type, gender, and ethnicity.

    The NCCN guidelines on distress are vague as far as how often it is recommended that patients be screened for distress. Perhaps using a cutoff point of 3 at the first assessment will help providers identify patients who are distressed and, from there, make a plan moving forward. The DT can be used once psychosocial resources are established to ensure that distress has decreased. In addition, a DT score of 3 could alert a nurse to conduct further assessment to determine in which domain the concern lies (e.g., practical, emotional, spiritual, physical). Of note, individuals who are female, older, less educated, and carry a higher symptom burden are more likely to endorse symptoms of distress (Waller et al., 2013). Patients who are at higher psychosocial risk should be assessed more frequently and thoroughly.

    As noted previously, a majority of participants in the sample were cancer survivors. These patients who were no longer in treatment may have been feeling physically better and less acutely anxious or distressed. Previous research has demonstrated that, as time since diagnosis increases, patients tend to experience an improved physical and mental status. At the same time, survivors face many social and emotional challenges when coming off of treatment (Jarrett et al., 2013). Future research with the DT is needed to establish cutoff points for screening survivors who are experiencing a whole new set of stressors as they enter into formal survivorship.

    Implications for Nursing and Conclusion

    In a comprehensive cancer center, nurses often are responsible for psychosocial screening of patients, whether using a validated screening measure or clinical judgment (Werner et al., 2012). Screening patients with cancer and providing resources for distress throughout the treatment trajectory is important, even when preparing to transition from acute treatment. Nurses are becoming more involved with a growing practice of survivorship care planning, which helps survivors prepare for the transition back to primary care after the completion of cancer treatment. The process can be emotionally difficult for survivors who have grown accustomed to the social support of the oncology team (Lester, Wessels, & Jung, 2014). These additional demands of assessing and providing resources to patients are being placed on nurses, who are already an overworked and systematically underappreciated population (Kaddourah, Khalidi, Abu-Shaheen, & Al-Tannir, 2013). However, even brief screening measures of distress have been supported as efficacious. Developing ways to administer measures of distress while minimizing burden on nurses is important. This may be done through brief screeners, such as the DT. Regardless of the assessment tools chosen by oncology care providers, the future of managing psychosocial issues in patients will need to involve efficient options that can minimize burden on nurses and maximize benefits to patients.

    About the Author(s)

    Cutillo is a graduate student in the Department of Psychology at the University of Alabama in Birmingham; O’Hea is a professor in the Department of Psychology at Stonehill College in Easton, MA; Person is an associate professor and Lessard is a senior biostatistician, both in the Department of Quantitative Health Sciences at the University of Massachusetts Medical School in Worchester; Harralson is a science director at Polaris Health Directions in Langhorne, PA; and Boudreaux is a professor in the Department of Emergency Medicine at the University of Massachusetts Medical School in Worchester. This research was funded by a grant (R42 MH078732) from the National Institutes of Health/National Institute of Mental Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. O’Hea and Boudreaux have previously consulted for Polaris Health Directions. O’Hea and Boudreaux contributed to the conceptualization and design. Cutillo, O’Hea, Harralson, and Boudreaux completed the data collection. Cutillo, Person, Lessard, Harralson, and Boudreaux provided statistical support. Cutillo, Person, Harralson, and Boudreaux provided the analysis. Cutillo, O’Hea, Person, Harralson, and Boudreaux contributed to the manuscript preparation. Cutillo can be reached at acutillo@uab.edu, with copy to editor at ONFEditor@ons.org. Submitted February 2016. Accepted for publication August 31, 2016.

     

    References

    Akin, S., Can, G., Avdiner, A., Ozdilli, K., & Durna, Z. (2010). Quality of life, symptom experience and distress of lung cancer patients undergoing chemotherapy. European Journal of Oncology Nursing, 14, 400–409. doi:10.1016/j.ejon.2010.01.003

    Akizuki, N., Akechi, T., Nakanishi, T., Yoshikawa, E., Okamura, M., Nakano, T., . . . Uchitomi, Y. (1997). Development of a brief screening interview for adjustment disorders and major depression in patients with cancer. Cancer, 97, 2605–2613. doi:10.1002/cncr.11358

    Boudreaux, E.D., O’Hea, E.L., Grissom, G., Lord, S., Houseman, J., & Grana, G. (2011). Initial development of the Mental Health Assessment and Dynamic Referral for Oncology (MHADRO). Journal of Psychosocial Oncology, 29, 83–102. doi:10.1080/07347332.2010.532299

    Carlson, L.E., Angen, M., Cullum, J., Goodey, E., Koopmans, J., Lamont, L., . . . Bultz, B.D. (2004). High levels of untreated distress and fatigue in cancer patients. British Journal of Cancer, 90, 2297–2304. doi:10.1038/sj.bjc.6601887

    Carlson, L.E., & Bultz, B.D. (2004). Efficacy and medical cost offset of psychosocial interventions in cancer care: Making the case for economic analyses. Psycho-Oncology, 13, 837–849. doi:10.1002/pon.832

    Carlson, L.E., Groff, S.L., Maciejewski, O., & Bultz, B.D. (2010). Screening for distress in lung and breast cancer outpatients: A randomized controlled trial. Journal of Clinical Oncology, 28, 4884–4891. doi:10.1200/JCO.2009.27.3698

    Carlson, L.E., Waller, A., & Mitchell, A.J. (2012). Screening for distress and unmet needs in patients with cancer: Review and recommendations. Journal of Clinical Oncology, 30, 1160–1177. doi:10.1200/JCO.2011.39.5509

    Chambers, S.K., Zajdlewicz, L., Youlden, D.R., Holland, J.C., & Dunn, J. (2014). The validity of the distress thermometer in prostate cancer populations. Psycho-Oncology, 23, 195–203. doi:10.1002/pon.3391

    Dabrowski, M., Boucher, K., Ward, J.H., Lovell, M.M., Sandre, A., Bloch, J., . . . Buys, S.S. (2007). Clinical experience with the NCCN Distress Thermometer in breast cancer patients. Journal of the National Comprehensive Cancer Network, 5, 104–111.

    Donovan, K.A., Grassi, L., McGinty, H.L., & Jacobsen, P.B. (2014). Validation of the distress thermometer worldwide: State of the science. Psycho-Oncology, 23, 241–250. doi:10.1002/pon.3430

    Ellis, J., Lin, J., Walsh, A., Lo, C., Shepherd, F.A., Moore, M., . . . Rodin, G. (2009). Predictors of referral for specialized psychosocial oncology care in patients with metastatic cancer: The contributions of age, distress, and marital status. Journal of Clinical Oncology, 27, 699–705. doi:10.1200/JCO.2007.15.4864

    Faller, H., Schuler, M., Richard, M., Heckl, U., Weis, J., & Küffner, R. (2013). Effects of psycho-oncologic interventions on emotional distress and quality of life in adult patients with cancer: Systematic review and meta-analysis. Journal of Clinical Oncology, 31, 782–793. doi:10.1200/JCO.2011.40.8922

    Goebel, S., & Mehdorn, H.M. (2011). Measurement of psychological distress in patients with intracranial tumours: The NCCN Distress Thermometer. Journal of Neuro-Oncology, 104, 357–364. doi:10.1007/s11060-010-0501-5

    Grassi, L., Johansen, C., Annunziata, M.A., Capovilla, E., Costantini, A., Gritti, P., . . . Bellani, M. (2013). Screening for distress in cancer patients: A multicenter, nationwide study in Italy. Cancer, 119, 1714–1721. doi:10.1002/cncr.27902

    Grissom, G.R., Lyons, J.S., & Lutz, W. (2002). Standing on the shoulders of a giant: Development of an outcome management system based on the dose model and phase model of psychotherapy. Psychotherapy Research, 12, 397–412. doi:10.1093/ptr/12.4.397

    Holland, J.C., & Bultz, B.D. (2007). The NCCN guideline for distress management: A case for making distress the sixth vital sign. Journal of the National Comprehensive Cancer Network, 5, 3–7.

    Howard, K.I., Kopta, S.M., Krause, M.S., & Orlinsky, D.E. (1986). The dose-effect relationship in psychotherapy. American Psychologist, 4, 159–164.

    Howard, K.I., Lueger, R.J., Maling, M.S., & Martinovich, Z. (1993). A phase model of psychotherapy outcome: Causal mediation of change. Journal of Consulting and Clinical Psychology, 61, 678–685.

    Iskandarsyah, A., de Klerk, C., Suardi, D.R., Soemitro, M.P., Sadarjoen, S.S., & Passchier, J. (2013). The Distress Thermometer and its validity: A first psychometric study in Indonesian women with breast cancer. PLOS ONE, 8, e56353.

    Jacobsen, P.B., Donovan, K.A., Trask, P.C., Fleishman, S.B., Zabora, J., Baker, F., & Holland, J.C. (2005). Screening for psychological distress in ambulatory cancer patients. Cancer, 103, 1494–1502. doi:10.1002/cncr.20940

    Jacobson, P.B., & Ransom, S. (2007). Implementation of NCCN distress management guidelines by member institutions. Journal of the National Comprehensive Cancer Network, 5, 99–103.

    Jarrett, N., Scott, I., Addington-Hall, J., Amir, Z., Brearley, S., Hodges, L., . . . Foster, C. (2013). Informing future research priorities into the psychological and social problems faced by cancer survivors: A rapid review and synthesis of the literature. European Journal of Oncology Nursing, 17, 510–520. doi:10.1016/j.ejon.2013.03.003

    Kaddourah, B.T., Khalidi, A., Abu-Shaheen, A.K., & Al-Tannir, M.A. (2013). Factors impacting job satisfaction among nurses from a tertiary care centre. Journal of Clinical Nursing, 22, 3153–3159.

    Keir, S.T., Calhoun-Eagan, R.D., Swartz, J.J., Saleh, O.A., & Friedman, H.S. (2008). Screening for distress in patients with brain cancer using the NCCN’s rapid screening measure. Psycho-Oncology, 17, 621–625. doi:10.1002/pon.1271

    Kramer, C.Y. (1956). Extension of multiple range tests to group means with unequal numbers of replications. Biometrics, 12, 307–310. doi:10.2307/3001469

    Lam, W.W., Shing, Y.T., Bonanno, G.A., Mancini, A.D., & Fielding, R. (2012). Distress trajectories at the first year diagnosis of breast cancer in relation to 6 years survivorship. Psycho-Oncology, 21, 90–99. doi:10.1002/pon.1876

    Lester, J.L., Wessels, A.L., & Jung, Y. (2014). Oncology nurses’ knowledge of survivorship care planning: The need for education [Online exclusive]. Oncology Nursing Forum, 41, E35–E43. doi:10.1188/14.ONF.E35-E43

    Mansourabadi, A., Moogooei, M., & Nozari, S. (2014). Evaluation of distress and stress in cancer patients in AMIR Oncology Hospital in Shiraz. Iranian Journal of Pediatric Hematology Oncology, 4(4), 131–140.

    Martinez, P., Galdón, M.J., Andreu, Y., & Ibáñez, E. (2013). The Distress Thermometer in Spanish cancer patients: Convergent validity and diagnostic accuracy. Supportive Care in Cancer, 21, 3095–3102. doi:10.1007/s00520-013-1883-7

    Mehnert, A., & Koch, U. (2007). Prevalence of acute and post-traumatic stress disorder and comorbid mental disorders in breast cancer patients during primary cancer care: A prospective study. Psycho-Oncology, 16, 181–188. doi:10.1002/pon.1057

    Mitchell, A.J. (2010). Short screening tools for cancer-related distress: A review and diagnostic validity meta-analysis. Journal of the National Comprehensive Cancer Network, 8, 487–494.

    Mitchell, A.J., Chan, M., Bhatti, H., Halton, M., Grassi, L., Johansen, C., & Meader, N. (2011). Prevalence of depression, anxiety, and adjustment disorder in oncological, haematological, and palliative-care settings: A meta-analysis of 94 interview-based studies. Lancet Oncology, 12, 160–174.

    Mitchell, A.J., Hussain, N., Grainger, L., & Symonds, P. (2011). Identification of patient-reported distress by clinical nurse specialists in routine oncology practice: A multicentre UK study. Psycho-Oncology, 20, 1076–1083. doi:10.1002/pon.1815

    Mitchell, A.J., Vahabzadeh, A., & Magruder, K. (2011). Screening for distress and depression in cancer settings: 10 lessons from 40 years of primary-care research. Psycho-Oncology, 20, 572–584. doi:10.1002/pon.1943

    National Cancer Institute. (2007). NCI community cancer centers program pilot: 2007–2010. Retrieved from http://www.georgiacancer.org/pdfs/NCI-Fact-sheet.pdf

    National Cancer Institute. (2013). Adjustment to cancer: Anxiety and distress (PDQ®)–patient version. Retrieved from https://www.cancer.gov/about-cancer/coping/feelings/anxiety-distress-pd…

    National Comprehensive Cancer Network. (2016). NCCN Clinical Practice Guidelines in Oncology: Distress management [v.2.2016]. Retrieved from https://www.nccn.org/professionals/physician_gls/pdf/distress.pdf

    Nosarti, C., Roberts, J.V., Crayford, T., McKenzie, K., & David, A.S. (2002). Early psychological adjustment in breast cancer patients: A prospective study. Journal of Psychosomatic Research, 53, 1123–1130.

    O’Hea, E.L., Cutillo, A., Dietzen, L., Harralson, T., Grissom, G., Person, S., & Boudreaux, E.D. (2013). Randomized control trial to test a computerized psychosocial cancer assessment and referral program: Methods and research design. Contemporary Clinical Trials, 35, 15–24. doi:10.1016/j.cct.2013.02.001

    Psychosocial Distress Practice Guidelines Panel. (1999). NCCN practice guidelines for the management of psychosocial distress. Oncology, 13(5A), 113–147.

    Roth, A.J., Kornblith, A.B., Batel-Copel, L., Peabody, E., Scher, H.I., & Holland, J.C. (1998). Rapid screening for psychological distress in men with prostate carcinoma: A pilot study. Cancer, 82, 1904–1908.

    Thalén-Lindström, A., Larsson, G., Hellbom, M., Glimelius, B., & Johansson, B. (2013). Validation of the Distress Thermometer in a Swedish population of oncology patients; accuracy of changes during six months. European Journal of Oncology Nursing, 17, 625–631. doi:10.1016/j.ejon.2012.12.005

    Thomas, B.C., NandaMohan, V., Nair, M.K., & Pandey, M. (2010). Gender, age and surgery as a treatment modality leads to higher distress in patients with cancer. Supportive Care in Cancer, 19, 239–250. doi:10.1007/s00520-009-0810-4

    Tuinman, M.A., Gazendam-Donofrio, S.M., & Hoekstra-Weebers, J.E. (2008). Screening and referral for psychosocial distress in oncologic practice: Use of the Distress Thermometer. Cancer, 113, 870–878. doi:10.1002/cncr.23622

    van Scheppingen, C., Schroevers, M.J., Smink, A., van der Linden, Y.M., Mul, V.E., Langendijk, J.A., . . . Sanderman, R. (2011). Does screening for distress efficiently uncover meetable unmet needs in cancer patients? Psycho-Oncology, 20, 655–663. doi:10.1002/pon.1939

    Verdonck-de Leeuw, I.M., de Bree, R., Keizer, A.L., Houffelaar, T., Cuijpers, P., van der Linden, M.H., & Leemans, C.R. (2009). Computerized prospective screening for high levels of emotional distress in head and neck cancer patients and referral rate to psychosocial care. Oral Oncology, 45, E129–E133. doi:10.1016/j.oraloncology.2009.01.012

    Waller, A., Williams, A., Groff, S.L., Bultz, B.D., & Carlson, L.E. (2013). Screening for distress, the sixth vital sign: Examining self-referral in people with cancer over a one-year period. Psycho-Oncology, 22, 388–395. doi:10.1002/pon.2102

    Wang, W.T, Tu, P.C., Liu, T.J., Yeh, D.C., & Hsu, W.Y. (2013). Mental adjustment at different phases in breast cancer trajectory: Re-examination of factor structure of the Mini-MAC and its correlation with distress. Psycho-Oncology, 22, 768–774. doi:10.1002/pon.3065

    Werner, A., Stenner, C., & Schüz, J. (2012). Patient versus clinician symptom reporting: How accurate is the detection of distress in the oncologic after-care? Psycho-Oncology, 21, 818–826. doi:10.1002/pon.1975

    Zebrack, B., Kayser, K., Sundstrom, L., Savas, S.A., Henrickson, C., Acquati, C., & Tamas, R.L. (2015). Psychosocial distress screening implementation in cancer care: An analysis of adherence, responsiveness, and acceptability. Journal of Clinical Oncology, 33, 1165–1170. doi:10.1200/JCO.2014.57.4020

    Ziegler, L., Hill, K., Neilly, L., Bennett, M.I., Higginson, I.J., Murray, S.A., & Stark, D. (2011). Identifying psychological distress at key stages of the cancer illness trajectory: A systematic review of validated self-report measures. Journal of Pain and Symptom Management, 41, 619–636. doi:10.1016/j.jpainsymman.2010.06.024