Where There’s Smoke: Validating a Nonproprietary Single-Item Burnout-Impacting-Safety Scale


  • Amanda M. Pearl, PhD Penn State Milton S. Hershey Medical Center https://orcid.org/0000-0002-4044-8067
  • Steve Mrozowski, MHA, NRP Penn State Milton S. Hershey Medical Center
  • Daniel E. Shapiro, PhD Penn State Milton S. Hershey Medical Center




patient safety, burnout, medical errors, patient care, area under curve


Where There's Smoke - visual abstract

This study aimed to assess the utility of a single survey item to predict the impact of burnout on safety and quality of healthcare providers as perceived by their colleagues. The primary objective is to determine if the item predicts the frequency of patient safety event reports within certain clinical departments. The secondary objective will be to determine if there is an acceptable cutoff score for the item which predicts low versus high numbers of safety events reported by healthcare providers in each clinical department. Participants were 424 healthcare providers in an academic medical center in the mid-Atlantic region of the United States. The item was designed to assess for the perception of the impact of burnout on work in terms of quality or safety using a 5-point Likert scale. Data from a patient safety event reporting system was accessed for the year of survey completion (2017). A negative binomial regression was used to assess the ability of the item to predict reported patient safety event reports. The item was found to significantly predict objective safety event data. Sensitivity and specificity, as well as receiver operating characteristic (ROC) curve analyses, were conducted to determine appropriateness of cutoff scores to identify low- and high-risk clinical departments. The item was found to demonstrate adequate sensitivity (82%) using a cutoff score of 4 on the survey item. However, the area under the curves (AUCs) which assess diagnostic accuracy fell in the poor range. These results suggest that healthcare administrators could deploy this single item as a brief pulse or screener of teams of individuals who are within a work unit and use a cutoff score of 4 as a means to assess for hot spots where healthcare provider burnout may be putting patients at high risk in terms of safety.

Correction: This article was corrected on June 4, 2021, to fix a numerical error in the x-axis of Figure 1.

Author Biographies

Amanda M. Pearl, PhD, Penn State Milton S. Hershey Medical Center

Amanda Pearl (amp230@psu.edu) is the director of Faculty Analytics and an associate professor of Psychiatry and Behavioral Health at the Penn State College of Medicine.

Steve Mrozowski, MHA, NRP, Penn State Milton S. Hershey Medical Center

Steve Mrozowski is the director of patient safety at the Penn State Health Milton S. Hershey Medical Center.

Daniel E. Shapiro, PhD, Penn State Milton S. Hershey Medical Center

Dan Shapiro is a professor and the vice dean for Faculty and Administrative Affairs at the Penn State College of Medicine.


Maslach C, Jackson SE. The Measurement of Experienced Burnout. J Occup Behav. 1981;2:99-113.

Maslach C, Schaufeli WB, Leiter MP. Job Burnout. Ann Rev Psychol 2001;52:397-422.

Shanafelt TD, Boone S, Tan L, et al. Burnout and Satisfaction With Work-Life Balance Among US Physicians Relative to the General US Population. Arch Intern Med 2012;172:1377-1385.

Shanafelt TD, Hasan O, Dyrbye LN, et al. Changes in Burnout and Satisfaction With Work-Life Balance in Physicians and the General US Working Population Between 2011 and 2014. Mayo Clin Proc 2015;90:1600-1613.

Shanafelt TD, Balch CM, Bechamps G, et al. Burnout and Medical Errors Among American Surgeons. Ann Surg. 2010;251:995-1000.

Shanafelt TD, Gradishar WJ, Kosty M, et al. Burnout and Career Satisfaction Among US Oncologists. J Clin Oncol. 2014;32:678-686.

Noseworthy J, Madara J, Cosgrove, et al. Physician Burnout is a Public Health Crisis: A Message to our Fellow Health Care CEOs [Internet]. Health Affairs; 2017 [accessed 2020 Nov 27]. Available at https://www.healthaffairs.org/do/10.1377/hblog20170328.059397/full/.

Han S, Shanafelt TD, Sinsky CA, et al. Estimating the Attributable Cost of Physician Burnout in the United States. Ann Intern Med. 2019;170:784-790.

Anagnostopoulos F, Liolios E, Persefonis G, et al. Physician burnout and Patient Satisfaction With Consultation in Primary Health Care Settings: Evidence of Relationships From One-With-Many Design. J Clin Psychol Med S. 2012;19:401-410.

Klein J, Frie KG, Blum K, et al. Psychosocial Stress at Work and Perceived Quality of Care Among Clinicians in Surgery. BMC Health Serv Res. 2007;11:109.

Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press, 2001.

Thomas LR, Ripp JA, West CP. Charter on Physician Well-Being. JAMA. 2018;319:1541-1542.

World Health Organization. Quality of Care: A Process for Making Strategic Choices in Health Systems. Geneva, 2006.

Wen J, Cheng Y, Hu X, et al. Workload, Burnout, and Medical Mistakes Among Physicians in China: A Cross-Sectional Study. Biosci Trends 2016;10:27-33

Welp A, Meier LL, Manser T. Emotional Exhaustion and Workload Predict Clinician-Related and Objective Patient Safety. Front Psychol. 2015;5:1-13.

Hall LH, Johnson J, Heyhoe J, et al. Exploring the Impact of Primary Care Physician Burnout and Wellbeing on Patient Care: A Focus Group Study. J Patient Saf. 2017.

Levinson DR, Inspector General. Adverse Events in Hospitals: National Incidence Among Medicare Beneficiaries. Department of Health and Human Services Office of the Inspector General 2010.

de Vries EN, Ramrattan MA, Smorenburg SM, et al. The Incidence and Nature of In-Hospital Adverse Events: A Systematic Review. BMJ Quality & Safety. 2008;17:216-223.

Van Gerven E, Bruyneel L, Panella M, et al. Psychological Impact and Recovery After Involvement in a Patient Safety Incident: A Repeated Measures Analysis. BMJ Open. 2016;6:e011403.

Balch CM, Oresckovish MR, Dyrbye LN, et al. Personal Consequences of Malpractice Lawsuits on American Surgeons. J Am Coll Surg. 2011;213:657-667.

Williams ES, Manwell LB, Konrad TR, et al. The Relationship of Organizational Culture, Stress, Satisfaction, and Burnout With Physician-Reported Error and Suboptimal Patient Care: Results From the MEMO study. Health Care Manage R. 2007;3:203-212.

Firth-Cozens J, Greenhalgh J. Doctors’ Perceptions of the Links Between Stress and Lowered Clinical Care. Soc Sci Med. 1997;44:1017-1022.

Shanfelt TD, Bradley KA, Wipf JE, et al. Burnout and Self-Reported Patient Care in an Internal Medicine Residency Program. Ann Intern Med. 2002:136:358-367.

West CP, Huschka MM, Novotny PJ, et al. Association of Perceived Medical Errors with Resident Distress and Empathy: A Prospective Longitudinal Study. JAMA. 2006;296:1071-1078.

Moss M, Good VS, Gozal D, et al. An Official Critical Care Societies Collaborative Statement: Burnout Syndrome in Critical Care Health Care Professionals: A Call for Action. Am J Crit Care. 2016;25:368-376.

Wright AA, Katz IT. Beyond Burnout – Redesigning Care to Restore Meaning and Sanity for Physicians. N Engl J Med. 2018;374:309-311.

Dewa CS, Loong D, Bonato S, et al. The Relationship Between Physician Burnout and Quality of Healthcare in Terms of Safety and Acceptability: A Systematic Review. BMJ Open. 2017;7:e015141.

Hall LH, Johnson J, Watt I, et al. Healthcare Staff Wellbeing, Burnout, and Patient Safety: A Systematic Review. PLoS One. 2016;11:e0159015.

Panagioti M, Geraghty K, Johnson J, et al. Association Between Physician Burnout and Patient Safety, Professionalism, and Patient Satisfaction: A Systematic Review and Meta-Analysis. JAMA Intern Med. 2018;178:1317-1331.

Tawfik DS, Profit J, Morgenthaler TI, et al. Physician Burnout, Well-Being, and Work Unit Safety Grades in Relationship to Reported Medical Errors. Mayo Clin Proc. 2018;93:1571-1580.

Shanafelt TD, West, CP, Sinsky C, et al. Changes in Burnout and Satisfaction With Work-Life Integration in Physicians and the General US Working Population between 2011 and 2017. Mayo Clin Proc. 2019;94:1681-1694.

Wu AW. Medical Error: The Second Victim. BMJ. 2000;320:726-727.

Lander LI, Connor JA, Shah RK, et al. Otolaryngologists’ Responses to Errors and Adverse Events. Laryngoscope. 2006;116:1114-1120.

Scott SD. The Second Victim Phenomenon: A Harsh Reality of Health Care Professions. Perspect Saf. 2011.

Edrees HH, Paine LA, Feroli ER, et al. Health Care Workers as Second Victims of Medical Errors. Pol Arch Med Wewn. 2011;121:101-108.

Seys D, Wu AW, Van Gerven E, et al. Health Care Professionals as Second Victims After Adverse Events: A Systematic Review. Eval Health Prof. 2013;36:135-162.

White AA, Waterman AD, McCotter P, et al. Supporting Health Care Workers After Medical Error: Considerations for Health Care Leaders. J Clin Outcomes Manag. 2008;15:240-247.

Lewis E, Baernholdt M, Hamric AB. Nurses’ Experience of Medical Errors: An Integrative Literature Review. J Nurs Care Qual. 2013;28:153-161.

Scott SD, Hirschinger LE, Cox KR, et al. Caring For Our Own: Deploying a Systemwide Second Victim Rapid Response Team. Joint Comm J Qual Patient Saf. 2010;36:233-240.

West CP, Tan AD, Habermann TM, et al. Association of Resident Fatigue and Distress With Perceived Medical Errors. JAMA. 2009;302:1294-1300.

Hall, LH, Johnson J, Watt I, et al. Association of GP Wellbeing and Burnout With Patient Safety in UK Primary Care: A Cross-Sectional Survey. Brit J Gen Pract. 2019;69:e507-514.

Hayashino Y, Utsugi-Ozaski M, Feldman MD, et al. Hope Modified the Association Between Distress and Incidence of Self-Perceived Medical Errors Among Practicing Physicians: Prospective Cohort Study. PloS One. 2012;7:e35585.

Podsakoff PM, MacKenzie SB, Podsakoff NP. Sources of Method Bias in Social Science Research and Recommendations on How to Control It. Annu Rev Psychol. 2012;63:539-569.

Raphael K. Recall bias: A Proposal for Assessment and Control. Int J Epidemiol. 1987;16:167-170.

Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of Physician Self-Assessment Compared With Observed Measures of Competence: A Systematic Review. JAMA. 2006;9:1094-1102.

Kaldjian LC, Jones EW, Wu BJ, et al. Reporting Medical Errors to Improve Patient Safety: A Survey of Physicians in Teaching Hospitals. Arch Intern Med. 2008;168:40-46.

Shanafelt TD. Enhancing meaning in work: A Prescription for Preventing Physician Burnout and Promoting Patient-Centered Care. JAMA. 2009;302:1338-1340.

Tanaka M, Tanaka K, Takano T, et al. Analysis of Risk of Medical Errors Using Structure-Equation Modelling: A 6-Month Prospective Cohort Study. BMJ Qual Saf. 2012;21:784-790.

Garrouste-Orgeas M, Perrin M, Soufir L, et al. The Latroref Study: Medical Errors are Associated With Symptoms of Depression in ICU Staff But Not Burnout or Safety Culture. Intens Care Med. 2015;41:273-284.

Salah AM, Awadalla NJ, El-masri YM, et al. Impacts of Nurses’ Circadian Rhythm Sleep Disorders, Fatigue, and Depression on Medication Administration Errors. Egypt J Chest Dis Tuberc. 2014;63:145-153.

Linzer M, Manwell LB, Williams ES, et al. Working Conditions in Primary Care: Physician Reactions and Care Quality. Ann Intern Med. 2009;151:28-36.

Cimiotti JP, Aiken LH, Sloane DM, et al. Nurse Staffing, Burnout, and Health Care-Associated Infection. Am J Infect Control. 2012;40:486-490.

Dugan J, Lauer E, Bouquot Z, et al. Stressful Nurses: The Effect on Patient Outcomes. J Nurs Care Qual. 1996;10:46-58.

Fahrenkopf AM, Sectish TC, Barger LK, et al. Rates of Medication Errors Among Depressed and Burnt Out Residents: Prospective Cohort Study. BMJ. 2008;336:488-491.

Halbesleben JR, Wakefield BJ, Wakefield DS, et al. Nurse Burnout and Patient Safety Outcomes: Nurse Safety Perception Versus Reporting Behavior. Western J Nurs Res. 2008;30:560-577.

Adair JG. The Hawthorne effect: A Reconsideration of the Methodological Artifact. J Appl Psychol. 1984;69:334.

Weingart SN, Callanan LD, Ship AN, Araonson MD. A Physician-Based Voluntary Reporting System For Adverse Events and Medical Errors. J Gen Intern Med. 2001;16(12):8009-814.

Griffin FA, Resar RK. IHI Global Trigger Tool for Measuring Adverse Events (Second Edition). IHI Innovation Series white paper. Cambridge, MA: Institute for Healthcare Improvement; 2009. (available on www.ihi.org)

Swets JA, Dawes RM, Monahan J. Psychological Science Can Improve Diagnostic Decisions. Psychol Sci Public Interest. 2000;1:1-26.

Kraemer HC. How Many Raters? Toward the Most Reliable Diagnostic Consensus. Stat Med. 1992;11:317-331.

Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. New York: Oxford University Press, 2003.

Youngstrom EA. A Primer on Receiver Operating Characteristic Analysis and Diagnostic Efficiency Statistics for Pediatric Psychology: We Are Ready to ROC. J Pediatr Psychol. 2014;39:204-221.

Youngstrom EA, Meyers OI, Youngstrom JK, et al. Comparing the Effects of Sampling Designs on the Diagnostic Accuracy of Eight Promising Screening Algorithms for Pediatric Bipolar Disorder. Biol Psychiat. 2006;60:1013-1019.

Straus SE, Glasziou P, Richardson WS, et al. Evidence-Based Medicine: How to Practice and Teach EBM. New York: Churchill Livingstone, 2011.

Sutinen R, Kivimaki M, Elovainio E, et al. Associations Between Stress at Work and Attitudes Towards Retirement in Hospital Physicians. Work Stress. 2005;19:177-185.

Van Bogaert P, Clarke S, Roelant E, et al. Impacts of Unit-Level Nurse Practice Environment and Burnout on Nurse-Reported Outcomes: A Multilevel Modelling Approach. J Clin Nurs. 2010;19:1664-1674.

Bakker AB, Demerouti E, Euwema MC. Job Resources Buffer the Impact of Job Demands on Burnout. J Occup Health Psychol. 2005;10:170.

Sari ABA, Shaledon TA, Cracknell A, et al. Sensitivity of Routine System for Reporting Patient Safety Incidents in an NHS Hospital: Retrospective patient care note review. BMJ. 2007;334:79.

Santos MFS, Rodrigues Neto JF, Leite MTDS. Health Education in the Context of Family Health From the User’s Perspective. Interface. 2012;16:315-329.

Laschinger HKS, Leiter MP. The Impact of Nursing Work Environments on Patient Safety Outcomes: The Mediating Role of Burnout/Engagement. J Nurs Adm. 2006;36:259-267.

Gilles I, Mayer M, Courvoisier N, et al. Joint Analyses of Open Comments and Quantitative Data: Added Value in a Job Satisfaction Survey of Hospital Professionals. PloS One. 2017;12:e0173950.

Garcia CDL, Abreau LCD, Ramos JLS, et al. Influence of Burnout on Patient Safety: Systematic Review and Meta-Analysis. Medicina. 2019;55:553.

Blue background with physician hand pointing to a patient file. Laptop to the right of the hand and file.

Additional Files



How to Cite

Pearl, A., Mrozowski, S., & Shapiro, D. E. (2021). Where There’s Smoke: Validating a Nonproprietary Single-Item Burnout-Impacting-Safety Scale. Patient Safety, 3(2), 34–44. https://doi.org/10.33940/culture/2021.6.3



Original Research and Articles
Bookmark and Share