678 - Optimizing a sepsis prediction tool for use in a new environment

Due to the morbidity associated with hospital acquired sepsis, clinicians have developed tools and algorithms to identify sepsis earlier. At Monroe Carell Jr. Children’s Hospital at Vanderbilt (MCJCHV), we adopted an electronic medical record (EMR)-based tool designed at another children's hospital and adapted it to accurately predict sepsis in our setting.

Objective:

Optimize a previously designed, EMR-based sepsis prediction tool for use in a new environment.

Design/Methods:

Our team conducted multiple PDSA cycles to determine how best to integrate an EMR-based sepsis prediction tool and improve performance in our acute care areas. A best practice advisory was designed to fire when a patient’s sepsis risk score, calculated by the prediction tool, reached a critical threshold. Tool performance was measured using statistical process control charts. Percent of sepsis alerts that were true positives was determined by a two-physician scoring team; true positives were those where the alert fired and the patient met Goldstein criteria for sepsis. Clinicians who received the sepsis alert were asked for feedback following events. Emergency transfers due to septic shock were monitored to determine tool performance.

Results:

The sepsis prediction tool was initially validated using retrospective patient charts and sepsis billing codes. Changes were made to the variables mapped to the tool. The alert was then monitored silently for six months before going live on one acute care floor. Multiple changes were made to improve predictions, and based on feedback from clinicians, to limit false positives. Following go-live, 397 alerts fired from March through December, 2022. We excluded vital signs obtained in the emergency department and revised variable mapping to prevent erroneous scores. We excluded behavioral health patients as well as trauma patients in the first 72 hours of admission. These changes decreased complaints about overfires but did not improve the positive predictive value of the tool. We then changed how length of stay was calculated to ensure points for recent admission were limited to the first 12 hours of care. Percent of true positive alerts increased from 51% to 65% (p=.14) with a trend toward greater accuracy. Review of severe sepsis patients for whom the alert did not fire confirmed the tool had adequate sensitivity and revealed gaps in documentation of sepsis risk factors.

Conclusion(s):

To adopt a prediction tool created at an outside institution, teams must attend to idiosyncrasies within their system. Changes made to the sepsis tool adopted at MCJCHV led to improved accuracy of the tool.