644 - Cross Institutional Validation of a Machine Learning Predictive Analytic Tool to Differentiate Necrotizing Enterocolitis from Spontaneous Intestinal Perforation

Monday, May 1, 2023

9:30 AM – 11:30 AM ET

Poster Number: 644
Publication Number: 644.456

Allison C. Lure, Nationwide Children's Hospital, Columbus, OH, United States; Young-Joong Kim, Duke University, Pratt School of Engineering, Vancouver, BC, Canada; Xinsong Du, University of Florida, Gainesville, FL, United States; Raechel Irons, Golisano Children’s Hospital of Southwest Florida, Fort Myers, FL, United States; Daniel R. Gipson, University of Florida College of Medicine, Gainesville, FL, United States; Dominick J.. Lemas, University of Florida College of Medicine, Gainesville, FL, United States; Erik Black, University of Florida College of Nursing, Gainesville, FL, United States; Janice A.. Taylor, University of Florida College of Medicine, Gainesville, FL, United States; Jonathan L. Slaughter, Nationwide Children's Hospital/The Ohio State University, Columbus, OH, United States; Diomel de la cruz, University of Florida College of Medicine, Gainesville, FL, United States; Josef Neu, University of Florida, Gainesville, FL, United States

Presenting Author(s)

Allison C. Lure, MD (she/her/hers)

Neonatology-Perinatology Fellow
Nationwide Children's Hospital
Columbus, Ohio, United States

Background: We previously built and validated random forest (RF) and ridge logistic regression (LR) machine learning models that were able to differentiate between necrotizing enterocolitis (NEC) and spontaneous intestinal perforation (SIP) prior to surgery at a single institution. However, such models need multi-institutional validation.

Objective: To determine the feasibility of multi-institutional cross validation of a novel machine learning predictive analytic tool.

Design/Methods: Neonates who underwent drain placement or laparotomy for an intestinal injury at two institutions were categorized into NEC or SIP via bowel visualization intraoperatively or on autopsy. RF and LR models were trained on single institutional data and were cross validated using the other institution’s data. A third model pooled data in a 70:30 training-test split. The test subset was further stratified by institution. Input variables included birth history, maternal history, medications, imaging findings, and laboratory values. Variables were limited to those that would be available prior to or at the time of diagnosis, and before surgery proceeded in order to be clinically useful.

Results: Of the 40 patients from Institution 1, 72.5% (n=29) had NEC, and 27.5% (n=11) had SIP. Of the 30 patients from Institution 2, 53.3% (n=16) had NEC and 46.7% (n=14) had SIP. Cross validated models had decreased area under the curve (AUC) compared to the original, and the pooled model further reduced AUC (Figure 1). Accuracy of models ranged from 50% to 91% (Figure 2).

Conclusion(s):

Cross validation models had reduced performance compared to the single institution model. This was likely multifactorial due to institutional differences partially attributed to patient population and clinical and laboratory practices. While the models outperformed random chance, they may not surpass clinical judgement without further modification. Attempting to account for institutional differences, a new model was trained on 70% of the data from both hospitals and tested on the remaining 30%. However, when test data were further stratified by hospital, the numbers were too small and random iterations of the model greatly affected the results.

In order to improve machine learning modeling in neonatology, standardization of lab values and imaging findings, in addition to larger data sets are needed. A predictive analytic tool built at one institution may not be generalizable across multiple institutions. However, a model constructed using data from a neonatal research network spanning multiple hospitals could result in higher AUC and improved accuracy.