643 - Creating a Framework and Application to Compare Anthropometric Cleaning Methodologies
Monday, May 1, 2023
9:30 AM – 11:30 AM ET
Poster Number: 643 Publication Number: 643.456
Hannah De los Santos, MITRE, Schenectady, NY, United States; Max Olivier, The MITRE Corporation, McLean, VA, United States; Carrie Daymont, Penn State College of Medicine, Hershey, PA, United States; Daniel L. Chudnov, MITRE Corporation, Mclean, VA, United States
Lead Data Scientist MITRE Schenectady, New York, United States
Background: In clinical care and research, there are often multiple existing anthropometric cleaning algorithms for a given problem. As such, it is important to identify the best-performing algorithm for a specific task and to facilitate modifications to improve their performance. Objective: We sought to create an adaptable framework to visualize and evaluate quantitative differences in algorithm results and the processes that drive their results. We developed the Anthropometric Cleaning Methods Explorer (ACME), to evaluate algorithms that “clean” infant anthropometric data from electronic health records. Design/Methods: ACME has two primary functions: “Explore” allows for a high-level exploration of results, including interactive visualizations of measurements excluded by each method and a comparison of method performance to expert annotation, if available. “Compare” allows users to focus on a singular subject and “step through” the intermediate processes for a methodology, which allows the user to understand the decision points for any given method. We implemented five published methods for evaluation in ACME and tested the application using outpatient measurement data from a large tertiary care center, randomly selecting 10,000 children 0 to 3.5 years for evaluation. We compared the overall proportion of excluded measurements by each method. After identifying differences in results for particular subject data types, we used the “Compare” function to identify processes within the algorithms that drove these differences. This framework was deployed in R as a Shiny application. The algorithms for comparison were first implemented in R using their paper descriptions or translated from available code. Results: There were 200,000 measurements for 10,000 subjects. The proportions of values identified as implausible varied widely (p < 0.001) (Table 1). Reasons for differences between results for certain data types were identified. For example, one method often marked values as implausible for final values with large time gaps from the last value, due to its conditional growth cutoffs. This knowledge can be used to loosen cutoffs over time.
Conclusion(s): ACME creates an adaptable framework that allows the comparison of anthropometric cleaning algorithm results and the processes that drive these results. This framework can support choosing the best performing algorithm for a given task and improving the performance of existing algorithms. In future, we would like to apply ACME to additional age ranges and hone its features accordingly.
