637 - Use of Natural Language Processing to Identify Future Imaging and Follow-up Plans in Provider Notes

The safety and quality of ambulatory clinic visits relies on successful completion of the recommended plan of care, which may include future visits, diagnostic testing, prescriptions, or referrals. Assessment of follow-up plan completion begins with identifying the planned components and ends with verifying whether they were completed within the recommended timeframe. Traditional approaches for assessing completion rates of actionable provider recommendations, as documented in the clinical narrative portion of a visit note, are labor intensive. 

Objective:

To assess the feasibility of using Natural Language Processing (NLP) of provider visit notes to identify whether future imaging and clinic visits were components of the follow-up plan of care after an ambulatory visit to a pediatric orthopedic clinic.

Design/Methods:

We screened a random sample of 4570 outpatient pediatric orthopedic clinic notes from a 6-month period during 2022 at a tertiary care urban medical center. The notes were a mix of resident, attending and physician assistant notes, with no standardized templates. Using an NLP graphic user interface (GUI) we trained a document classifier to identify notes in which the provider planned for future imaging and/or return clinic appointments. We used keywords and regular expressions to generate a training and a validation set using 15% of the notes.

We report the sensitivity and specificity of the models for identifying whether imaging or return clinic appointments were components of the follow-up care plan. Using the prevalence in our sample, we also calculated positive and negative predictive values.

Results:

Plans for future imaging were documented in 513/685 documents (75%; 95% CI: 71-78%), of which 472 (92%) provided a clear timeline and 41 (8%) addressed future imaging in a negation form (i.e., not required). Follow-up clinic visits with a timeline were documented in 505/685 (74%) and on an “as needed” basis in 137/685 (20%).

Performance of the NLP models and Receiver Operating Curves (ROC) are presented in Figure 1. Our model performance had 95.2% sensitivity and 64.2% specificity for imaging plans and 95.4% sensitivity and 67.3% specificity for future follow-up visits.

A screenshot of the GUI for manual screening of imaging/follow-up plan surveillance is provided in Figure 2.

Conclusion(s):

Detailed data regarding need for future imaging and follow-up visits exist in a substantial number of provider notes. NLP provides a feasible and sensitive method to extract these data, which may have significant applications to clinical and safety improvement efforts.