719 - Testing inter-rater reliability of a novel checklist for evaluating the shared decision making process in consultation for extremely preterm delivery

Parents facing extremely preterm delivery must partner with neonatologists and obstetricians to make decisions for the care of their child. Shared decision-making (SDM) between parents and the medical team is recommended to facilitate integration of parental values with physician’s medical knowledge to develop the best course of action but is difficult to accomplish. An objective scoring tool may be useful to aid in teaching and assessing trainee performance and parent satisfaction with shared decision making.

Objective:

To test inter-rater reliability and internal consistency of an original literature-based checklist of SDM practices of neonatologists counseling a standardized patient facing possible extremely preterm delivery.

Design/Methods:

The checklist was developed after literature review and based on the researchers’ clinical experience. It consists of information topics and the SDM process. The SDM component of the checklist has 3 subparts of observed behaviors rated as “yes/no.” Neonatologists from 4 U.S. centers were video recorded performing mock antenatal consultations for a patient facing possible delivery after premature rupture of membranes at  22 weeks gestation. Each consultation was evaluated using the checklist by 4 “experts” (reviewers involved in checklist development) and 4 “novices” (reviewers not involved in development of the checklist). Intraclass correlation coefficient (ICC) and Cronbach’s alpha were calculated to assess inter-rater agreement and internal consistency, respectively. Initial review demonstrated moderate agreement between expert reviewers, however agreement was poor among novice reviewers.

Modifications were made to the checklist based on reviewer feedback. Additional structured training on checklist items was provided to all reviewers. Video recorded consults were re-evaluated using the revised checklist. Additional consults were completed and included in subsequent evaluation and are included in the results.

Results:

Component ICCs for the SDM checklist were calculated for inter-expert and inter-novice raters as shown. Overall, the checklist was moderately reliable for all reviewers in aggregate. There were differences between subcategories within classes of reviewer, as well as between reviewer classes for the same subcategory. Expert ratings were more reliable than novices. Exclusion of six items deemed most difficult to assess improved ICC. Internal consistency of the checklist was excellent (Cronbach’s alpha 0.93).

 

Conclusion(s):

This novel checklist for evaluating shared decision-making shows moderate reliability. Evaluation of validity is planned.