495 - Validity Evidence for a Novel Leadership Assessment Tool for Pediatric Subspecialty Fellows

Leadership is an essential skill for pediatric subspecialists but a tool that can be used to assess fellows across different subspecialties is lacking. Using previously identified leadership competencies important for graduating critical care fellows, we developed and piloted a new assessment tool.

Objective:

To gather validity evidence for a novel leadership assessment tool when used to assess fellows in different pediatric subspecialties.

Design/Methods:

We performed a cross-sectional study of fellows in multiple subspecialty programs. The assessment tool contained 19 behavior-based items measuring domains of self-awareness, self-management, and team and task management. A minimum of 3 faculty were asked to complete the tool for each fellow. We also collected demographic data on each fellow as well as program director and faculty perceptions on the ease of use and value of the tool. Internal consistency was assessed using Cronbach’s alpha and sources of score variation examined through generalizability theory. We estimated changes in average leadership score (average of item scores) by year of training through linear mixed effect models, allowing for clustering by learner, faculty, program and subspecialty.

Results:

Forty-three fellowships from 21 institutions participated in the study with critical care, child abuse and hospital medicine the most common subspecialties (Table). Most (77%) programs did not have an existing leadership curriculum. Using the tool, 377 faculty performed 565 assessments on 210 fellows with similar numbers of fellows in each training year. Overall, the tool was perceived as valuable for assessing leadership skills. Faculty reported ease in completing the tool and that it covered important leadership domains. The overall alpha for the tool was 0.98, with all items highly correlated with overall score (range 0.75-0.86). The greatest sources of variability in scores were: 1) the learner (34%), 2) the faculty rater (18%), and 3) subspecialty (12%). A minimum of 4 items was required with 1 rater, with fewer items necessary for increasing raters per learner (Fig 1). After adjusting for gender and prior chief residency, the average leadership tool score increased with year of training (Fig 2).

Conclusion(s):

We demonstrated strong validity evidence, with high internal consistency and feasibility, for a novel leadership tool to assess fellows across multiple different pediatric subspecialties. Additional validity evidence should focus on relationships with other leadership measures and use in supporting learning plans and advancement decisions.