Psychometric Evaluation of the Modified Neonatal Resuscitation Program^® Adherence Assessment Tool When Utilized for In Situ Simulation and Telesimulation Scenarios

Objective: Neonatal resuscitation is a high acuity, low occurrence event that requires precision, teamwork, and rapid decision-making. Simulation using NRP^® guidelines allows teams to build competency and preparedness, improving neonatal outcomes. Accurate, reliable performance evaluation during simulation is essential to the provision of meaningful feedback. Evidence supporting the reliability and validity of existing tools when used in telesimulation (TS) is limited. We sought to evaluate the psychometric properties of the modified NRP^® (mNRP^®) tool when assessing interprofessional team performance in onsite and TS environments. Study Design: We employed a methodological design to conduct secondary analyses of data from a larger study using simulation to enhance resuscitation skills as assessed by the mNRP^®. Item-level data from 96 simulations was subjected to Classical Test Theory-based item analyses including evaluation of item difficulty, discrimination, and item-to-total correlation; reliability; and exploratory factor analysis (EFA). Interrater reliability (IRR) between novice and expert raters was assessed for a subset of cases. Results: Item difficulties reflected a desirable mix of difficulty in endorsement, suggesting that items capture a range of guideline adherence. Most item discrimination (range: 0.05-0.81) and corrected item-total correlation (range: 0.005-0.68) values had moderate-to-strong, positive correlations with total scores, indicating discriminative ability. EFA yielded three-and-four-components for the onsite and TS groups, respectively. Cronbach’s alpha was 0.76 (onsite) or 0.78 (TS). Lowest vs. highest mNRP^® quartiles differed significantly in both formats (p<0.001 for each), supporting construct validity. IRR ranged from 0.5 to 0.9, supporting moderate to good agreement between novice and expert raters. Conclusions: Findings provide evidence supporting the reliability and validity of the mNRP^® tool when applied in both the in situ and TS settings. IRR was acceptable for expert and novice evaluators. This analysis provides additional validity evidence for the mNRP^® when used to evaluate interprofessional team performance in both onsite and TS formats.