Download PDF
Am J Perinatol 2022; 39(02): 195-203
DOI: 10.1055/s-0040-1715530
Original Article

Validation of an Instrument for Real-Time Assessment of Neonatal Intubation Skills: A Randomized Controlled Simulation Study

Lindie J.M.K. Kuijpers‡   *
1   Department of Neonatology, Radboud University Medical Center Amalia Children's Hospital, Nijmegen, The Netherlands
,
Mathijs Binkhorst  ‡   *
1   Department of Neonatology, Radboud University Medical Center Amalia Children's Hospital, Nijmegen, The Netherlands
,
Nicole K. Yamada
2   Division of Neonatal and Developmental Medicine, Department of Pediatrics, Center for Advanced Pediatric and Perinatal Education (CAPE), Stanford University School of Medicine, Palo Alto, California
,
Romy N. Bouwmeester
1   Department of Neonatology, Radboud University Medical Center Amalia Children's Hospital, Nijmegen, The Netherlands
,
Arno F.J. van Heijst
1   Department of Neonatology, Radboud University Medical Center Amalia Children's Hospital, Nijmegen, The Netherlands
,
Louis P. Halamek
2   Division of Neonatal and Developmental Medicine, Department of Pediatrics, Center for Advanced Pediatric and Perinatal Education (CAPE), Stanford University School of Medicine, Palo Alto, California
,
Marije Hogeveen
1   Department of Neonatology, Radboud University Medical Center Amalia Children's Hospital, Nijmegen, The Netherlands
› Author Affiliations Funding The study was partially supported by the Endowment for the Center for Advanced Pediatric and Perinatal Education (CAPE). The project was also supported by the Agency for Healthcare Research and Quality (grant number P30HS023506). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
› Further Information
    Permissions and Reprints

Abstract

Objective This study aimed to evaluate the construct validity and reliability of real-time assessment of a previously developed neonatal intubation scoring instrument (NISI).

Study Design We performed a randomized controlled simulation study at a simulation-based research and training facility. Twenty-four clinicians experienced in neonatal intubation (“experts”) and 11 medical students (“novices”) performed two identical elective intubations on a neonatal patient simulator. Subjects were randomly assigned to either the intervention group, receiving predefined feedback between the two intubations, or the control group, receiving no feedback. Using the previously developed NISI, all intubations were assessed, both in real time and remotely on video. Construct validity was evaluated by (1) comparing the intubation performances, expressed as percentage scores, with and without feedback, and (2) correlating the intubation performances with the subjects' level of experience. The intrarater reliability, expressed as intraclass correlation coefficient (ICC), of real-time assessment compared with video-based assessment was determined.

Results The intervention group contained 18 subjects, the control group 17. Background characteristics and baseline intubation scores were comparable in both groups. The median (IQR) change in percentage scores between the first and second intubation was significantly different between the intervention and control group (11.6% [4.7–22.8%] vs. 1.4% [0.0–5.7%], respectively; p = 0.013). The 95% CI for this 10.2% difference was 2.2 to 21.4%. The subjects' experience level correlated significantly with their percentage scores (Spearman's R = 0.70; p <0.01). ICC's were 0.95 (95% CI: 0.89–0.97) and 0.94 (95% CI: 0.89–0.97) for the first and second intubation, respectively.

Conclusion Our NISI has construct validity and is reliable for real-time assessment.

Key Points

  • Our neonatal intubation scoring instrument has construct validity.

  • Our instrument can be reliably employed to assess neonatal intubation skills directly in real time.

  • It is suitable for formative assessment, i.e., providing direct feedback during procedural training.

Keywords

education - intubation - neonatal - simulation-based training - scoring instrument

Note

This study was approved by the Institutional Review Board Committee at Stanford University (#44321). All subjects provided informed written consent. The ISRCTN registry trial ID is ISRCTN47516971.


Authors' Contributions

Data collection was done by L.J.M.K.K., N.K.Y., and L.P.H. Data analysis and interpretation was done by L.J.M.K.K., M.B., and M.H. Initial manuscript was drafted by L.J.M.K.K., and M.B. The manuscript was critically reviewed and finally approved by all the authors. All authors agree to be accountable for all the aspects of the work, had full access to all of the data, and can take responsibility for the integrity and accuracy of data analysis and reporting. Study conceptualization and design was done by all the authors.


* These authors contributed equally to this work.


Supplementary Material



Publication History

Received: 07 May 2020

Accepted: 07 July 2020

Article published online:
08 September 2020

© 2020. Thieme. All rights reserved.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

 

  • References

  • 1 Klotz JJ, Dooley-Hash SL, House JB, Andreatta PB. Pediatric and neonatal intubation training gap analysis: instruction, assessment, and technology. Simul Healthc 2014; 9 (06) 377-383
    CrossrefPubMedGoogle Scholar
  • 2 Wyllie JP. Neonatal endotracheal intubation. Arch Dis Child Educ Pract Ed 2008; 93 (02) 44-49
    CrossrefPubMedGoogle Scholar
  • 3 Weiner GM, Zaichkin J. eds. Textbook of Neonatal Resuscitation. 7th ed. Elk Grove Village, IL: American Academy of Pediatrics and American Heart Association; 2016
    CrossrefGoogle Scholar
  • 4 O'Donnell CP, Kamlin CO, Davis PG, Morley CJ. Endotracheal intubation attempts during neonatal resuscitation: success rates, duration, and adverse effects. Pediatrics 2006; 117 (01) e16-e21
    CrossrefPubMedGoogle Scholar
  • 5 Dongara AR, Modi JJ, Nimbalkar SM, Desai RG. Proficiency of residents and fellows in performing neonatal intubation. Indian Pediatr 2014; 51 (07) 561-564
    CrossrefPubMedGoogle Scholar
  • 6 Haubner LY, Barry JS, Johnston LC. et al. Neonatal intubation performance: room for improvement in tertiary neonatal intensive care units. Resuscitation 2013; 84 (10) 1359-1364
    CrossrefPubMedGoogle Scholar
  • 7 Hatch LD, Grubb PH, Lea AS. et al. Endotracheal intubation in neonates: a prospective study of adverse safety events in 162 infants. J Pediatr 2016; 168: 62-66.e6
    CrossrefPubMedGoogle Scholar
  • 8 Doglioni N, Cavallin F, Zanardo V, Trevisanuto D. Intubation training in neonatal patients: a review of one trainee's first 150 procedures. J Matern Fetal Neonatal Med 2012; 25 (08) 1302-1304
    CrossrefPubMedGoogle Scholar
  • 9 Andreatta PB, Klotz JJ, Dooley-Hash SL, Hauptman JG, Biddinger B, House JB. Performance-based comparison of neonatal intubation training outcomes: simulator and live animal. Adv Neonatal Care 2015; 15 (01) 56-64
    CrossrefPubMedGoogle Scholar
  • 10 Downes KJ, Narendran V, Meinzen-Derr J, McClanahan S, Akinbi HT. The lost art of intubation: assessing opportunities for residents to perform neonatal intubation. J Perinatol 2012; 32 (12) 927-932
    CrossrefPubMedGoogle Scholar
  • 11 Sawyer T, Foglia E, Hatch LD. et al. Improving neonatal intubation safety: a journey of a thousand miles. J Neonatal Perinatal Med 2017; 10 (02) 125-131
    CrossrefPubMedGoogle Scholar
  • 12 Leone TA, Rich W, Finer NN. Neonatal intubation: success of pediatric trainees. J Pediatr 2005; 146 (05) 638-641
    CrossrefPubMedGoogle Scholar
  • 13 Ernst KD, Cline WL, Dannaway DC. et al. Weekly and consecutive day neonatal intubation training: comparable on a pediatrics clerkship. Acad Med 2014; 89 (03) 505-510
    CrossrefPubMedGoogle Scholar
  • 14 O'Donnell CPF. Intubation difficulty in neonatology: are you experienced?. Arch Dis Child Fetal Neonatal Ed 2019; 104 (05) F458-F460
    CrossrefPubMedGoogle Scholar
  • 15 Andreatta PB, Dooley-Hash SL, Klotz JJ, Hauptman JG, Biddinger B, House JB. Retention curves for pediatric and neonatal intubation skills after simulation-based training. Pediatr Emerg Care 2016; 32 (02) 71-76
    CrossrefPubMedGoogle Scholar
  • 16 DeMeo SD, Katakam L, Goldberg RN, Tanaka D. Predicting neonatal intubation competency in trainees. Pediatrics 2015; 135 (05) e1229-e1236
    CrossrefPubMedGoogle Scholar
  • 17 Moussa A, Luangxay Y, Tremblay S. et al. Videolaryngoscope for teaching neonatal endotracheal intubation: a randomized controlled trial. Pediatrics 2016; 137 (03) e20152156
    CrossrefPubMedGoogle Scholar
  • 18 Hatala R, Cook DA, Zendejas B, Hamstra SJ, Brydges R. Feedback for simulation-based procedural skills training: a meta-analysis and critical narrative synthesis. Adv Health Sci Educ Theory Pract 2014; 19 (02) 251-272
    CrossrefPubMedGoogle Scholar
  • 19 Bouwmeester RN, Binkhorst M, Yamada NK. et al. Appraisal of a scoring instrument for training and testing neonatal intubation skills. Arch Dis Child Fetal Neonatal Ed 2019; 104 (05) F521-F527
    CrossrefPubMedGoogle Scholar
  • 20 van der Heide PA, van Toledo-Eppinga L, van der Heide M, van der Lee JH. Assessment of neonatal resuscitation skills: a reliable and valid scoring system. Resuscitation 2006; 71 (02) 212-221
    CrossrefPubMedGoogle Scholar
  • 21 Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15 (02) 155-163
    Google Scholar
  • 22 Jones SR, Carley S, Harrison M. An introduction to power and sample size estimation. Emerg Med J 2003; 20 (05) 453-458
    CrossrefPubMedGoogle Scholar
  • 23 Kelleher DC, Carter EA, Waterhouse LJ, Parsons SE, Fritzeen JL, Burd RS. Effect of a checklist on advanced trauma life support task performance during pediatric trauma resuscitation. Acad Emerg Med 2014; 21 (10) 1129-1134
    CrossrefPubMedGoogle Scholar
  • 24 Johnston L, Sawyer T, Nishisaki A. et al; INSPIRE Research Network. Neonatal intubation competency assessment tool: development and validation. Acad Pediatr 2019; 19 (02) 157-164
    CrossrefPubMedGoogle Scholar
  • 25 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33 (01) 159-174
    CrossrefPubMedGoogle Scholar
  • 26 Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005; 85 (03) 257-268
    CrossrefPubMedGoogle Scholar