Association between Subspecialty In-Training Examination Scores and First Attempt Results in Neonatal-Perinatal Medicine Boards

Abstract

Objective

No multicenter study has examined the relationship between subspecialty in-training examination (SITE) scores and certification exam (CE) outcomes in pediatric subspecialties. This study addresses this gap by investigating whether performance on the Neonatal-Perinatal Medicine (NPM) SITE predicts first-attempt results on the NPM-CE administered by the American Board of Pediatrics.

Study Design

A multicenter, retrospective cohort study was conducted across NPM fellowship programs in the United States. Data from 326 fellows across 15 programs who took the NPM-CE between 2014 and 2024 were analyzed. De-identified SITE scores and CE results were collected via REDCap. Year-specific Z-scores for SITE performance were calculated, and mixed-model generalized linear equations and logistic regression were used to assess the odds ratio (OR) and 95% confidence intervals (CI) for CE failure, adjusting for clustering within programs and program characteristics.

Results

Out of 326 fellows in this cohort, 36 failed the CE. Fellows with an average SITE score of < 55% correct answers in the first 2 years of training had four times higher odds of failing the NPM-CE compared with those with ≥ 55% (OR: 4.28; 95% CI: 1.77, 10.33; p < 0.001). Similarly, those with an average SITE Z-score ≤ −1.0 had 12 times higher odds of CE failure compared with those with Z-scores > − 1.0 (OR: 12.12; 95% CI: 5.39, 27.25; p < 0.001). Fellowship program characteristics did not significantly influence CE outcomes.

Conclusion

Performance on the NPM SITE during the first 2 years of fellowship predicts first-attempt NPM-CE results. Early identification of at-risk fellows through SITE performance may enable targeted educational interventions to improve certification outcomes. This study highlights the utility of SITE as a predictive tool in pediatric subspecialty training.

Keywords

SITE - NPM board results - NPMCE-CE - multicenter study - medical education

• References

• 1 Babbott SF, Beasley BW, Hinchey KT, Blotzer JW, Holmboe ES. The predictive validity of the internal medicine in-training examination. Am J Med 2007; 120 (08) 735-740
• 2 Collichio FA, Hess BJ, Muchmore EA. et al. Medical knowledge assessment by hematology and medical oncology in-training examinations are better than program director assessments at predicting subspecialty certification examination performance. J Cancer Educ 2017; 32 (03) 647-654
• 3 Grabovsky I, Hess BJ, Haist SA. et al. The relationship between performance on the infectious diseases in-training and certification examinations. Clin Infect Dis 2015; 60 (05) 677-683
• 4 Indik JH, Duhigg LM, McDonald FS. et al. Performance on the cardiovascular in-training examination in relation to the ABIM cardiovascular disease certification examination. J Am Coll Cardiol 2017; 69 (23) 2862-2868
• 5 Jurich D, Duhigg LM, Plumb TJ. et al. Performance on the nephrology in-training examination and ABIM nephrology certification examination outcomes. Clin J Am Soc Nephrol 2018; 13 (05) 710-717
• 6 Juul D, Flynn FG, Gutmann L. et al. Association between performance on neurology in-training and certification examinations. Neurology 2013; 80 (02) 206-209
• 7 Leigh TM, Johnson TP, Pisacano NJ. Predictive validity of the American Board of Family Practice In-Training Examination. Acad Med 1990; 65 (07) 454-457
• 8 Lohr KM, Clauser A, Hess BJ. et al; American College of Rheumatology Committee on Rheumatology Training and Workforce Issues. Performance on the adult rheumatology in-training examination and relationship to outcomes on the rheumatology certification examination. Arthritis Rheumatol 2015; 67 (11) 3082-3090
• 9 Rollins LK, Martindale JR, Edmond M, Manser T, Scheld WM. Predicting pass rates on the American Board of Internal Medicine certifying examination. J Gen Intern Med 1998; 13 (06) 414-416
• 10 Brateanu A, Yu C, Kattan MW, Olender J, Nielsen C. A nomogram to predict the probability of passing the American Board of Internal Medicine examination. Med Educ Online 2012; 17 (01) 18810
• 11 Kay C, Jackson JL, Frank M. The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. Acad Med 2015; 90 (01) 100-104
• 12 Rayamajhi S, Dhakal P, Wang L, Rai MP, Shrotriya S. Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?. BMC Med Educ 2020; 20 (01) 79
• 13 McCrary HC, Colbert-Getz JM, Poss WB, Smith BK. A systematic review of the relationship between in-training examination scores and specialty board examination scores. J Grad Med Educ 2021; 13 (01) 43-57
• 14 McCaskill QE, Kirk JJ, Barata DM, Wludyka PS, Zenni EA, Chiu TT. USMLE step 1 scores as a significant predictor of future board passage in pediatrics. Ambul Pediatr 2007; 7 (02) 192-195
• 15 Welch TR, Olson BG, Nelsen E, Beck Dallaghan GL, Kennedy GA, Botash A. United States Medical Licensing Examination and American Board of Pediatrics certification examination results: does the residency program contribute to trainee achievement. J Pediatr 2017; 188: 270-274.e3
• 16 Althouse LA, McGuinness GA. The in-training examination: an analysis of its predictive value on performance on the general pediatrics certification examination. J Pediatr 2008; 153 (03) 425-428
• 17 Atsawarungruangkit A. Relationship of residency program characteristics with pass rate of the American Board of Internal Medicine certifying exam. Med Educ Online 2015; 20: 28631
• 18 Chase LH, Highbaugh-Battle AP, Buchter S. Residency factors that influence pediatric in-training examination score improvement. Hosp Pediatr 2012; 2 (04) 210-214
• 19 Moons KG, Altman DG, Reitsma JB. et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015; 162 (01) W1–73
• 20 Hemingway H, Croft P, Perel P. et al; PROGRESS Group. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ 2013; 346: e5595
• 21 Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008; 54 (01) 17-23
• 22 Steyerberg EW, Moons KG, van der Windt DA. et al; PROGRESS Group. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013; 10 (02) e1001381
• 23 Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One 2015; 10 (03) e0118432
• 24 Muthusi JK, Young PW, Mboya FO, Mwalili SM. %diag_test: a generic SAS macro for evaluating diagnostic accuracy measures for multiple diagnostic tests. BMC Med Inform Decis Mak 2025; 25 (01) 21
• 25 Shu CH, Zebda R, Espinosa C. et al. Early prediction of mortality and morbidities in VLBW preterm neonates using machine learning. Pediatr Res 2025; 97 (06) 2056-2064
• 26 Withiam-Leitch M, Olawaiye A. Resident performance on the in-training and board examinations in obstetrics and gynecology: implications for the ACGME outcome project. Teach Learn Med 2008; 20 (02) 136-142
• 27 Lingenfelter BM, Jiang X, Schnatz PF, O'Sullivan DM, Minassian SS, Forstein DA. CREOG in-training examination results: contemporary use to predict ABOG written examination outcomes. J Grad Med Educ 2016; 8 (03) 353-357
• 28 Juul D, Schneidman BS, Sexson SB. et al. Relationship between resident-in-training examination in psychiatry and subsequent certification examination performances. Acad Psychiatry 2009; 33 (05) 404-406
• 29 Harmouche E, Goyal N, Pinawin A, Nagarwala J, Bhat R. USMLE scores predict success in ABEM Initial Certification: a multicenter study. West J Emerg Med 2017; 18 (03) 544-549
• 30 O'Neill TR, Peabody MR, Song H. The predictive validity of the National Board of Osteopathic Medical Examiners' COMLEX-USA examinations with regard to outcomes on American Board of Family Medicine Examinations. Acad Med 2016; 91 (11) 1568-1575
• 31 Ost SR, Wells D, Goedecke PJ, Tolley EA, Kleinman M, Thompson NS. Relationship between standardized test scores and board certification exams in a combined internal medicine/pediatrics residency program. Cureus 2021; 13 (02) e13567
• 32 Aeder L, Fogel J, Schaeffer H. Pediatric board review course for residents “at risk.”. Clin Pediatr (Phila) 2010; 49 (05) 450-456
• 33 Langenau EE, Fogel J, Schaeffer HA. Correlation between an email based board review program and American Board of Pediatrics General Pediatrics certifying examination scores. Med Educ Online 2009; 14: 18