Download PDF
Am J Perinatol
DOI: 10.1055/a-2008-8598
Original Article

The Application of a Standard Risk Threshold for the Stratification of Maternal Morbidity among Population Subgroups

Mark A. Clapp 
1   Department of Obstetrics and Gynecology, Massachusetts General Hospital, Boston, Massachusetts
,
Kaitlyn E. James
1   Department of Obstetrics and Gynecology, Massachusetts General Hospital, Boston, Massachusetts
,
Thomas H. Mccoy Jr.
2   Center for Quantitative Health, Massachusetts General Hospital, Boston, Massachusetts
3   Department of Psychiatry, Massachusetts General Hospital, Boston, Massachusetts
,
Roy H. Perlis
2   Center for Quantitative Health, Massachusetts General Hospital, Boston, Massachusetts
3   Department of Psychiatry, Massachusetts General Hospital, Boston, Massachusetts
,
Anjali J. Kaimal
1   Department of Obstetrics and Gynecology, Massachusetts General Hospital, Boston, Massachusetts
4   Department of Population Medicine, Harvard Medical School, Boston, Massachusetts
› Author Affiliations Funding M.A.C. serves on the scientific advisory board and holds private equity for Delfina Health. M.A.C.'s work was supported by the American Association of Obstetricians and Gynecologists Foundation/American Board of Obstetricians and Gynecologists Research Scholar Award. R.H.P. has received fees for consulting or service on scientific advisory boards for Genomind, Psy Therapeutics, Outermost Therapeutics, RID Ventures, and Takeda. He has received patent royalties from Massachusetts General Hospital. He holds equity in Psy Therapeutics and Outermost Therapeutics. The funder had no role in the design, analysis, writing, or submission of this work.
› Further Information
    Permissions and Reprints

Abstract

Objective The aim of this study was to determine if a universally applied risk score threshold for severe maternal morbidity (SMM) resulted in different performance characteristics among subgroups of the population.

Study Design This is a retrospective cohort study of deliveries that occurred between July 1, 2016, and June 30, 2020, in a single health system. We examined the performance of a validated comorbidity score to stratify SMM risk in our cohort. We considered the risk score that was associated with the highest decile of predicted risk as a “screen positive” for morbidity. We then used this same threshold to calculate the sensitivity and positive predictive value (PPV) of this “highest risk” designation among subgroups of the overall cohort based on the following characteristics: age, race/ethnicity, parity, gestational age, and planned mode of delivery.

Results In the overall cohort of 53,982 women, the C-statistic was 0.755 (95% confidence interval [CI], 0.741–0.769) and calibration plot demonstrated that the risk score was well calibrated. The model performed less well in the following groups: non-White or Hispanic (C-statistic, 0.734; 95% CI, 0.712–0.755), nulliparas (C-statistic, 0.735; 95% CI, 0.716–0.754), term deliveries (C-statistic, 0.712; 95% CI, 0.694–0.729), and planned vaginal delivery (C-statistic, 0.728; 95% CI, 0.709–0.747). There were differences in the PPVs by gestational age (7.8% term and 29.7% preterm) and by planned mode of delivery (8.7% vaginal and 17.7% cesarean delivery). Sensitivities were lower in women who were <35 years (36.6%), non-White or Hispanic (40.7%), nulliparous (38.9%), and those having a planned vaginal delivery (40.9%) than their counterparts.

Conclusion The performance of a risk score for SMM can vary by population subgroups when using standard thresholds derived from the overall cohort. If applied without such considerations, such thresholds may be less likely to identify certain subgroups of the population that may be at increased risk of SMM.

Key Points

  • Predictive risk models are helpful at condensing complex information into an interpretable output.

  • Model performance may vary among different population subgroups.

  • Prediction models should be examined for their potential to exacerbate underlying disparities.

Keywords

maternal morbidity - delivery - prediction - risk - risk stratification


Publication History

Received: 25 February 2022

Accepted: 30 December 2022

Accepted Manuscript online:
06 January 2023

Article published online:
16 February 2023

© 2023. Thieme. All rights reserved.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

 

  • References

  • 1 Kleinrouweler CE, Cheong-See FM, Collins GS. et al. Prognostic models in obstetrics: available, but far from applicable. Am J Obstet Gynecol 2016; 214 (01) 79-90.e36
    CrossrefPubMedGoogle Scholar
  • 2 Ledford H. Millions of black people affected by racial bias in health-care algorithms. Nature 2019; 574 (7780): 608-609
    CrossrefPubMedGoogle Scholar
  • 3 Benjamin R. Assessing risk, automating racism. Science 2019; 366 (6464): 421-422
    CrossrefPubMedGoogle Scholar
  • 4 Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366 (6464): 447-453
    CrossrefPubMedGoogle Scholar
  • 5 Wynants L, van Smeden M, McLernon DJ, Timmerman D, Steyerberg EW, Van Calster B. Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. Three myths about risk thresholds for prediction models. BMC Med 2019; 17 (01) 192
    CrossrefPubMedGoogle Scholar
  • 6 ACOG Committee Opinion No. ACOG Committee Opinion No. 649: racial and ethnic disparities in obstetrics and gynecology. Obstet Gynecol 2015; 126 (06) e130-e134
    CrossrefPubMedGoogle Scholar
  • 7 Jain JA, Temming LA, D'Alton ME. et al. SMFM special report: putting the “M” back in MFM: reducing racial and ethnic disparities in maternal morbidity and mortality: a call to action. Am J Obstet Gynecol 2018; 218 (02) B9-B17
    CrossrefPubMedGoogle Scholar
  • 8 Park Y, Hu J, Singh M. et al. Comparison of methods to reduce bias from clinical prediction models of postpartum depression. JAMA Netw Open 2021; 4 (04) e213909
    CrossrefPubMedGoogle Scholar
  • 9 Bateman BT, Mhyre JM, Hernandez-Diaz S. et al. Development of a comorbidity index for use in obstetric patients. Obstet Gynecol 2013; 122 (05) 957-965
    CrossrefPubMedGoogle Scholar
  • 10 Leonard SA, Kennedy CJ, Carmichael SL, Lyell DJ, Main EK. An expanded obstetric comorbidity scoring system for predicting severe maternal morbidity. Obstet Gynecol 2020; 136 (03) 440-449
    CrossrefPubMedGoogle Scholar
  • 11 Escobar GJ, Liu VX, Schuler A, Lawson B, Greene JD, Kipnis P. Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med 2020; 383 (20) 1951-1960
    CrossrefPubMedGoogle Scholar
  • 12 Venkatesh KK, Strauss RA, Grotegut CA. et al. Machine learning and statistical models to predict postpartum hemorrhage. Obstet Gynecol 2020; 135 (04) 935-944
    CrossrefPubMedGoogle Scholar
  • 13 Luchristt D, Brown O, Pidaparti M, Kenton K, Lewicky-Gaupp C, Miller ES. Predicting obstetrical anal sphincter injuries in patients who undergo vaginal birth after cesarean delivery. Am J Obstet Gynecol 2021; 225 (02) 173.e1-173.e8
    CrossrefPubMedGoogle Scholar
  • 14 Admon LK, Winkelman TNA, Zivin K, Terplan M, Mhyre JM, Dalton VK. Racial and ethnic disparities in the incidence of severe maternal morbidity in the United States, 2012-2015. Obstet Gynecol 2018; 132 (05) 1158-1166
    CrossrefPubMedGoogle Scholar
  • 15 Centers for Disease Control and Prevention. . How does CDC identify severe maternal morbidity? Published December 26, 2019. Accessed April 21, 2020 at: https://www.cdc.gov/reproductivehealth/maternalinfanthealth/smm/severe-morbidity-ICD.htm
    PubMed
  • 16 DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44 (03) 837-845
    CrossrefPubMedGoogle Scholar