A Neurosurgical Readmissions Reduction Program in an Academic Hospital Leveraging Machine Learning, Workflow Analysis, and Simulation

Abstract

Background Predicting 30-day hospital readmissions is crucial for improving patient outcomes, optimizing resource allocation, and achieving financial savings. Existing studies reporting the development of machine learning (ML) models predictive of neurosurgical readmissions do not report factors related to clinical implementation.

Objectives Train individual predictive models with good performance (area under the receiver operating characteristic curve or AUROC > 0.8), identify potential interventions through semi-structured interviews, and demonstrate estimated clinical and financial impact of these models.

Methods Electronic health records were utilized with five ML methodologies: gradient boosting, decision tree, random forest, ridge logistic regression, and linear support vector machine. Variables of interest were determined by domain experts and literature. The dataset was split divided 80% for training and validation and 20% for testing randomly. Clinical workflow analysis was conducted using semi-structured interviews to identify possible intervention points. Calibrated agent-based models (ABMs), based on a previous study with interventions, were applied to simulate reductions of the 30-day readmission rate and financial costs.

Results The dataset covered 12,334 neurosurgical intensive care unit (NSICU) admissions (11,029 patients); 1,903 spine surgery admissions (1,641 patients), and 2,208 traumatic brain injury (TBI) admissions (2,185 patients), with readmission rate of 13.13, 13.93, and 23.73%, respectively. The random forest model for NSICU achieved best performance with an AUROC score of 0.89, capturing potential patients effectively. Six interventions were identified through 12 semi-structured interviews targeting preoperative, inpatient stay, discharge phases, and follow-up phases. Calibrated ABMs simulated median readmission reduction rates and resulted in 13.13 to 10.12% (NSICU), 13.90 to 10.98% (spine surgery), and 23.64 to 21.20% (TBI). Approximately $1,300,614.28 in saving resulted from potential interventions.

Conclusion This study reports the successful development and simulation of an ML-based approach for predicting and reducing 30-day hospital readmissions in neurosurgery. The intervention shows feasibility in improving patient outcomes and reducing financial losses.

 

Scroll to top

Keywords

EHRs and systems - machine learning - workflows and human interactions - simulation and modeling - clinical decision support system

Background and Significance

Unplanned hospital readmissions refer to any admission postdischarge that are not in the initial plan of care or treatment. Unplanned readmissions are associated with adverse effects such as increased risk of mortality during the patient's hospital stay or shortly after, negatively impacting the quality of care.[1] Thirty-day hospital readmission has long been established as an important and measurable indicator of the quality of care.[2] Unplanned readmissions contribute significantly annually to the cost of the U.S. health care system, costing over $52 billion for all unplanned readmissions and $3.8 million for 30-day readmissions.[3] Financial hospital reimbursement policies have made it critical to identify and act on risk factors for unplanned readmissions.[4] In the fiscal year 2013, the Centers for Medicare and Medicaid Services initiated the Hospital Readmission Reduction Program to encourage hospitals to reduce avoidable readmissions by adjusting the payments based on their performance in reducing readmissions.

Hospital readmission can have a wide range of rates. A systematic review showed significantly high 30-day readmission rates ranging from 6.9 to 23.89% for neurosurgery patients.[5] While there have been quite a few studies identifying risk factors and trends of unplanned readmissions,[6] [7] [8] [9] [10] there are very few interventional studies in neurosurgery actively attempting to reduce 30-day readmission rates.[11] [12] While these interventions have displayed a decrease in the general rate of 30-day unplanned readmissions, the interventions were applied to all participants in the intervention group. The patients were not stratified based on their risk of readmissions, which incurs substantial avoidable expenses and time inefficiencies. It is therefore crucial to identify patients specifically at risk of unplanned readmissions for intervention to better improve patient outcomes and reduce cost.

After identifying potential risk factors, advanced methods of prediction such as machine learning and artificial intelligence (ML/AI) can be helpful to predict hospital readmissions. Literature reports existing development and use of ML-based models in predicting neurosurgical outcomes.[13] [14] There are also developed models to predict 30-day unplanned readmissions in neurosurgery. One study developed an ML model for stroke patients with an area under the receiver operating characteristic curve (AUROC) of 0.74[15] and another study developed a model for patients after pituitary adenoma resection with an AUROC of 0.76.[16] Although these models achieved relatively good performance, these studies have not considered factors related to the clinical implementation through the clinical workflow and decision point analysis. There is research suggesting the potential benefit of clinical implementation of advanced algorithms in reducing unplanned hospital readmissions.[17] [18] However, no such studies have been done in neurosurgery specifically to the best of our knowledge. Moreover, the potential clinical and health impact of the models were not demonstrated in a laboratory or field experiment. All of the above limit the usefulness of the predictive models.

In this study, we reported the development and simulated evaluation of a novel Neurosurgical Readmission Reduction Program (NSRRP) at an adult academic health center in the Midwestern United States. The development of NSRRP utilized not only ML-based approaches and electronic health records (EHRs), but also clinical workflow analysis to understand the context of the decisions. The simulated evaluation utilizes simulation to demonstrate potential clinical and health impact to get leadership support before the actual implementation. Specifically, this study aimed to: (1) train individual predictive models on three patient groups in neurosurgical intensive care unit (NSICU), spine surgery, and traumatic brain injury (TBI) with good performance (AUROC > 0.8), (2) identify decision points and potential interventions through semi-structured interviews, and (3) use agent-based modeling (ABM) to demonstrate potential clinical and health impact.

 

Scroll to top

Methods

Study Setting

This study involved a collaborative effort within the University of Cincinnati College of Medicine (UCCoM) between the Department of Neurosurgery (Neuroinformatics Laboratory), the Department of Biomedical Informatics (Center for Health Informatics), and the University of Cincinnati Medical Center (UCMC). The Department of Neurosurgery provided seed funds to establish the Neuroinformatics Laboratory and partnered with the Center for Health Informatics to develop the data infrastructure. This study was one of the pilot studies to demonstrate the efficacy of the data infrastructure and the synergy that the Neuroinformatics Laboratory could offer. The study was also a part of the 2023 AI Evaluation Showcase, an initiative organized by the American Medical Informatics Association (AMIA). The study followed the three-stage design of the AMIA AI Evaluation Showcase, namely, technical performance, workflow study, and clinical impact. We formed an interdisciplinary team including domain experts (clinicians), a data analyst, a designer, and a statistician to conduct the study and discuss the findings. The protocols for retrospective chart review and clinical workflow analysis were approved by our institutional review board (IRB): #2019-1403 for the retrospective chart review and #2022-0635 for the clinical workflow analysis. It is worth noting that in the workflow study, the identity of the interview participants was removed, and no patient data were collected.

 

Scroll to top

Patient Population

The data were extracted from the institution's EHRs through the Center for Health Informatics, an honest data broker between UCCoM and UCMC. Three patient cohorts, including those from NSICU, spine surgery, and TBI, were identified for the study. NSICU patients were selected from those admitted to the NSICU Department. The spine surgery patients comprised patients who underwent cervical or lumbar surgery based on the International Classification of Diseases procedure codes. Patients in the TBI cohort were identified using neurotrauma flowsheet, diagnosis codes, Diagnostic Related Group codes, and Current Procedural Terminology codes. Detailed inclusion criteria were listed in [Supplementary Material S1] (available in the online version only). The readmissions were limited to inpatient-related admission classes, and admissions marked as deceased in discharge disposition was removed. A 30-day unplanned readmission was defined as an unexpected hospital encounter occurring within 30 days after discharge, which follows a redefined rule in our hospital's EHR system after consulted with our data analytics team.

 

Scroll to top

Machine Learning Models

A three-stage analytic development process was applied to predict 30-day readmissions in neurosurgical patients with disorders of the spine, TBI, and NSICU. Risk factors were first identified from literature research and then expanded as well as categorized by the domain experts in the research team. A subset of risk factors (variables of interest) in each of the three patient cohorts were further determined by the domain experts. Five classic ML classification algorithms including gradient boosting, decision tree, random forest, ridge logistic regression, and linear support vector machine. The selected ML models for comparison offer a diverse range of strengths. Gradient boosting handles complex data and improves weak learners for accurate predictions. Decision tree provides interpretability and capture nonlinear relationships, whereas random forest provides robustness and feature importance estimates. Ridge logistic regression offers simplicity and efficiency as a baseline model, and linear support vector machine was included in finding optimal hyperplanes for robust generalization, particularly in high-dimensional spaces. The dataset was randomly divided into 80% for training and validation, with the remaining 20% reserved for testing. In the training process, a 5-fold cross-validation with hyperparameter tuning was employed. The AUROC was used to determine the model performance. On testing dataset, the model performance was reported using sensitivity (recall), specificity (true negative rate), precision (positive predictive value), and accuracy. To identify all patients who may face unplanned readmission, we sought a threshold based on achieving a sensitivity of one. While specificity may be lower and false positive rate can be higher, our approach is still preferable to a blanket intervention for all identified patients.

 

Scroll to top

Clinical Workflow Analysis

This study aimed to recruit a minimum of 10 participants from three teams: NSICU, spine surgery, and TBI. The purpose of these interviews was to explore potential context and use cases of the readmission predictive model in reducing readmission rates as well as the decision points in the workflow. The interview questionnaires can be found in [Supplementary Material S2] ([Supplementary Figs. S1] and [S2], available in the online version). The participants were asked to describe their work processes, identify decision points, and provide potential use cases. Since the interviews were semi structured, follow-up questions were asked based on the participants' responses. The participants were recruited through convenience sampling, using the research team's network, and snowball sampling, by asking participants to provide names at the end of their interview. In addition to physicians, nurse practitioners and navigators were also primary targets for interviews. At the beginning of each interview, the participants were introduced to the project and verbally consented. After this, each participant received information about the predictive model variables and performance details. All interviews were conducted online; each lasted 30 to 45 minutes and was recorded and transcribed automatically using Microsoft Teams. The transcripts were reviewed and further analyzed to generate workflow diagrams and to summarize possible interventions using the predictive models developed for each patient group.

 

Scroll to top

Simulation-Based Evaluation

The clinical and health impact of the predictive models were estimated in terms of the reduction of the 30-day readmission rate and financial costs using ABM, which is a computational technique simulating the behaviors and interactions of individual agents, or patients in this context, within a system to understand its overall dynamics. In the health care setting, ABM has emerged as a powerful tool to assess clinical impact. We assumed the readmission probability is equally likely within each agent, with a mean calculated from the readmission rate and a standard deviation of 0.0005, ensuring positive probabilities within 30 days after discharge. The study focused on a series of interventions aimed at reducing readmission probabilities for each of the three patient groups with 100 simulations on 30-day readmission. A prior study estimated that completing a series of interventions would result in a reduction of the readmission rate from 5.8 to 2.5% for 416 patients, equivalent to reducing the readmission probability by 0.00149 for patients who received complete intervention within 24 hours after discharge.[12] These interventions, including a preadmission overview, recorded anticipated discharge date, daily communication with the transitional care team through the hospital stay, advance notice before a day of discharge, prescription filling in the hospital, a 30-minute exit consultation and education with a skilled nurse at discharge, and a follow-up telephone call within 2 days after discharge, were designed to address various factors contributing to readmissions.

The simulation models were calibrated in two steps. First, a baseline model was developed to match the readmission rates before and after the intervention in the previous study. Specifically, the Wilcoxon signed-rank test was used to determine if the median simulated readmission rate aligns with the actual readmission rates (5.8–2.5%).[12] Second, the baseline model was extended to simulate the readmission rates of our three patient groups with adjusted parameters. The same test was used to determine if the simulated readmission rates match the actual readmission rates without the intervention. Once the calibration was done, each simulation models ran 100 times to generate the simulated readmission rates after the intervention. The same test was applied to determine significant median differences between the simulated readmission rates before and after the intervention in each patient group.

The financial impact was quantified by calculating the reduction in readmission-related costs used in the previous study. These costs encompass the expenses associated with hospital stays minus the cost of training individuals to implement the interventions. For NSICU, spine, and TBI, the mean costs of hospital stay expense and care transitions coaches for the intervention were estimated to be $38,601.90,[19] $27,926.47,[20] and $20,030.54,[8] respectively. The intervention cost (transition of care) was estimated to be $264.46 in a TBI setting[11] and applied to other two settings since there was no available intervention cost in the literature. The costs were adjusted for inflation from January 2019 to January 2020 to June 2023 in U.S. dollars using the U.S. Bureau of Labor Statistics Consumer Price Index Calculator.[21] The cumulative reduction in readmission rates can directly translate to a decrease in financial burden for the health care system. The estimated cost saving can increase through the predictive models by removing patients who are unlikely readmitted.

 

Scroll to top

 

Scroll to top

Results

The data were extracted for the time span between July 2017 and November 2022. It comprised a total of 12,334 admissions for 11,029 distinct patients in NSICU, 1,903 admissions for 1,641 patients undergoing spine surgery, and 2,208 admissions for 2,185 patients with TBI. The readmission rates for NSICU, spine surgery, and TBI were 13.13, 13.93, and 23.73%, respectively.

Model Performance

In total, 28 unique risk factors were selected from the literature by the domain experts and put into six categories. Our developed model utilized a subset of risk factors as follows: (1) patient demographics (N = 2); (2) medical history (N = 12); (3) diagnosis (N = 5); (4) admission Information (N = 3); (5) hospitalization outcome (N = 3); and (6) discharge disposition (N = 3). The detailed variables are available in [Supplementary Table S1] (available in the online version). Among the 28 unique risk factors, 16 of them were selected by the domain experts and fed into the ML development process. As shown in [Fig. 1A], the random forest model achieved the highest performance with an AUROC score of 0.89. This model demonstrated sensitivity of 1.0, specificity of 0.4, precision of 0.2, and accuracy of 0.48. Top features including discharge disposition, length of stay, number of emergent admissions, age, and chronic obstructive pulmonary disease. Next, or spine 30-day readmission prediction, the literature review identified a total of 32 unique risk factors: (1) sociodemographic factors (N = 6); (2) medical history (N = 14); (3) surgical information (N = 7); (4) hospitalization outcomes (N = 4); (5) discharge disposition (N = 1). The detailed variables are located within [Supplementary Table S2] (available in the online version). Among the 32 risk factors, 17 of them were selected and fed into the ML development process. The results revealed that the random forest model had the highest performance with an AUROC of 0.84 in [Fig. 1B]. The model also achieved a sensitivity of 0.91, specificity of 0.62, precision of 0.28, and accuracy of 0.66. Additionally, we observed that with the same AUROC score, the model demonstrated a sensitivity of 1.00, specificity of 0.28, precision of 0.18, and accuracy of 0.38. The top five features of the predictive model are the discharge disposition, has ICU stay, length of stay, body mass index, and age. Lastly, a total of 24 unique predictors for TBI readmission were identified: (1) sociodemographic factors (N = 7); (2) nature of TBI (N = 8); (3) non-TBI medical cause (N = 5); (4) discharge disposition (N = 4). The detailed variables are attached in [Supplementary Table S3] (available in the online version). Among the 24 risk factors, 13 of them were selected. As presented in [Fig. 1C], the logistic regression model achieved the best performance at a sensitivity of 1.00 with specificity of 0.09, precision of 0.26, and accuracy of 0.31. The top features in the predicting model included discharge disposition, hospital length of stay, age, initial Glasgow Coma Scale score, and brain image results.

Fig. 1 The 30-day readmission prediction performance for NSICU (A), spine surgery (B), and TBI (C). AUC, area under the curve; NSICU, neurosurgical intensive care unit; ROC, receiver operating characteristic; TBI, traumatic brain injury.

 

Scroll to top

Workflow Diagrams and Decision Points

The workflow analysis involved interviews with a total of 12 participants from different teams: three physicians from the spine surgery team; one physician, one nursing navigator, and one medical assistant from the TBI team; and two physicians, two nurse practitioners, one registered nurse, and one social worker from the NSICU team. The participants' identity was removed, and no patient data were collected in this process. The feedback collectively generated six areas for intervention that would benefit from utilization of the predictive models as shown: (1) best practice alert (preoperative), (2) EHR patient main page (inpatient stay), (3) discharge plan (inpatient stay), (4) checklist/reminder (discharge, transfer, downgrade), (5) patient education (discharge), and (6) follow-up plan. NSICU has more areas for intervention because acute care patients can either be discharged directly or downgraded to a lower level of care. [Fig. 2] shows the general workflows of three teams with the potential interventions at the specific point of care (represented by red dots) to reduce readmissions. [Table 1] presents the comprehensive list of potential interventions along with their corresponding use cases at various stages, all of which were derived from the interview process.

Fig. 2 Workflows of NSICU (A), spine (B), and TBI (C) teams with six types of potential interventions: (1) best practice alert, (2) patient main page, (3) discharge plan, (4) checklist reminder, (5) patient education, and (6) follow-up. ICU, intensive care unit; NSICU, neurosurgical intensive care unit. TBI, traumatic brain injury.

Table 1 The list of potential interventions and use cases from the interview

Using cases	Time point	Primary users	Scenarios
Best practice alert	Preoperative	Physicians nursing practitioner	A best practice alert in the form of a pop-up window appears before closing a patient's chart, highlighting the patient's higher risk of readmissions due to specific reasons. This prompt serves as a reminder for health care providers to consider the information while making decisions for the patient
Patient main page	Inpatient stay	Nursing navigator	Nurse navigators routinely access patient accounts throughout the day, including responding to patient inquiries, scheduling postoperative appointments, and monitoring patient care across all stages in EPIC. The display of high readmission rates on the patient's main page would assist nurse navigators in proactively reviewing patient conditions and intervening as necessary
Discharge plan	Inpatient stay intensive care unit stay (when making discharge plan)	All team members	Displaying the readmission rate on patients' dashboards during disposition or discharge planning would be a valuable tool for health care providers. By considering the readmission rates and other relevant variables, health care providers can more effectively determine the appropriate disposition location and schedule the next appointment or extend the days in the hospital for more observation or exams. This approach can optimize patient care and reduce readmission
Checklist reminder	Discharging transferring (downgrading)	Nursing practitioner	A standardized checklist could ensure all the important variables are addressed before discharging a patient to minimize the risk of readmission. This would help providers feel confident and provide better patient outcomes. Furthermore, the checklist would streamline the discharge process, making it more efficient, and improve patient satisfaction
Patient education	Discharging	Nursing navigator	Incorporating families in patient education by providing them with information on potential readmission factors can enhance postdischarge care and possibly mitigate the risk of readmission
Follow-up	Follow-up	Nursing navigator	With this model, the nurse navigator can proactively follow up on patients with a high risk of readmission and implement necessary interventions. She reviews the list of all patients every day, ensuring they have appropriate follow-up care. A flag or alert indicating a patient's readmission risk would be helpful, allowing for timely interventions such as a follow-up call

 

Scroll to top

Simulation

The first step of the model calibration between the simulated and the actual readmission rates before and after the intervention in the baseline model showed p-values of 0.35 and 0.63, indicating indifference (agreement) between the simulated and the empirical scenarios. The second step of the calibration between the median simulated readmission rates derived from three discrete ABMs representing the NSICU, spine, and TBI showed p-values of 0.93, 0.83, and 0.64, respectively. This indicates that no substantial disparities exist between the observed and simulated readmission rates. These results warranted the efficacy of the models, calibrated with specific parameters and configurations, in accurately replicating the observed readmission rates. By applying the calibrated ABMs to the individual patient populations, the simulated median readmission rates exhibit reductions ranging from 13.13 to 10.12% for NSICU, 13.90 to 10.98% for the spine department, and 23.64 to 21.20% for the TBI department. These findings are demonstrated through [Fig. 3], where the box plots graphically depict the outcomes of the simulations. Moreover, the one-sided Wilcoxon signed rank test further strengthens the argument, providing compelling evidence that the median simulated readmission rates following the interventions are indeed significantly lower than those in the absence of intervention. This conclusion is substantiated by the p-values less than 0.0001 for each respective case.

Fig. 3 Boxplots of readmission rate with/without intervention simulated on NSICU (A), spine (B), and TBI (C). NSICU, neurosurgical intensive care unit; TBI, traumatic brain injury.

The cost-saving were summarized in [Table 2]. The savings with intervention were calculated by the readmission cost per discrepant readmitted patient. On the other hand, the intervention spending was calculated by totaling the cost of the transition coach per readmission for intervention. The ML predictive model removed any unlikely readmissions, leading to the adjusted intervention spending, which was the monthly cost of the transition coach per readmission when the sensitivity equals 1.0 in the predictive model. This adjustment narrowed the focus to patients more inclined to face readmission. Finally, the overall savings were calculated in the unadjusted versus adjusted scenario. By integrating the suggested intervention into the operational workflow, a significant amount of roughly $12,548,618.71 was effectively preserved across all three patient groups. Additionally, the use of ML predictions can result in an incremental reduction of approximately $1,300,614.28.

Table 2 Cost analysis estimated from the result of agent-based model on neurosurgical intensive care unit, spine, and traumatic brain injury

Title	NSICU	Spine	TBI
Without ML model
No. of readmissions intervened	12,344	1,903	2,208
No. of readmissions saved	1,250	209	468
Cost per readmission	$38,601.90	$27,926.47	$20,030.54
Saving with Intervention (S)	$14,282,703.00	$1,535,955.85	$1,081,649.16
Intervention spending (IS)	$3,264,494.24	$503,267.38	$583,927.68
Total saving (S)—(IS)	$11,018,208.76	$1,032,688.47	$497,721.48
With ML model
Readmission rate without intervention	13.13%	13.90%	23.64%
Readmission rate with intervention	10.12%	10.98%	21.20%
No. of of readmission intervened	7,906	1,583	2,048
No. of of readmission saved	1,250	209	468
False positive rate	0.600	0.811	0.908
Adjusted intervention spending (AIS)	$2,090,820.76	$418,640.18	$541,614.08
Adjusted total saving (S)—(AIS)	$12,191,882.24	$1,117,315.67	$540,035.08

Abbreviation: ML, machine learning; NSICU, neurosurgical intensive care unit; TBI, traumatic brain injury.

Notes: The top table shows the cost-saving as a result of a hypothetical intervention without using the best-performing ML model; the bottom table shows the cost-saving with the ML model. The differences between IS and AIS come from the use of the ML model.

 

Scroll to top

 

Scroll to top

Discussion

Key Findings

This study highlights the successful application of an ML-based approach in predicting 30-day hospital readmissions among patients with neurosurgical disorders, resulting in satisfactory predictive performance. The study extended the predictive modeling and introduced an analysis of clinical workflows, with the overarching goal of creating decision support tools tailored to mitigate neurosurgical readmissions. Participants of the study reveals a unanimous and robust endorsement of the NSRRP. Participants believe that such an initiative holds the potential to empower health care providers to concentrate their efforts on patients at elevated risk, thereby facilitating timely interventions and ultimately enhancing patient outcomes. Moreover, the study incorporates ABMs capable of reproducing readmission scenarios, enabling the evaluation of intervention effectiveness and the estimation of potential cost reductions. These findings collectively underscore the feasibility and potential impact of deploying advanced analytics in the neurosurgical domain to address hospital readmissions and emphasize the significance of such efforts in the realm of health care improvement.

 

Scroll to top

Implications

The findings of this study showcase the considerable prospects associated with the integration of ML methodologies to predict instances of hospital readmissions among patients undergoing neurosurgical procedures. As reported in a few other studies,[17] [18] the deployment of advanced predictive modeling techniques as decision support tools hold promises in addressing the complex challenge of readmissions, thereby augmenting the quality of patient care. The affirmative feedback received from study participants resonates with the notion that the incorporation of such predictive tools within clinical settings holds the potential for favorable reception among medical practitioners. This receptiveness, in turn, could translate into tangible enhancements in patient outcomes. Furthermore, the efficacy of the ABMs in accurately forecasting the reduction in readmission rates subsequent to the implementation of interventions bears implications for cost-effectiveness. By offering a means to quantitatively estimate the cost savings resulting from intervention strategies, this model contributes to the strategic allocation of resources and health care planning.

 

Scroll to top

Limitations

This study has few limitations. First, the study only used EHR data but no other data sources in the readmission prediction. This might lead to the omission of potential variables that exert influence but are not documented clearly, such as social determinants (or drivers) of health, which have been documented to be an important contributor to hospital readmission.[22] These unrecorded factors could impact the model's accuracy by not being incorporated into the predictive framework. Second, the predictive models were not retrained based on the decision points in the workflow. Some information may not be available at a specific point of care and can affect the overall model performance. Third, while convenience sampling would allow quick access to potential participants, it may lead to bias. We mitigated this bias by combining convenience sampling with snow sampling. Given the small size of the clinical teams in this study setting, we believe that this method can collect enough information to summarize workflow issues and bottlenecks. Last but not least, the current intervention simulations entail uniform probability reduction and intervention cost across all three patient groups. This approach could introduce bias to the results, as each department possesses unique dynamics. Deviations in intervention effectiveness and cost implications might lead to misinterpretations of the model's impact on readmission reduction.

 

Scroll to top

Future Work

Several avenues for future research warrant exploration. First, there is substantial potential for enhancing model performance by integrating additional variables of interest into the existing predictive framework. Incorporating these variables, such as genetic markers or novel biomarkers, could lead to a more accurate and comprehensive prediction model. Moreover, delving into the realm of model explainability is imperative for building trust and understanding in predictive health care applications. The utilization of tools like SHapley Additive exPlanations (SHAP) values can aid in identifying key features and their corresponding weights, shedding light on the decision-making process of the model. Transitioning from admission-based predictions to patient-based predictions presents an intriguing avenue for research. The adoption of deep learning architectures, such as Long Short-Term Memory, could offer improved sequential data analysis capabilities, particularly when dealing with patient history and longitudinal data. Integrating clinical notes extracted through natural language processing as model parameters presents another exciting direction. This integration has the potential to enhance prediction accuracy by harnessing unstructured data within EHRs. Expanding the predictive pipeline to encompass patients with neurotrauma and those in neurocritical care settings would extend the model's clinical applicability. Incorporating novel factors identified through comprehensive literature searches can enrich the predictive capacity of the model. Additionally, tailoring readmission probability adjustments based on demographic variables acknowledges the heterogeneity of patient populations and contributes to more personalized predictions. Lastly, the incorporation of environmental variables within ABM could provide a holistic perspective on patient outcomes, considering the broader context in which health care decisions are made. In conclusion, these proposed avenues for future research collectively represent a substantial step forward in predictive modeling for medical prognosis and offer promising contributions to both clinical practice and research.

 

Scroll to top

 

Scroll to top

Conclusion

In this study, we successfully developed ML-based models for the prediction of 30-day hospital readmissions for neurosurgery patients involved in NSICU, TBIs, and spine surgeries with good predictive capabilities. For the desired goal of creating a clinical support decision tool, we conducted clinical workflow analysis to better tailor the model in specific decision points. Through the use of ABMs, we were able to successfully reproduce readmission situations to evaluate the model's clinical effectiveness as well as financial cost reductions, supporting the clinical efficacy of this intervention. We will continue developing the NSRRP and pilot the decision support tools in clinical routines to demonstrate clinical and health impact through field experiments.

 

Scroll to top

Clinical Relevance Statement

Unplanned hospital readmissions in neurosurgery contributes significantly to suboptimal patient health outcomes as well as increased financial burdens on both the patient and hospital. While previous studies have focused on the sole development of models to predict 30-day readmissions, an important indicator of quality of care, we focus on not only the model development, but also the clinical workflow and potential clinical and financial effect. With multiple steps taken, we believe that this intervention would translate very well in mitigating and reducing neurosurgical hospital readmissions.

 

Scroll to top

Multiple-Choice Questions

1. Which of the following was a finding of semi-structured interviews with providers regarding the use of a machine learning interventional program for reducing hospital readmissions?

   • This program enables efforts to be concentrated on patients with elevated risks

   • This program would help decrease provider burnout and work hours

   • Participants in this study did not endorse support for this type of intervention

   • Clinical workflow analysis did not identify any potential points of intervention

   Correct Answer: The correct answer is option a. According to the results and discussion of this paper, this is the only answer to correctly identify a finding of clinical workflow analysis after performing semi-structured interviews. Providers did not mention any effect on burnout or work hours. Participants did in fact provide unanimous support for this intervention and did identify common points of interventions for our clinical workflow analysis.

2. Which of the following methodologies was implemented in the development and evaluation of our Neurosurgical Readmissions Reduction Program?

   • Machine learning model development

   • Clinical workflow analysis using semi-structured interviews

   • Simulation through agent-based modeling

   • Literature analysis for determining of variables predictive of neurosurgery readmission

   • All of the above

   Correct Answer: The correct answer is option e. Machine learning, clinical workflow analysis, simulations, and literature analysis were used in the development and evaluation of the Neurosurgical Readmission Reduction Program.

 

Scroll to top

 

Scroll to top

Conflict of Interest

None declared.

Acknowledgments

We thank Ms. Shwetha Bindhu, an alumna of UCCoM, for her effort on the TBI literature review, and Mr. Scott Vennemeyer at UCCoM for sharing his knowledge in decision and cost analysis. We also thank the staff in the UC Center for Health Informatics for their effort on data extraction and management for the Neuroinformatics Laboratory. Finally, we thank AMIA 2023 AI Evaluation Showcase for the opportunities to develop and present our work.

Protection of Human and Animal Subjects

The protocols for retrospective chart review and clinical workflow analysis were approved by our IRB: #2019-1403 for the retrospective chart review and #2022-0635 for the clinical workflow analysis. It is worth noting that in the workflow study, the identity of the interview participants was removed, and no patient data were collected.

• References

• 1 Yen HY, Chi MJ, Huang HY. Effects of discharge planning services and unplanned readmissions on post-hospital mortality in older patients: a time-varying survival analysis. Int J Nurs Stud 2022; 128: 104175
• 2 Jha AK, Orav EJ, Epstein AM. Public reporting of discharge planning and rates of readmissions. N Engl J Med 2009; 361 (27) 2637-2645
• 3 Weiss AJ, Jiang HJ. Overview of clinical conditions with frequent and costly hospital readmissions by payer, 2018. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality (US); 2006. Accessed August 9, 2023 at: http://www.ncbi.nlm.nih.gov/books/NBK573265/
• 4 Buchanan CC, Hernandez EA, Anderson JM. et al. Analysis of 30-day readmissions among neurosurgical patients: surgical complication avoidance as key to quality improvement. J Neurosurg 2014; 121 (01) 170-175
• 5 Cusimano MD, Pshonyak I, Lee MY, Ilie G. A systematic review of 30-day readmission after cranial neurosurgery. J Neurosurg 2017; 127 (02) 342-352
• 6 Lopez Ramos C, Brandel MG, Rennert RC. et al. Clinical risk factors and postoperative complications associated with unplanned hospital readmissions after cranial neurosurgery. World Neurosurg 2018; 119: e294-e300
• 7 Sherrod BA, Johnston JM, Rocque BG. Risk factors for unplanned readmission within 30 days after pediatric neurosurgery: a nationwide analysis of 9799 procedures from the American College of Surgeons National Surgical Quality Improvement Program. J Neurosurg Pediatr 2016; 18 (03) 350-362
• 8 Hoffman H, Furst T, Jalal MS, Chin LS. Costs and predictors of 30-day readmissions after craniotomy for traumatic brain injury: a nationwide analysis. J Neurosurg 2019; 133: 875-883
• 9 Cho PG, Kim TH, Lee H, Ji GY, Park SH, Shin DA. Incidence, reasons, and risk factors for 30-day readmission after lumbar spine surgery for degenerative spinal disease. Sci Rep 2020; 10 (01) 12672
• 10 Wadhwa RK, Ohya J, Vogel TD. et al. Risk factors for 30-day reoperation and 3-month readmission: analysis from the Quality and Outcomes Database lumbar spine registry. J Neurosurg Spine 2017; 27 (02) 131-136
• 11 Linton KF, Stamegna C, Zepeda V. et al. The impact of a care transitions intervention on people with brain injuries and their caregivers. Adv Soc Work 2020; 20 (03) 725-738
• 12 Robertson FC, Logsdon JL, Dasenbrock HH. et al. Transitional care services: a quality and safety process improvement program in neurosurgery. J Neurosurg 2018; 128 (05) 1570-1577
• 13 Senders JT, Staples PC, Karhade AV. et al. Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg 2018; 109: 476-486.e1
• 14 Senders JT, Zaki MM, Karhade AV. et al. An introduction and overview of machine learning in neurosurgical care. Acta Neurochir (Wien) 2018; 160 (01) 29-38
• 15 Darabi N, Hosseinichimeh N, Noto A, Zand R, Abedi V. Machine learning-enabled 30-day readmission model for stroke patients. Front Neurol 2021; 12: 638267
• 16 Crabb BT, Hamrick F, Campbell JM. et al. Machine learning-based analysis and prediction of unplanned 30-day readmissions after pituitary adenoma resection: a multi-institutional retrospective study with external validation. Neurosurgery 2022; 91 (02) 263-271
• 17 Romero-Brufau S, Wyatt KD, Boyum P, Mickelson M, Moore M, Cognetta-Rieke C. Implementation of artificial intelligence-based clinical decision support to reduce hospital readmissions at a regional hospital. Appl Clin Inform 2020; 11 (04) 570-577
• 18 Wu CX, Suresh E, Phng FWL. et al. Effect of a real-time risk score on 30-day readmission reduction in Singapore. Appl Clin Inform 2021; 12 (02) 372-382
• 19 Snell KP, Beiter CL, Hall EL. et al. A novel approach to ICU survivor care: a population health quality improvement project. Crit Care Med 2020; 48 (12) e1164-e1170
• 20 Lee NJ, Fields MW, Boddapati V. et al. The risks, reasons, and costs for 30- and 90-day readmissions after fusion surgery for adolescent idiopathic scoliosis. J Neurosurg Spine 2020; 34 (02) 245-253
• 21 CPI Inflation Calculator. Accessed August 9, 2023 at: https://data.bls.gov/cgi-bin/cpicalc.pl
• 22 Feldman SS, Davlyatov G, Hall AG. Toward understanding the value of missing social determinants of health data in care transition planning. Appl Clin Inform 2020; 11 (04) 556-563

Publication History

Received: 08 November 2023

Accepted: 26 April 2024

Article published online:
19 June 2024