Machine Learning Refinement of the NSQIP Risk Calculator: Who Survives the "Hail Mary" Case?
Academic Article
Overview
abstract
BACKGROUND: The American College of Surgeons (ACS) NSQIP risk calculator helps guide operative decision making. In patients with significant surgical risk, it may be unclear whether to proceed with "Hail Mary"-type interventions. To refine predictions, a local interpretable model-agnostic explanations machine (LIME) learning algorithm was explored to determine weighted patient-specific factors' contribution to mortality. STUDY DESIGN: The ACS-NSQIP database was queried for all surgical patients with mortality probability greater than 50% between 2012 and 2019. Preoperative factors (n = 38) were evaluated using stepwise logistic regression; 26 significant factors were used in gradient boosted machine (GBM) modeling. Data were divided into training and testing sets, and model performance was substantiated with 10-fold cross validation. LIME provided individual subject mortality. The GBM-trained model was interpolated to LIME, and predictions were made using the test dataset. RESULTS: There were 6,483 deaths (53%) among 12,248 admissions. GBM modeling displayed good performance (area under the curve = 0.65, 95% CI 0.636-0.671). The top 5 factors (% contribution) to mortality included: septic shock (27%), elevated International Normalized Ratio (22%), ventilator-dependence (14%), thrombocytopenia (14%), and elevated serum creatinine (5%). LIME modeling subset personalized patients by factors and weights on survival. In the entire cohort, mortality positive predictive value with 2 factor combinations was 53.5% (specificity 0.713), 3 combinations 64.2% (specificity 0.835), 4 combinations 72.1% (specificity 0.943), and all 5 combinations 77.9% (specificity 0.993). Conversely, mortality positive predictive value fell to 34% in the absence of 4 factors. CONCLUSIONS: Through the application of machine learning algorithms (GBM and LIME), our model individualized predicted mortality and contributing factors with substantial ACS-NSQIP predicted mortality. USE of machine learning techniques may better inform operative decisions and family conversations in cases of significant surgical risk.
