Performance of 4 Methods to Assess Health-Related Social Needs.
Academic Article
Overview
abstract
IMPORTANCE: Organizations use health-related social needs (HRSN) information to identify patients in need of referrals, to increase clinician awareness, to improve analytics, and for quality reporting. OBJECTIVE: To contrast the performance of screening questionnaires, natural language processing (NLP) of clinical notes, rule-based computable phenotypes, and machine learning (ML) classification models in measuring HRSNs. DESIGN, SETTING, AND PARTICIPANTS: This cross-sectional study assessed 4 measurement approaches for 5 HRSNs in parallel. Each approach was treated as a screening test. Data included notes from adult patients treated at primary care clinics in 2 health systems in Indianapolis, Indiana, from January 2022 to June 2023. Data were analyzed from December 2024 to February 2025. EXPOSURES: Reference standard instruments measured food insecurity, housing instability, financial strain, transportation barriers, and history of legal problems. Participants completed the HRSN screening questions in the electronic health record (EHR). NLP algorithms, gradient-boosted decision tree ML classifiers, and refined versions of human-defined rule-based computable phenotypes were applied to participants' past 12 months EHR data. MAIN OUTCOMES AND MEASURES: Sensitivity, specificity, area under the curve (AUC), and positive predictive values (PPV) described performance of each approach against the reference standard measures. False-negative rates were used to explore fairness. RESULTS: Data from a total of 1252 adult patients (407 [32.51%] aged 30 to 49 years; 821 [65.58%] female) were assessed, including 94 (7.51%) who identified as Hispanic, 602 (48.08%) as non-Hispanic Black or African American, and 442 (35.30%) as non-Hispanic White. The screening questions method had the strongest overall performance for food insecurity (AUC, 0.94; 95% CI, 0.93-0.95), housing instability (AUC, 0.78; 95% CI, 0.75-0.80), transportation barriers (AUC, 0.77; 95% CI, 0.74-0.79), and legal problems (AUC, 0.81; 95% CI, 0.77-0.85). The screening questions had poor performance for financial strain (AUC, 0.62; 95% CI, 0.60-0.65). The PPV for screening tools ranged from 0.77 to 0.92, indicating utility for individual-level decision-making. NLP and rule-based computable phenotypes had poor performance. ML classification resulted in higher sensitivities than the other methods. False-negative rates indicated differential, unfair performance for all measurement approaches by gender, race and ethnicity, and age groups. CONCLUSIONS AND RELEVANCE: In this cross-sectional study of HRSN measurement, no approach performed strongly for every HRSN, and every approach had indication of unfair performance. These findings suggest that practitioners, health care and public health organizations, researchers, and policymakers who rely on a single method to collect HRSN data will likely underestimate patients' true social burden.
