Validation of Mortality Data Sources Compared to the National Death Index in the Healthcare Integrated Research Database

Sponholtz T., Jamal‐Allial A., Vojjala S., Papazian A., Eshete B., Paullin M., Mahmoudpour S., Verpillat P., Beachler D.

ABSTRACTPurposeWe investigated time trends in validation performance characteristics for six sources of death data available within the Healthcare Integrated Research Database (HIRD) over 8 years.MethodsWe conducted a secondary analysis of a cohort of advanced cancer patients with linked National Death Index (NDI) data identified in the HIRD between 2010 and 2018. We calculated sensitivity, specificity, positive predictive value, and negative predictive value for six sources of death status data and an algorithm combining data from available sources using NDI data as the reference standard. Measures were calculated for each year of the study including all members in the cohort for at least 1 day in that year.ResultsWe identified 27 396 deaths from any source among 40 692 cohort members. Between 2010 and 2018, the sensitivity of the Death Master File (DMF) decreased from 0.77 (95% CI = 0.76, 0.79) to 0.12 (95% CI = 0.11, 0.14). In contrast, the sensitivity of online obituary data increased from 0.43 (95% CI = 0.41, 0.45) in 2012 to 0.71 (95% CI = 0.68, 0.73) in 2018. The sensitivity of the composite algorithm remained above 0.83 throughout the study period. PPV was observed to be high from 2010 to 2016 and decrease thereafter for all sources. Specificity and NPV remained at high levels throughout the study.ConclusionsWe observed that the sensitivity of mortality data sources compared with the NDI could change substantially between 2010 and 2018. Other validation characteristics were less variable. Combining multiple sources of mortality data may be necessary to achieve adequate performance particularly for multiyear studies.

Garry E.M., Weckstein A.R., Quinto K., Bradley M.C., Lasky T., Chakravarty A., Leonard S., Vititoe S.E., Easthausen I.J., Rassen J.A., Gatto N.M.

Algorithms for classification of inpatient COVID-19 severity are necessary for confounding control in studies using real-world data.Using Healthverity chargemaster and claims data, we selected patients hospitalized with COVID-19 between April 2020 and February 2021, and classified them by severity at admission using an algorithm we developed based on respiratory support requirements (supplemental oxygen or non-invasive ventilation, O2/NIV, invasive mechanical ventilation, IMV, or NEITHER). To evaluate the utility of the algorithm, patients were followed from admission until death, discharge, or a 28-day maximum to report mortality risks and rates overall and by stratified by severity. Trends for heterogeneity in mortality risk and rate across severity classifications were evaluated using Cochran-Armitage and Logrank trend tests, respectively.Among 118 117 patients, the algorithm categorized patients in increasing severity as NEITHER (36.7%), O2/NIV (54.3%), and IMV (9.0%). Associated mortality risk (and 95% CI) was 11.8% (11.6-12.0%) overall and increased with severity [3.4% (3.2-3.5%), 11.5% (11.3-11.8%), 47.3% (46.3-48.2%); p < 0.001]. Mortality rate per 1000 person-days (and 95% CI) was 15.1 (14.9-15.4) overall and increased with severity [5.7 (5.4-6.0), 14.5 (14.2-14.9), 32.7 (31.8-33.6); p < 0.001].As expected, we observed a positive association between the algorithm-defined severity on admission and 28-day mortality risk and rate. Although performance remains to be validated, this provides some assurance that this algorithm may be used for confounding control or stratification in treatment effect studies.

Lerman M.H., Holmes B., St Hilaire D., Tran M., Rioth M., Subramanian V., Winzeler A.M., Brown T.

PURPOSE This study tested whether a composite mortality score could overcome gaps and potential biases in individual real-world mortality data sources. Complete and accurate mortality data are necessary to calculate important outcomes in oncology, including overall survival. However, in the United States, there is not a single complete and broadly applicable mortality data source. It is further likely that available data sources are biased in their coverage of sex, race, age, and socioeconomic status (SES). METHODS Six individual real-world data sources were combined to develop a high-quality composite mortality score. The composite score was benchmarked against the gold standard for mortality data, the National Death Index. Subgroup analyses were then conducted to evaluate the completeness and accuracy by sex, race, age, and SES. RESULTS The composite mortality score achieved a sensitivity of 94.9% and specificity of 92.8% compared with the National Death Index, with concordance within 1 day of 98.6%. Although some individual data sources show significant coverage gaps related to sex, race, age, and SES, the composite score maintains high sensitivity (84.6%-96.1%) and specificity (77.9%-99.2%) across subgroups. CONCLUSION A composite score leveraging multiple scalable sources for mortality in the real-world setting maintained strong sensitivity, specificity, and concordance, including across sex, race, age, and SES subgroups.

Buonanno P., Puca M.

Conway R.B., Armistead M.G., Denney M.J., Smith G.S.

Abstract Background Though electronic health record (EHR) data have been linked to national and state death registries, such linkages have rarely been validated for an entire hospital system's EHR. Objectives The aim of the study is to validate West Virginia University Medicine's (WVU Medicine) linkage of its EHR to three external death registries: the Social Security Death Masterfile (SSDMF), the national death index (NDI), the West Virginia Department of Health and Human Resources (DHHR). Methods Probabilistic matching was used to link patients to NDI and deterministic matching for the SSDMF and DHHR vital statistics records (WVDMF). In subanalysis, we used deaths recorded in Epic (n = 30,217) to further validate a subset of deaths captured by the SSDMF, NDI, and WVDMF. Results Of the deaths captured by the SSDMF, 59.8 and 68.5% were captured by NDI and WVDMF, respectively; for deaths captured by NDI this co-capture rate was 80 and 78%, respectively, for the SSDMF and WVDMF. Kappa statistics were strongest for NDI and WVDMF (61.2%) and NDI and SSDMF (60.6%) and weakest for SSDMF and WVDMF (27.9%). Of deaths recorded in Epic, 84.3, 85.5, and 84.4% were captured by SSDMF, NDI, and WVDMF, respectively. Less than 2% of patients' deaths recorded in Epic were not found in any of the death registries. Finally, approximately 0.2% of “decedents” in any death registry re-emerged in Epic at least 6 months after their death date, a very small percentage and thus further validating the linkages. Conclusion NDI had greatest validity in capturing deaths in our EHR. As a similar, though slightly less capture and agreement rate in identifying deaths is observed for SSDMF and state vital statistics records, these registries may be reasonable alternatives to NDI for research and quality assurance studies utilizing entire EHRs from large hospital systems. Investigators should also be aware that there will be a very tiny fraction of “dead” patients re-emerging in the EHR.

Zhang Y., Wang S., Hermann A., Joly R., Pathak J.

There is a scarcity in tools to predict postpartum depression (PPD). We propose a machine learning framework for PPD risk prediction using data extracted from electronic health records (EHRs).Two EHR datasets containing data on 15,197 women from 2015 to 2018 at a single site, and 53,972 women from 2004 to 2017 at multiple sites were used as development and validation sets, respectively, to construct the PPD risk prediction model. The primary outcome was a diagnosis of PPD within 1 year following childbirth. A framework of data extraction, processing, and machine learning was implemented to select a minimal list of features from the EHR datasets to ensure model performance and to enable future point-of-care risk prediction.The best-performing model uses from clinical features related to mental health history, medical comorbidity, obstetric complications, medication prescription orders, and patient demographic characteristics. The model performances as measured by area under the receiver operating characteristic curve (AUC) are 0.937 (95% CI 0.912 - 0.962) and 0.886 (95% CI 0.879-0.893) in the development and validation datasets, respectively. The model performances were consistent when tested using data ending at multiple time periods during pregnancy and at childbirth.The prevalence of PPD in the study data represented a treatment prevalence and is likely lower than the illness prevalence.EHRs and machine learning offer the ability to identify women at risk for PPD early in their pregnancy. This may facilitate scalable and timely prevention and intervention, reducing negative outcomes and the associated burden.

Navar A.M., Peterson E.D., Steen D.L., Wojdyla D.M., Sanchez R.J., Khan I., Song X., Gold M.E., Pencina M.J.

Despite its documented undercapture of mortality data, the US Social Security Administration Death Master File (SSDMF) is still often used to provide mortality end points in retrospective clinical studies. Changes in death data reporting to SSDMF in 2011 may have further affected the reliability of mortality end points, with varying consequences over time and by state.To evaluate the reliability of mortality rates in the SSDMF in a cohort of patients with atherosclerotic cardiovascular disease (ASCVD).This observational analysis used the IBM MarketScan Medicare and commercial insurance databases linked to mortality information from the SSDMF. Adults with ASCVD who had a clinical encounter between January 1, 2012, and December 31, 2013, at least 2 years of follow-up, and from states with 1000 or more eligible adults with ASCVD were included in the study. Data analysis was conducted between April 18 and May 21, 2018.Kaplan-Meier analyses were conducted to estimate state-level mortality rates for adults with ASCVD, stratified by database (commercial or Medicare). Constant hazards of mortality by state were tested, and individual state Kaplan-Meier curves for temporal changes were evaluated. For states in which the hazard of death was constant over time, mortality rates for adults with ASCVD were compared with state-level, age group-specific overall mortality rates in 2012, as reported by the National Center for Health Statistics (NCHS).This study of mortality data of 667 516 adults with ASCVD included 274 005 adults in the commercial insurance database cohort (171 959 male [62.8%] and median [interquartile range (IQR)] age of 58 [52-62] years) and 393 511 in the Medicare database cohort (245 366 male [62.4%] and median [IQR] age of 76 [70-83] years). Of the 41 states included, 11 states (26.8%) in the commercial cohort and 18 states (43.9%) in the Medicare cohort had a change in the hazard of death after 2012. Among states with constant hazard, state-level mortality rates using the SSDMF ranged widely, from 0.06 to 1.30 per 100 person-years (commercial cohort) and from 0.83 to 6.07 per 100 person-years (Medicare cohort). Variability between states in mortality estimates for adults with ASCVD using SSDMF data greatly exceeded variability in overall mortality from the NCHS. No correlation was found between NCHS mortality estimates and those from the SSDMF (ρ = 0.29 [P = .06] for age 55-64 years; ρ = 0.18 [P = .27] for age 65-74 years).The SSDMF appeared to markedly underestimate mortality rates, with variable undercapture among states and over time; this finding suggests that SSDMF data are not reliable and should not be used alone by researchers to estimate mortality rates.

Levin M.A., Lin H., Prabhakar G., McCormick P.J., Egorova N.N.

Objective To determine the reliability of the Social Security Death Master File (DMF) after the November 2011 changes limiting the inclusion of state records. Data Sources Secondary data from the DMF, New York State (NYS) and New Jersey (NJ) Vital Statistics (VS), and institutional data warehouse. Study Design Retrospective study. Two cohorts: discharge date before November 1, 2011, (pre-2011) or after (post-2011). Death in-hospital used as gold standard. NYS VS used for out-of-hospital death. Sensitivity, specificity, Cohen's Kappa, and 1-year survival calculated. Data Collection Methods Patients matched to DMF using Social Security Number, or date of birth and Soundex algorithm. Patients matched to NY and NJ VS using probabilistic linking. Principal Findings 97 069 patients January 2007-March 2016: 39 075 pre-2011; 57 994 post-2011. 3777 (3.9 percent) died in-hospital. DMF sensitivity for in-hospital death 88.9 percent (κ = 0.93) pre-2011 vs 14.8 percent (κ = 0.25) post-2011. DMF sensitivity for NY deaths 74.6 percent (κ = 0.71) pre-2011 vs 26.6 percent (κ = 0.33) post-2011. DMF sensitivity for NJ deaths 62.6 percent (κ = 0.64) pre-2011 vs 10.8 percent (κ = 0.15) post-2011. DMF sensitivity for out-of-hospital death 71.4 percent pre-2011 (κ = 0.58) vs 28.9 percent post-2011 (κ = 0.34). Post-2011, 1-year survival using DMF data was overestimated at 95.8 percent, vs 86.1 percent using NYS VS. Conclusions The DMF is no longer a reliable source of death data. Researchers using the DMF may underestimate mortality.

Trevethan R.

Within the context of screening tests, it is important to avoid misconceptions about sensitivity, specificity, and predictive values. In this article, therefore, foundations are first established concerning these metrics along with the first of several aspects of pliability that should be recognized in relation to those metrics. Clarification is then provided about the definitions of sensitivity, specificity, and predictive values and why researchers and clinicians can misunderstand and misrepresent them. Arguments are made that sensitivity and specificity should usually be applied only in the context of describing a screening test’s attributes relative to a reference standard; that predictive values are more appropriate and informative in actual screening contexts, but that sensitivity and specificity can be used for screening decisions about individual people if they are extremely high; that predictive values need not always be high and might be used to advantage by adjusting the sensitivity and specificity of screening tests; that, in screening contexts, researchers should provide information about all four metrics and how they were derived; and that, where necessary, consumers of health research should have the skills to interpret those metrics effectively for maximum benefit to clients and the healthcare system.

Rhee C., Dantes R., Epstein L., Murphy D.J., Seymour C.W., Iwashyna T.J., Kadri S.S., Angus D.C., Danner R.L., Fiore A.E., Jernigan J.A., Martin G.S., Septimus E., Warren D.K., Karcz A., et. al.

Estimates from claims-based analyses suggest that the incidence of sepsis is increasing and mortality rates from sepsis are decreasing. However, estimates from claims data may lack clinical fidelity and can be affected by changing diagnosis and coding practices over time.To estimate the US national incidence of sepsis and trends using detailed clinical data from the electronic health record (EHR) systems of diverse hospitals.Retrospective cohort study of adult patients admitted to 409 academic, community, and federal hospitals from 2009-2014.Sepsis was identified using clinical indicators of presumed infection and concurrent acute organ dysfunction, adapting Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria for objective and consistent EHR-based surveillance.Sepsis incidence, outcomes, and trends from 2009-2014 were calculated using regression models and compared with claims-based estimates using International Classification of Diseases, Ninth Revision, Clinical Modification codes for severe sepsis or septic shock. Case-finding criteria were validated against Sepsis-3 criteria using medical record reviews.A total of 173 690 sepsis cases (mean age, 66.5 [SD, 15.5] y; 77 660 [42.4%] women) were identified using clinical criteria among 2 901 019 adults admitted to study hospitals in 2014 (6.0% incidence). Of these, 26 061 (15.0%) died in the hospital and 10 731 (6.2%) were discharged to hospice. From 2009-2014, sepsis incidence using clinical criteria was stable (+0.6% relative change/y [95% CI, -2.3% to 3.5%], P = .67) whereas incidence per claims increased (+10.3%/y [95% CI, 7.2% to 13.3%], P < .001). In-hospital mortality using clinical criteria declined (-3.3%/y [95% CI, -5.6% to -1.0%], P = .004), but there was no significant change in the combined outcome of death or discharge to hospice (-1.3%/y [95% CI, -3.2% to 0.6%], P = .19). In contrast, mortality using claims declined significantly (-7.0%/y [95% CI, -8.8% to -5.2%], P < .001), as did death or discharge to hospice (-4.5%/y [95% CI, -6.1% to -2.8%], P < .001). Clinical criteria were more sensitive in identifying sepsis than claims (69.7% [95% CI, 52.9% to 92.0%] vs 32.3% [95% CI, 24.4% to 43.0%], P < .001), with comparable positive predictive value (70.4% [95% CI, 64.0% to 76.8%] vs 75.2% [95% CI, 69.8% to 80.6%], P = .23).In clinical data from 409 hospitals, sepsis was present in 6% of adult hospitalizations, and in contrast to claims-based analyses, neither the incidence of sepsis nor the combined outcome of death or discharge to hospice changed significantly between 2009-2014. The findings also suggest that EHR-based clinical data provide more objective estimates than claims-based data for sepsis surveillance.

Skopp N.A., Smolenski D.J., Schwesinger D.A., Johnson C.J., Metzger-Abamukong M.J., Reger M.A.

Accurate knowledge of the vital status of individuals is critical to the validity of mortality research. National Death Index (NDI) and NDI-Plus are comprehensive epidemiological resources for mortality ascertainment and cause of death data that require additional user validation. Currently, there is a gap in methods to guide validation of NDI search results rendered for active duty service members. The purpose of this research was to adapt and evaluate the CDC National Program of Cancer Registries (NPCR) algorithm for mortality ascertainment in a large military cohort.We adapted and applied the NPCR algorithm to a cohort of 7088 service members on active duty at the time of death at some point between 2001 and 2009. We evaluated NDI validity and NDI-Plus diagnostic agreement against the Department of Defense's Armed Forces Medical Examiner System (AFMES).The overall sensitivity of the NDI to AFMES records after the application of the NPCR algorithm was 97.1%. Diagnostic estimates of measurement agreement between the NDI-Plus and the AFMES cause of death groups were high.The NDI and NDI-Plus can be successfully used with the NPCR algorithm to identify mortality and cause of death among active duty military cohort members who die in the United States.

Soowamber M.L., Granton J.T., Bavaghar-Zaeimi F., Johnson S.R.

da Graca B., Filardo G., Nicewander D.

> In November 2011, the Social Security Administration removed ≈5% of death records from its Death Master File and started excluding ≈40% of new death records, having determined that data submitted electronically by states cannot be publicly shared. Before this determination, the Death Master File provided an accessible source of national vital status data with a short time lag and high specificity and sensitivity and was routinely used by healthcare researchers and hospitals to determine study participants’ survival and to monitor postdischarge outcomes. Its effective loss means comparative effectiveness studies will be unnecessarily delayed, more costly, or unfeasible. Likewise, timely identification and correction of poor hospital performance will be more difficult, undermining the safety and quality of care and threatening hospital financing as the Centers for Medicare and Medicaid launch the Readmissions Reduction Program in October 2012 and link reimbursement to 30-day mortality under the Value-Based Purchasing Program in 2013. In summary, the action of the Social Security Administration will substantially hamper healthcare research and quality. We describe the origins of the Death Master File and the basis for excluding electronically submitted state data. We then examine the consequences for healthcare research and operations, consider alternative sources, and evaluate possible mechanisms to restore a timely national data source. On November 1, 2011, the Social Security Administration (SSA) removed ≈5% of the data in its publicly available Death Master File (DMF) and stopped reporting ≈40% of new deaths.1 The SSA explained that it had determined that §205(r) of the Social Security Act (added by the Act of April 20, 1983)2 prohibits the disclosure of state records that the SSA has been including in the public version of the DMF since 2002.1 This is a “demise of a vital resource”3 that will hamper healthcare outcomes research, as well as …

Chase H.C.