Spectrum bias in algorithms derived by artificial intelligence: a case study in detecting aortic stenosis using electrocardiograms
Aims
Spectrum bias can arise when a diagnostic test is derived from study populations with different disease spectra than the target population, resulting in poor generalizability. We used a real-world artificial intelligence (AI)-derived algorithm to detect severe aortic stenosis (AS) to experimentally assess the effect of spectrum bias on test performance.
Methods and results
All adult patients at the Mayo Clinic between 1 January 1989 and 30 September 2019 with transthoracic echocardiograms within 180 days after electrocardiogram (ECG) were identified. Two models were developed from two distinct patient cohorts: a whole-spectrum cohort comparing severe AS to any non-severe AS and an extreme-spectrum cohort comparing severe AS to no AS at all. Model performance was assessed. Overall, 258 607 patients had valid ECG and echocardiograms pairs. The area under the receiver operator curve was 0.87 and 0.91 for the whole-spectrum and extreme-spectrum models, respectively. Sensitivity and specificity for the whole-spectrum model was 80% and 81%, respectively, while for the extreme-spectrum model it was 84% and 84%, respectively. When applying the AI-ECG derived from the extreme-spectrum cohort to patients in the whole-spectrum cohort, the sensitivity, specificity, and area under the curve dropped to 83%, 73%, and 0.86, respectively.
Conclusion
While the algorithm performed robustly in identifying severe AS, this study shows that limiting datasets to clearly positive or negative labels leads to overestimation of test performance when testing an AI algorithm in the setting of classifying severe AS using ECG data. While the effect of the bias may be modest in this example, clinicians should be aware of the existence of such a bias in AI-derived algorithms.
Top-30
Journals
|
1
|
|
|
International Journal of Arrhythmia
1 publication, 7.69%
|
|
|
Journal of the American Society of Echocardiography
1 publication, 7.69%
|
|
|
Revista española de cardiología (English ed.)
1 publication, 7.69%
|
|
|
Revista Espanola de Cardiologia
1 publication, 7.69%
|
|
|
Canadian Journal of Cardiology
1 publication, 7.69%
|
|
|
Cochrane Database of Systematic Reviews
1 publication, 7.69%
|
|
|
BIO Web of Conferences
1 publication, 7.69%
|
|
|
Cureus
1 publication, 7.69%
|
|
|
BMJ Digital Health & AI
1 publication, 7.69%
|
|
|
International Journal of Cardiovascular Sciences
1 publication, 7.69%
|
|
|
Shiraz E Medical Journal
1 publication, 7.69%
|
|
|
Pharmaceutics
1 publication, 7.69%
|
|
|
1
|
Publishers
|
1
2
3
4
|
|
|
Elsevier
4 publications, 30.77%
|
|
|
Springer Nature
2 publications, 15.38%
|
|
|
American Society of Echocardiography
1 publication, 7.69%
|
|
|
Wiley
1 publication, 7.69%
|
|
|
EDP Sciences
1 publication, 7.69%
|
|
|
BMJ
1 publication, 7.69%
|
|
|
Sociedade Brasileira de Cardiologia
1 publication, 7.69%
|
|
|
Brieflands
1 publication, 7.69%
|
|
|
MDPI
1 publication, 7.69%
|
|
|
1
2
3
4
|
- We do not take into account publications without a DOI.
- Statistics recalculated weekly.