Open Access

Scientific Reports

, volume 13 , issue 1 , publication number 4171

Medical image captioning via generative pretrained transformers

Alexander Selivanov ^{1, 2}

Oleg Y Rogov ¹

Daniil Chesakov ^{1, 3}

Artem Shelmanov ^{1, 3}

Irina Fedulova ²

Dmitry V. Dylov ¹

Hide authors affiliations Show authors affiliations: 3 affiliations

Skolkovo Institute of Science and technology, Moscow, Russia |

Philips (Russia), Moscow, Russia |

AIRI, Moscow, Russia |

Publication type: Journal Article

Publication date: 2023-03-13

Springer Nature

Scientific Reports

scimago Q1

wos Q1

SJR: 0.874

CiteScore: 6.7

Impact factor: 3.9

ISSN: 20452322

DOI: 10.1038/s41598-023-31223-5

Copy DOI

PubMed ID: 36914733

Multidisciplinary

Abstract

The proposed model for automatic clinical image caption generation combines the analysis of radiological scans with structured patient information from the textual records. It uses two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The generated textual summary contains essential information about pathologies found, their location, along with the 2D heatmaps that localize each pathology on the scans. The model has been tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO, and the results measured with natural language assessment metrics demonstrated its efficient applicability to chest X-ray image captioning.

Found

Top-30

Journals

	1 2 3 4
Lecture Notes in Computer Science	Lecture Notes in Computer Science, 4, 5.88% Lecture Notes in Computer Science 4 publications, 5.88%
Engineering Applications of Artificial Intelligence	Engineering Applications of Artificial Intelligence, 3, 4.41% Engineering Applications of Artificial Intelligence 3 publications, 4.41%
Lecture Notes in Networks and Systems	Lecture Notes in Networks and Systems, 3, 4.41% Lecture Notes in Networks and Systems 3 publications, 4.41%
Applied Sciences (Switzerland)	Applied Sciences (Switzerland), 2, 2.94% Applied Sciences (Switzerland) 2 publications, 2.94%
Multimedia Tools and Applications	Multimedia Tools and Applications, 2, 2.94% Multimedia Tools and Applications 2 publications, 2.94%
IEEE Access	IEEE Access, 2, 2.94% IEEE Access 2 publications, 2.94%
Electronics (Switzerland)	Electronics (Switzerland), 2, 2.94% Electronics (Switzerland) 2 publications, 2.94%
European Radiology	European Radiology, 1, 1.47% European Radiology 1 publication, 1.47%
JAMA network open	JAMA network open, 1, 1.47% JAMA network open 1 publication, 1.47%
ACM Computing Surveys	ACM Computing Surveys, 1, 1.47% ACM Computing Surveys 1 publication, 1.47%
Engineering Reports	Engineering Reports, 1, 1.47% Engineering Reports 1 publication, 1.47%
Library Hi Tech	Library Hi Tech, 1, 1.47% Library Hi Tech 1 publication, 1.47%
Frontiers in Neurorobotics	Frontiers in Neurorobotics, 1, 1.47% Frontiers in Neurorobotics 1 publication, 1.47%
BMC Medical Informatics and Decision Making	BMC Medical Informatics and Decision Making, 1, 1.47% BMC Medical Informatics and Decision Making 1 publication, 1.47%
Proceedings : ... IEEE International Conference on Big Data. IEEE International Conference on Big Data	Proceedings : ... IEEE International Conference on Big Data. IEEE International Conference on Big Data, 1, 1.47% Proceedings : ... IEEE International Conference on Big Data. IEEE International Conference on Big Data 1 publication, 1.47%
Frontiers in Pharmacology	Frontiers in Pharmacology, 1, 1.47% Frontiers in Pharmacology 1 publication, 1.47%
International Journal of Multimedia Information Retrieval	International Journal of Multimedia Information Retrieval, 1, 1.47% International Journal of Multimedia Information Retrieval 1 publication, 1.47%
Artificial Intelligence in Data and Big Data Processing	Artificial Intelligence in Data and Big Data Processing, 1, 1.47% Artificial Intelligence in Data and Big Data Processing 1 publication, 1.47%
Wireless Personal Communications	Wireless Personal Communications, 1, 1.47% Wireless Personal Communications 1 publication, 1.47%
BioMedInformatics	BioMedInformatics, 1, 1.47% BioMedInformatics 1 publication, 1.47%
Journal of the American Medical Informatics Association : JAMIA	Journal of the American Medical Informatics Association : JAMIA, 1, 1.47% Journal of the American Medical Informatics Association : JAMIA 1 publication, 1.47%
Journal of Healthcare Informatics Research	Journal of Healthcare Informatics Research, 1, 1.47% Journal of Healthcare Informatics Research 1 publication, 1.47%
Data in Brief	Data in Brief, 1, 1.47% Data in Brief 1 publication, 1.47%
Studies in Big Data	Studies in Big Data, 1, 1.47% Studies in Big Data 1 publication, 1.47%
Cognitive Computation	Cognitive Computation, 1, 1.47% Cognitive Computation 1 publication, 1.47%
IEEE Reviews in Biomedical Engineering	IEEE Reviews in Biomedical Engineering, 1, 1.47% IEEE Reviews in Biomedical Engineering 1 publication, 1.47%
npj Cardiovascular Health	npj Cardiovascular Health, 1, 1.47% npj Cardiovascular Health 1 publication, 1.47%
Nature Communications	Nature Communications, 1, 1.47% Nature Communications 1 publication, 1.47%
IEEE Internet of Things Journal	IEEE Internet of Things Journal, 1, 1.47% IEEE Internet of Things Journal 1 publication, 1.47%
	1 2 3 4

Publishers

	5 10 15 20 25
Springer Nature	Springer Nature, 24, 35.29% Springer Nature 24 publications, 35.29%
Institute of Electrical and Electronics Engineers (IEEE)	Institute of Electrical and Electronics Engineers (IEEE), 15, 22.06% Institute of Electrical and Electronics Engineers (IEEE) 15 publications, 22.06%
Elsevier	Elsevier, 13, 19.12% Elsevier 13 publications, 19.12%
MDPI	MDPI, 5, 7.35% MDPI 5 publications, 7.35%
Frontiers Media S.A.	Frontiers Media S.A., 2, 2.94% Frontiers Media S.A. 2 publications, 2.94%
American Medical Association (AMA)	American Medical Association (AMA), 1, 1.47% American Medical Association (AMA) 1 publication, 1.47%
Association for Computing Machinery (ACM)	Association for Computing Machinery (ACM), 1, 1.47% Association for Computing Machinery (ACM) 1 publication, 1.47%
Wiley	Wiley, 1, 1.47% Wiley 1 publication, 1.47%
Emerald	Emerald, 1, 1.47% Emerald 1 publication, 1.47%
Oxford University Press	Oxford University Press, 1, 1.47% Oxford University Press 1 publication, 1.47%
Institute of Research and Community Services Diponegoro University (LPPM UNDIP)	Institute of Research and Community Services Diponegoro University (LPPM UNDIP), 1, 1.47% Institute of Research and Community Services Diponegoro University (LPPM UNDIP) 1 publication, 1.47%
JMIR Publications	JMIR Publications, 1, 1.47% JMIR Publications 1 publication, 1.47%
Taylor & Francis	Taylor & Francis, 1, 1.47% Taylor & Francis 1 publication, 1.47%
Public Library of Science (PLoS)	Public Library of Science (PLoS), 1, 1.47% Public Library of Science (PLoS) 1 publication, 1.47%
	5 10 15 20 25

We do not take into account publications without a DOI.
Statistics recalculated weekly.

Are you a researcher?

Create a profile to get free access to personal recommendations for colleagues and new articles.

PDF

Metrics

Cite this

GOST |

Cite this

GOST Copy

Selivanov A. et al. Medical image captioning via generative pretrained transformers // Scientific Reports. 2023. Vol. 13. No. 1. 4171

GOST all authors (up to 50) Copy

Selivanov A., Rogov O. Y., Chesakov D., Shelmanov A., Fedulova I., Dylov D. V. Medical image captioning via generative pretrained transformers // Scientific Reports. 2023. Vol. 13. No. 1. 4171

RIS |

Cite this

RIS Copy

TY - JOUR

DO - 10.1038/s41598-023-31223-5

UR - https://doi.org/10.1038/s41598-023-31223-5

TI - Medical image captioning via generative pretrained transformers

T2 - Scientific Reports

AU - Selivanov, Alexander

AU - Rogov, Oleg Y

AU - Chesakov, Daniil

AU - Shelmanov, Artem

AU - Fedulova, Irina

AU - Dylov, Dmitry V.

PY - 2023

DA - 2023/03/13

PB - Springer Nature

IS - 1

VL - 13

PMID - 36914733

SN - 2045-2322

ER -

BibTex

Cite this

BibTex (up to 50 authors) Copy

@article{2023_Selivanov,

author = {Alexander Selivanov and Oleg Y Rogov and Daniil Chesakov and Artem Shelmanov and Irina Fedulova and Dmitry V. Dylov},

title = {Medical image captioning via generative pretrained transformers},

journal = {Scientific Reports},

year = {2023},

volume = {13},

publisher = {Springer Nature},

month = {mar},

url = {https://doi.org/10.1038/s41598-023-31223-5},

number = {1},

pages = {4171},

doi = {10.1038/s41598-023-31223-5}

}

Publisher

Springer Nature

Journal

Scientific Reports

scimago Q1

wos Q1

SJR

0.874

CiteScore

6.7

Impact factor

3.9

ISSN

20452322 (Print, Electronic)

News

ИИ ускорит постановку диагноза по снимкам легкого