Open Access
Scientific Reports, volume 13, issue 1, publication number 4171
Medical image captioning via generative pretrained transformers
Selivanov Alexander
1, 2
,
Rogov Oleg Y
1
,
Chesakov Daniil
1, 3
,
Shelmanov Artem
1, 3
,
Fedulova Irina
2
,
Dylov Dmitry V.
1
2
Philips (Russia), Moscow, Russia
|
Publication type: Journal Article
Publication date: 2023-03-13
Multidisciplinary
Abstract
The proposed model for automatic clinical image caption generation combines the analysis of radiological scans with structured patient information from the textual records. It uses two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The generated textual summary contains essential information about pathologies found, their location, along with the 2D heatmaps that localize each pathology on the scans. The model has been tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO, and the results measured with natural language assessment metrics demonstrated its efficient applicability to chest X-ray image captioning.
Citations by journals
|
1
2
|
|
|
Applied Sciences (Switzerland)
|
Applied Sciences (Switzerland)
2 publications, 15.38%
|
|
European Radiology
|
European Radiology
1 publication, 7.69%
|
|
JAMA network open
|
JAMA network open
1 publication, 7.69%
|
|
ACM Computing Surveys
|
ACM Computing Surveys
1 publication, 7.69%
|
|
Engineering Reports
|
Engineering Reports
1 publication, 7.69%
|
|
Library Hi Tech
|
Library Hi Tech
1 publication, 7.69%
|
|
Frontiers in Neurorobotics
|
Frontiers in Neurorobotics
1 publication, 7.69%
|
|
BMC Medical Informatics and Decision Making
|
BMC Medical Informatics and Decision Making
1 publication, 7.69%
|
|
Multimedia Tools and Applications
|
Multimedia Tools and Applications
1 publication, 7.69%
|
|
Frontiers in Pharmacology
|
Frontiers in Pharmacology
1 publication, 7.69%
|
|
IEEE Access
|
IEEE Access
1 publication, 7.69%
|
|
1
2
|
Citations by publishers
|
1
2
3
|
|
|
Springer Nature
|
Springer Nature
3 publications, 23.08%
|
|
Multidisciplinary Digital Publishing Institute (MDPI)
|
Multidisciplinary Digital Publishing Institute (MDPI)
2 publications, 15.38%
|
|
Frontiers Media S.A.
|
Frontiers Media S.A.
2 publications, 15.38%
|
|
IEEE
|
IEEE
2 publications, 15.38%
|
|
American Medical Association (AMA)
|
American Medical Association (AMA)
1 publication, 7.69%
|
|
Association for Computing Machinery (ACM)
|
Association for Computing Machinery (ACM)
1 publication, 7.69%
|
|
Wiley
|
Wiley
1 publication, 7.69%
|
|
Emerald
|
Emerald
1 publication, 7.69%
|
|
1
2
3
|
- We do not take into account publications that without a DOI.
- Statistics recalculated only for publications connected to researchers, organizations and labs registered on the platform.
- Statistics recalculated weekly.
{"yearsCitations":{"type":"bar","data":{"show":true,"labels":[2023,2024],"ids":[0,0],"codes":[0,0],"imageUrls":["",""],"datasets":[{"label":"Citations number","data":[9,4],"backgroundColor":["#3B82F6","#3B82F6"],"percentage":["69.23","30.77"],"barThickness":null}]},"options":{"indexAxis":"x","maintainAspectRatio":true,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":1,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Citations per year","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}},"journals":{"type":"bar","data":{"show":true,"labels":["Applied Sciences (Switzerland)","European Radiology","JAMA network open","ACM Computing Surveys","Engineering Reports","Library Hi Tech","Frontiers in Neurorobotics","BMC Medical Informatics and Decision Making","Multimedia Tools and Applications","Frontiers in Pharmacology","IEEE Access"],"ids":[16650,17065,14458,2817,28594,12189,1580,533,21526,9788,25260],"codes":[0,0,0,0,0,0,0,0,0,0,0],"imageUrls":["\/storage\/images\/resized\/MjH1ITP7lMYGxeqUZfkt2BnVLgjkk413jwBV97XX_medium.webp","\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp","\/storage\/images\/resized\/brnEFyEeTGw9nlsHnEHbyCV6AqrisRCX7QF1dLtY_medium.webp","\/storage\/images\/resized\/XZDD1UbkaHV0BImS1Dm7kQfvovjiljgbqNi7vyqK_medium.webp","\/storage\/images\/resized\/bRyGpdm98BkAUYiK1YFNpl5Z7hPu6Gd87gbIeuG3_medium.webp","\/storage\/images\/resized\/y1FWuXRlUNwpMiGVNlyhHQIrgjjPdJWmhKMZwpoB_medium.webp","\/storage\/images\/resized\/4QWA67eqfcfyOiA8Wk7YnqroHFqQbTsmDJUYTCTg_medium.webp","\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp","\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp","\/storage\/images\/resized\/4QWA67eqfcfyOiA8Wk7YnqroHFqQbTsmDJUYTCTg_medium.webp","\/storage\/images\/resized\/6scCJegesojp2jubwY3uKCzTAmgsaH2GIFlg6Hfk_medium.webp"],"datasets":[{"label":"","data":[2,1,1,1,1,1,1,1,1,1,1],"backgroundColor":["#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6"],"percentage":[15.38,7.69,7.69,7.69,7.69,7.69,7.69,7.69,7.69,7.69,7.69],"barThickness":13}]},"options":{"indexAxis":"y","maintainAspectRatio":false,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":null,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Journals","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}},"publishers":{"type":"bar","data":{"show":true,"labels":["Springer Nature","Multidisciplinary Digital Publishing Institute (MDPI)","Frontiers Media S.A.","IEEE","American Medical Association (AMA)","Association for Computing Machinery (ACM)","Wiley","Emerald"],"ids":[8,202,208,6953,2676,1141,11,30],"codes":[0,0,0,0,0,0,0,0],"imageUrls":["\/storage\/images\/resized\/voXLqlsvTwv5p3iMQ8Dhs95nqB4AXOG7Taj7G4ra_medium.webp","\/storage\/images\/resized\/MjH1ITP7lMYGxeqUZfkt2BnVLgjkk413jwBV97XX_medium.webp","\/storage\/images\/resized\/4QWA67eqfcfyOiA8Wk7YnqroHFqQbTsmDJUYTCTg_medium.webp","\/storage\/images\/resized\/6scCJegesojp2jubwY3uKCzTAmgsaH2GIFlg6Hfk_medium.webp","\/storage\/images\/resized\/brnEFyEeTGw9nlsHnEHbyCV6AqrisRCX7QF1dLtY_medium.webp","\/storage\/images\/resized\/XZDD1UbkaHV0BImS1Dm7kQfvovjiljgbqNi7vyqK_medium.webp","\/storage\/images\/resized\/bRyGpdm98BkAUYiK1YFNpl5Z7hPu6Gd87gbIeuG3_medium.webp","\/storage\/images\/resized\/y1FWuXRlUNwpMiGVNlyhHQIrgjjPdJWmhKMZwpoB_medium.webp"],"datasets":[{"label":"","data":[3,2,2,2,1,1,1,1],"backgroundColor":["#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6","#3B82F6"],"percentage":[23.08,15.38,15.38,15.38,7.69,7.69,7.69,7.69],"barThickness":13}]},"options":{"indexAxis":"y","maintainAspectRatio":false,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":null,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Publishers","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}}}
Metrics
Cite this
GOST |
RIS |
BibTex
Cite this
GOST
Copy
Selivanov A. et al. Medical image captioning via generative pretrained transformers // Scientific Reports. 2023. Vol. 13. No. 1. 4171
GOST all authors (up to 50)
Copy
Selivanov A., Rogov O. Y., Chesakov D., Shelmanov A., Fedulova I., Dylov D. V. Medical image captioning via generative pretrained transformers // Scientific Reports. 2023. Vol. 13. No. 1. 4171
Cite this
RIS
Copy
TY - JOUR
DO - 10.1038/s41598-023-31223-5
UR - https://doi.org/10.1038%2Fs41598-023-31223-5
TI - Medical image captioning via generative pretrained transformers
T2 - Scientific Reports
AU - Selivanov, Alexander
AU - Rogov, Oleg Y
AU - Chesakov, Daniil
AU - Shelmanov, Artem
AU - Fedulova, Irina
AU - Dylov, Dmitry V.
PY - 2023
DA - 2023/03/13 00:00:00
PB - Springer Nature
IS - 1
VL - 13
SN - 2045-2322
ER -
Cite this
BibTex
Copy
@article{2023_Selivanov,
author = {Alexander Selivanov and Oleg Y Rogov and Daniil Chesakov and Artem Shelmanov and Irina Fedulova and Dmitry V. Dylov},
title = {Medical image captioning via generative pretrained transformers},
journal = {Scientific Reports},
year = {2023},
volume = {13},
publisher = {Springer Nature},
month = {mar},
url = {https://doi.org/10.1038%2Fs41598-023-31223-5},
number = {1},
doi = {10.1038/s41598-023-31223-5}
}