Cognitive Systems Research, volume 75, pages 16-24
Complexity of symbolic representation in working memory of Transformer correlates with the complexity of a task
Sagirova Alsu
1
,
Burtsev Mikhail
1, 2
Publication type: Journal Article
Publication date: 2022-09-01
Journal:
Cognitive Systems Research
Quartile SCImago
Q1
Quartile WOS
Q1
Impact factor: 3.9
ISSN: 13890417
Artificial Intelligence
Software
Experimental and Cognitive Psychology
Cognitive Neuroscience
Abstract
Even though Transformers are extensively used for Natural Language Processing tasks, especially for machine translation, they lack an explicit memory to store key concepts of processed texts. This paper explores the properties of the content of symbolic working memory added to the Transformer model decoder. Such working memory enhances the quality of model predictions in machine translation task and works as a neural-symbolic representation of information that is important for the model to make correct translations. The study of memory content revealed that translated text keywords are stored in the working memory, pointing to the relevance of memory content to the processed text. Also, the diversity of tokens and parts of speech stored in memory correlates with the complexity of the corpora for machine translation task. • Working memory in Transformer helps to improve the machine translation quality. • Translated text keywords are stored in the working memory. • Working memory diversity correlates with the corpora complexity.
Citations by journals
|
1
|
|
|
Computers in Biology and Medicine
|
Computers in Biology and Medicine
1 publication, 100%
|
|
1
|
Citations by publishers
|
1
|
|
|
Elsevier
|
Elsevier
1 publication, 100%
|
|
1
|
- We do not take into account publications that without a DOI.
- Statistics recalculated only for publications connected to researchers, organizations and labs registered on the platform.
- Statistics recalculated weekly.
{"yearsCitations":{"type":"bar","data":{"show":true,"labels":[2023],"ids":[0],"codes":[0],"imageUrls":[""],"datasets":[{"label":"Citations number","data":[1],"backgroundColor":["#3B82F6"],"percentage":["100"],"barThickness":null}]},"options":{"indexAxis":"x","maintainAspectRatio":true,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":1,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Citations per year","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}},"journals":{"type":"bar","data":{"show":true,"labels":["Computers in Biology and Medicine"],"ids":[10667],"codes":[0],"imageUrls":["\/storage\/images\/resized\/GDnYOu1UpMMfMMRV6Aqle4H0YLLsraeD9IP9qScG_medium.webp"],"datasets":[{"label":"","data":[1],"backgroundColor":["#3B82F6"],"percentage":[100],"barThickness":13}]},"options":{"indexAxis":"y","maintainAspectRatio":false,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":null,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Journals","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}},"publishers":{"type":"bar","data":{"show":true,"labels":["Elsevier"],"ids":[17],"codes":[0],"imageUrls":["\/storage\/images\/resized\/GDnYOu1UpMMfMMRV6Aqle4H0YLLsraeD9IP9qScG_medium.webp"],"datasets":[{"label":"","data":[1],"backgroundColor":["#3B82F6"],"percentage":[100],"barThickness":13}]},"options":{"indexAxis":"y","maintainAspectRatio":false,"scales":{"y":{"ticks":{"precision":0,"autoSkip":false,"font":{"family":"Montserrat"},"color":"#000000"}},"x":{"ticks":{"stepSize":null,"precision":0,"font":{"family":"Montserrat"},"color":"#000000"}}},"plugins":{"legend":{"position":"top","labels":{"font":{"family":"Montserrat"},"color":"#000000"}},"title":{"display":true,"text":"Publishers","font":{"size":24,"family":"Montserrat","weight":600},"color":"#000000"}}}}}
Metrics
Cite this
GOST |
RIS |
BibTex
Cite this
GOST
Copy
Sagirova A., Burtsev M. Complexity of symbolic representation in working memory of Transformer correlates with the complexity of a task // Cognitive Systems Research. 2022. Vol. 75. pp. 16-24.
GOST all authors (up to 50)
Copy
Sagirova A., Burtsev M. Complexity of symbolic representation in working memory of Transformer correlates with the complexity of a task // Cognitive Systems Research. 2022. Vol. 75. pp. 16-24.
Cite this
RIS
Copy
TY - JOUR
DO - 10.1016/j.cogsys.2022.05.002
UR - https://doi.org/10.1016%2Fj.cogsys.2022.05.002
TI - Complexity of symbolic representation in working memory of Transformer correlates with the complexity of a task
T2 - Cognitive Systems Research
AU - Sagirova, Alsu
AU - Burtsev, Mikhail
PY - 2022
DA - 2022/09/01 00:00:00
PB - Elsevier
SP - 16-24
VL - 75
SN - 1389-0417
ER -
Cite this
BibTex
Copy
@article{2022_Sagirova,
author = {Alsu Sagirova and Mikhail Burtsev},
title = {Complexity of symbolic representation in working memory of Transformer correlates with the complexity of a task},
journal = {Cognitive Systems Research},
year = {2022},
volume = {75},
publisher = {Elsevier},
month = {sep},
url = {https://doi.org/10.1016%2Fj.cogsys.2022.05.002},
pages = {16--24},
doi = {10.1016/j.cogsys.2022.05.002}
}
Profiles